Cloud rearrangement for fun and profit

By , May 17, 2015 4:42 am

In a populated compute cloud, there are several scenarios in which it’s beneficial to be able to rearrange VM guest instances into a different placement across the hypervisor hosts via migration (live or otherwise). These use cases typically fall into three categories:

  1. Rebalancing – spread the VMs evenly across as many physical VM host machines as possible (conceptually similar to vSphere DRS). Example use cases:
  2. Consolidation – condense VMs onto fewer physical VM host machines (conceptually similar to vSphere DPM). Typically involves some degree of defragmentation. Example use cases:
  3. Evacuation – free up physical servers:

Whilst one-shot manual or semi-automatic rearrangement can bring immediate benefits, the biggest wins often come when continual rearrangement is automated. The approaches can also be combined, e.g. first evacuate and/or consolidate, then rebalance on the remaining physical servers.

Other custom rearrangements may be required according to other IT- or business-driven policies, e.g. only rearrange VM instances relating to a specific workload, in order to increase locality of reference, reduce latency, respect availability zones, or facilitate other out-of-band workflows or policies (such as data privacy or other legalities).

In the rest of this post I will expand this topic in the context of OpenStack, talk about the computer science behind it, propose a possible way forward, and offer a working prototype in Python.

If you’re in Vancouver for the OpenStack summit which starts this Monday and you find this post interesting, ping me for a face-to-face chat!

Continue reading 'Cloud rearrangement for fun and profit'»


Tories to limit use of mathematics in amendment to anti-terrorism bill

By , May 9, 2015 3:45 am

Following on from the Conservative Party’s plans to take immediate advantage of their new majority in the House of Commons by pushing through surveillance powers known as the Snoopers’ Charter, the party has announced an amendment to the bill which will make it illegal for anyone to use any form of mathematics not on a government-approved whitelist.

In yesterday’s announcement, Theresa May, who as home secretary led the original legislation, said: “We were disappointed to receive feedback on the original Communications Data Bill from technology experts and civil liberties campaigners who considered it more important for citizens to be able to continue using encryption for non-essential activities like secure online shopping / banking, than for the police to be able to monitor the communications of anyone who could be a terrorist. The country was extremely healthy under John Major’s government in the 1990s before online services such as e-commerce and e-banking even existed, so it is a trivial and easily justifiable sacrifice to replace the freedom to use those services securely with laws creating a powerful deterrent for terrorists, who would face stiff fines and potentially even jail-time if found guilty of using encrypted communications.”

“However, during consultations with the financial sector in the City, we have been advised that banning use of all encryption software would prevent large UK corporations from trading on global markets.”

She continued, “We also discovered that communication can be encrypted non-electronically, for example using simple mathematical techniques on pen and paper, and we cannot in good conscience allow potential terrorists to use these techniques without fear of being arrested and detained for an arbitrary amount of questioning.”

“Therefore the only logical course of action is to amend the bill to ban use of all types of mathematics for which permission has not been explicitly granted by the government. A whitelist will be drafted for the upcoming debate on the bill. In order to avoid any impact on the economy, a special security exception will be made to allow financial institutions to continue using mathematics as before. For ordinary citizens, basic arithmetic will of course be allowed, although in financial contexts some restrictions will be imposed; for example, in the interests of national security, it will be forbidden for the general public to perform calculations relating to any personal expenditure of MPs or peers in the House of Lords.”

David Cameron issued a separate statement reinforcing the Home Secretary’s announcement and also rejecting an opposing argument which highlighted that whilst every year in the UK around 2,000 people die from traffic accidents and 65,000 from heart disease, in the past 5 years there have only been 2 people killed through terrorism. “Terrorism is a rising global threat, and must be countered at any cost, even at the expense of civil liberties and personal privacy”, the newly re-elected Prime Minster said. “If you have nothing to hide, why would you need privacy anyway? Everybody already shares everything on Facebook anyway.”


Why and how to correctly amend GitHub pull requests

By , March 24, 2015 3:00 pm

Like many F/OSS developers, I’m a heavy user of GitHub, collaborating on many projects which use the typical “fork & pull” workflow based on pull requests. The GitHub documentation on pull requests covers this workflow fairly comprehensively, but there seems to be one area which is significantly lacking in detail: why and how to amend existing pull requests. The article simply says:

After your pull request is sent, any new commits pushed to your branch will automatically be added to the pull request. This is especially useful if you need to make more changes.

The problem is that this completely ignores the fact that there are often very good reasons for amending existing commits within the pull request, not just for adding new commits it.

Why amend an existing pull request?

A peer review cycle can potentially reveal many issues which make the pull request unready for merging, e.g.

  • typos
  • bugs in the proposed code changes
  • missing features in the proposed code changes
  • incomplete test coverage
  • incomplete documentation changes
  • style inconsistencies (including whitespace issues)
  • incorrect or incomplete commit messages
  • the commits violate the rule of one logical change per commit
  • some changes are outside the scope of the pull request

This is of course what makes peer review of pull requests so valuable: the problems can be addressed even before they hit the master branch, which helps maintain high quality in the development trunk. But then how do we address the issues?

Continue reading 'Why and how to correctly amend GitHub pull requests'»


Announcing git-deps: commit dependency analysis / visualization tool

By , January 19, 2015 12:15 am

I’m happy to announce a new tool called git-deps which performs automatic analysis and visualization of dependencies between commits in a git repository. Here’s a screencast demonstration!

Back in 2013 I blogged about some tools I wrote which harness the notes feature of git to help with the process of porting commits from one branch to another. These are mostly useful in the cases where porting is more complex than just cherry-picking a small number of commits.

However, even in the case where there are a small number of desired commits, sometimes those commits have hidden dependencies on other commits which you didn’t particularly want, but need to pull in anyway, e.g. in order to avoid conflicts during cherry-picking. Of course those secondary commits may in turn require other commits, and before you know it, you’re in dependency hell, which is only supposed to happen if you’re trying to install Linux packages and it’s still 1998 … but in fact that’s exactly what happened to me at SUSEcon 2013, when I attempted to help a colleague backport a bugfix in OpenStack Nova from the master branch to a stable release branch.

At first sight it looked like it would only require a trivial git cherry-pick, but that immediately revealed conflicts due to related code having changed in master since the release was made. I manually found the underlying commit which the bugfix required by using git blame, and tried another cherry-pick. The same thing happened again. Very soon I found myself in a quagmire of dependencies between commits, with no idea whether the end was in sight.

So wouldn’t it be nice if you could see the dependency tree ahead of time, rather than spending a whole bunch of time resolving unexpected conflicts due to missing dependencies, only to realise that the tree’s way deeper than you expected, and that actually a totally different approach is needed? Well, I thought it would, and so git-deps was born!

In coffee breaks during the ensuing openSUSE conference at the same venue, I feverishly hacked together a prototype and it seemed to work. Then normal life intervened, and no progress was made for another year.

However thanks to SUSE’s generous Hack Week policy, I have had the luxury of being able to spending some of early January 2015 working to bring this tool to the next level. I submitted a Hack Week project page, announced my intentions on the git mailing list, started hacking, missed quite a bit of sleep, and finally recorded the above screencast.

The tool is available here:

Please give it a go and let me know what you think! I’m particularly interested in hearing ideas for use cases I didn’t think of yet, and proposals for integration with other git web front-ends.


Rethinking traditional scale practice on the cello

By , December 20, 2014 2:28 am

(Please note that while this post was written with the cello in mind, several of the principles apply to other instruments too.)

I’ve previously blogged about how back in 2011, I decided to get serious about trying to become a jazz cellist. Since then I’ve had some really interesting and fulfilling musical experiences, but it’s also been a struggle in some areas. I studied and performed classical (in the western sense) cello for many years, but the demands of jazz are so different to most things I worked on during my classical training that I’ve had to fundamentally rethink many aspects of the way I practise.

Scales are one good example of this. I was taught to practise B♭ major something like the following:

standard classical cellistic approach to practising a B♭ major scale

The first interesting thing to note about this is that it omits almost the entire bottom octave of the instrument! There’s really no good reason for this; the bottom octave deserves attention just like the rest of the instrument. It’s only taught this way because the tradition has become incredibly “tonic-biased”, with no awareness of the different modes which even beginner jazz students get taught about.

So one of the first adjustments I made (and I probably stole this idea from one of my mentors, the great jazz violinist Christian Howes, who calls it “extended range”) was to always start from the lowest note of the scale reachable on the instrument:

starting B♭ major on C

Similarly, if the top tonic is not near the top of the range you want to practise in, just keep going until you hit a note high enough to make you think “OK, I don’t need to practise any higher than this!” Whether or not that note happens to be the tonic of the scale should be irrelevant; the most important thing is that you are practising the full range of the instrument which you need to use when performing.

Turning it all upside-down

Another bias which exists in the western classical tradition for dubious reasons is that everything is bottom-heavy: almost all exercises start at the bottom, go up to the top, and come back down again. This inevitably results in the bottom end being practised more than the top, which is particularly bad because the top end is way harder to play well. So let’s swap it all around!

starting B♭ major at the top

(By the way, along similar lines I recommend this great blog post on back-to-front learning by Anton Schwarz.)

Developing new “real world” fingering systems

The next odd thing to note about the classical tradition is that scales are almost always played using a single fingering, and that fingering is typically tonic-biased in another way: the tonic notes are very commonly played using first finger. Perhaps the classical pedagogy evolved this way simply because a single fingering system for each scale was perceived as easier to teach and also for students to memorise. But this approach is hopelessly inconsistent with the real world. If you don’t believe me, take a look at every fragment of a B♭ major scale present in the solo part of Boccherini’s cello concerto in B♭ major, and count for how many of those fragments it would make sense to use the fingering system shown above. If we don’t use those fingering systems when playing actual music, why spend so much time practising them?

So it seems clear to me that practising multiple fingering systems is beneficial for technical reasons, and in the case of jazz for musical reasons too: the improvisatory nature of which means that unless you rely very heavily on licks, you rarely know far in advance what notes you want to play, let alone what fingerings to use. So for any given tonality and position on the instrument, you should practise as many fingering positions as possible, so that when you are in the moment of improvising a solo, whichever hand position you’re in, as many notes are physically accessible from there because you’ve already covered those transitions in practice.

A system for practising all fingerings

But where do we draw the line? After all there are an enormous number of different possible fingerings for a 3-octave scale, and there’s just not enough time in the day to practise even a fraction of them. Or is there?

Violin fingerings

On the violin, this is a bit simpler to understand because pretty much any scale can be played in any hand position without shifting (at least, until you hit the top or bottom notes in that position). So on violin, you can simply practise any scale in any position, and carve up your available practice time accordingly: for example on Monday you could focus on 1st position, on Tuesday 2nd position, and so on.

Cello fingerings

On cello, it’s not possible to play any diatonic scale without a considerable amount of shifting of the hand position, so the problem of how to methodically practise all possible fingerings is far less obvious.

I think the answer lies in considering how to group notes into hand positions. It’s clear from the above examples that typically the hand position shifts every three notes, i.e. the majority of notes in scales on the cello are grouped in threes: 1-2-4 or 1-3-4 in the lower octaves, and 1-2-3 when playing higher up the string. Most exceptions to this are when the open strings are played (but they could be considered as “outside” the groupings anyway), and in the higher octaves of most cello scales, where 2-note groupings are common (e.g. the 1-2 1-2 1-2-3 type of fingering system is very commonly taught in the classical tradition for the top octave of a 3-octave scale).

My opinion is that practising 2-note groupings is really not that useful (at least to me as a jazz cellist), because

  1. the shift distance is smaller so the shift is easier,
  2. 2-note groupings require 50% more frequent shifts when compared to 3-note groupings, which is a big hindrance when attempting to play at speed,
  3. every 2-note finger grouping is already covered by a 3-note grouping, and
  4. I’ve already logged years of classical practice covering 2-note groupings anyway!

So I decided to practise all my scales in only 3-note groupings:

using 3 note finger groupings

Notice here that I’ve also stopped using the open strings (except for the bottom C, since there’s no other way to play it). I made this decision for three reasons. Firstly, in jazz it’s important to be able to transpose to another key at any moment, so I don’t want to become too reliant on scale fingerings which use open strings, since they only work in one key.

Secondly, it makes it very easy to extend this system so that it covers every possible fingering of the scale using 3-note groupings! This is possible simply by shifting the starting hand position by a single note:

starting with the hand position offset

and then shifting it once more:

starting with the hand position offset twice

String variations

Thirdly, it allows freedom to introduce variations on which string is used for a particular group. For example bars 5 and 6 of the above example could be played like this:

string variations

I don’t use a strict system to ensure I always cover all possible string variations; instead I like to improvise the choice of string on the spur of the moment, since this matches more closely what can happen when improvising on stage.

Connecting the dots

Of course this system of 3-note groupings with 3 different starting positions for every scale triples the amount of scales to practise! However since I would normally practise each scale more than once anyway, I aim to seamlessly join up the different offsets in order to cover all combinations without requiring significantly more practice time:

connecting different hand position offsets

and so on. Actually the hand position “offset” can be changed at any point in the scale, and this is also something I like to improvise on.

Changing the tonality

So far the examples have all used a major scale. The classical tradition also covers harmonic minor and melodic minor (which is different ascending to descending). In the jazz world (especially for more modern jazz), the ascending form of the melodic minor is particularly important, so I practise that in both directions:

melodic minor practice

Jazz also uses the harmonic major scale and many other scales, so I mix some of these into my practice, although I find it easier to spend a few weeks/months focusing on one particular scale rather than attempt to cover all scales in every practice routine, which just seems too ambitious. Incidentally, back in 2013 I built the “Scale Matcher”, a web application to show which scales can be used for improvisation in any given chordal context.

Changing the key

Fortunately not all music in the world is written purely in B♭. Unfortunately this means we have to practise other keys too! I try to change key seamlessly without stopping, because this is one of the most fundamental requirements in jazz for improvising over a series of chord changes. Again I like to avoid any tonic bias by changing key on other degrees of the scale. For example, after completing one cycle of all 3 hand positions in B♭ minor, I could shift tonality up a semitone to B minor, changing near the top of the instrument:

seamlessly changing the key at the top

or on the lowest note of the instrument:

seamlessly changing the key at the bottom

or even somewhere in the middle:

seamlessly changing the key in the middle

This is a fun workout for the brain as well as the fingers!

Hand position “windows”

Above I’ve suggested a system for dealing with the cello’s limitation of not being able to play a continuous scale in any key without shifting the hand, which ensures that every possible fingering of a given scale can be methodically covered during practice.

However there’s a completely opposite approach, which is to embrace this limitation and simply fix your hand in a single position and only play the notes of the scale which you can reach from that position, dropping the others! This is particularly useful when improvising at speed, since it drastically reduces the amount of shifting required. I learnt this idea from three of my all-time cello heroes (Erik Friedlander, Mike Block, and Rushad Eggleston), and then extended it to apply to every scale I work on:

position windows in B♭ minor

I’ve found that improvising within a given “window” can result in some interesting melodic shapes.

Patterns / shapes

Rather than simply ascending or descending, there are an infinite number of patterns to choose from for varying the shapes generated by traversing the scale. Here are a few simple examples:

position patterns in B♭ minor

(The last two were shamelessly stolen from Christian Howes.)

Rhythm / metronome usage

So far I haven’t said anything about rhythm or use of the metronome (which in my opinion is an essential part of scale practice). In fact this topic is so huge I’ll leave it for another blog post at another time! But of course there are an infinite number of ways to vary rhythm, and the western tradition tends not to examine these in much detail. One thing I enjoy doing is freely improvising the rhythm when playing scales, even if everything else (notes, fingerings etc.) is fixed.


How to articulate with the bow is another huge topic, but this post is already way too long!


It’s clear that there are zillions of ways to vary scale practice and keep it interesting whilst challenging yourself in new ways. Perhaps the most important thing for any diligent musician is to constantly invent their own new practice systems, rather than just copying someone else’s verbatim.

If you’ve made it this far, congratulations! This is pretty dry reading, but I hope you found it useful or at least interesting. If you have any feedback of any sort, I’d love to hear it, so please leave a comment!

P.S. For the curious geeks, this blog post was created from a single text file with the combination of some incredible free software: the GNU Emacs editor, GNU Lilypond music typesetter, Org mode, org2blog, and the Org-babel-lilypond backend to Babel.


Panorama Theme by Themocracy