The first pair are available as patches to git itself:

• git-splice – splices commits onto/into/out of current branch

  Jump to 08:44 in the video to see me introduce and demo this, or look at the slides, or read the man page for example usage.

• git-transplant – transplants commits from current branch onto/into another

  Jump to 32:22 in the video to see me introduce and demo this, or look at the slides, or read the man page for example usage.

The second pair are available as Python modules on PyPI, but both have a command-line interface too:

• git-deps – automatically detects textual dependencies between commits

  Jump to 41:28 in the video to see me introduce and demo this, or look at the slides, or read the README (which has a couple of short YouTube videos demoing the tool including the fancy web interface), or read my previous blog post about this.

• git-explode – explodes a linear sequence of git commits into new independent topic branches

  Jump to 51:10 in the video to see me introduce and demo this, or look at the slides, or read the README.

It’s taken a while to get them to a reasonable degree of polish, during which this work has been mostly under the radar, but now I’m now going to make a more concerted effort to advertise the tools, hence this blog post.

(Jump to 02:27 in the video to see me talk about this, or look at the slides.)

All developers face common problems:

• It’s difficult to predict what you’ll be doing next. One minute you might be working on a feature, the next you find a bug you need to fix, and the next you find some technical debt which requires refactoring.
• Multi-tasking is almost unavoidable.
• We’re human, so we sometimes screw up when trying to follow best practice workflow.

What does this mean when using git? Typically it means we are constantly juggling branches, and often need to manipulate the history of branches for various reasons, for example:

• Maintaining clean separation of topics between branches to facilitate code reviews: one branch for a refactoring, one for a bugfix, one for a feature, and so on.
• Backporting or forward-porting series of patches between branches.

My feeling is that while the git ecosystem has continued to evolve incrementally over the last few years, not much progress has been made in terms of automating this janitorial work, and that most people either do it by hand, or don’t bother doing it at all. This leads to unclean history, and/or more work and frustration for reviewers.

So I wrote these tools (and others) in an attempt to push the boundaries of automating git workflows to higher levels, as I explained on the git mailing list last year. Even if all the use cases they can cover have yet to be discovered, I hope you find them useful, or at least thought-provoking enough that you might join the effort!

I’d love some feedback, so do let me know what you think in the comments below.

It is clear from the diversity of the use cases listed above that VM placement policies are likely to vary greatly across clouds, and sometimes even within a single cloud. OpenStack Compute (nova) has fairly sophisticated scheduling capabilities which can be configured to implement some of the above policies on an incremental basis, i.e. every time a VM instance is started or migrated, the destination VM host can be automatically chosen according to filters and weighted cost functions. However, this approach is somewhat limited for migration, because the placement policies are only considered one migration at a time. Like in Tetris, more efficient arrangements can be achieved by thinking further ahead!

OpenStack clouds can already be segregated in various ways via cells, regions, availability zones, host aggregates, and server groups, and so there are various techniques available for implementing different placement policies to different areas of your cloud. For example you could have groups of hosts dedicated to CPU-intensive or I/O-bound workloads using an anti-affinity placement policy, and other groups dedicated to light workloads using an affinity placement policy. But these policies are static in nature (e.g. nova filters and schedulers are configured by the cloud administrator in nova.conf), which somewhat limits their flexibility.

OpenStack is still relatively new, however with Cinder and Neutron rapidly evolving, it is at the point where a VM’s network and storage dependencies can be live migrated along with the workload in a near seamless fashion. So now we should be able to start developing mechanisms for implementing more sophisticated placement policies, where not only is VM rearrangement performed automatically, but the policies themselves can be varied dynamically over time as workload requirements change.

VMware advocates might instinctively recommend running vSphere clusters underneath nova, to harness vSphere’s pre-existing DRS / DPM functionality. However, there are still unresolved problems with this approach, as this article on nova-scheduler and DRS highlights. (Funnily enough, I encountered exactly these same problems a few years ago when I was part of a team adding vSphere integration into the orchestration component of what has since become NetIQ Cloud Manager …) Besides, a vSphere cluster and an OpenStack cloud are very different beasts, and of course not everyone wants to build their cloud exclusively with VMware technology anyway.

Unfortunately developing algorithms to determine optimal placement is distinctly non-trivial. For example, the consolidation scenario above is a complex variant of the bin packing problem, which is NP-hard. The following constraints add significant complexity to the problem:

• A useful real world solution should take into account not only the RAM footprint of the VMs, but also CPU, disk, and network.
• The algorithm needs to ensure that SLAs are maintained whilst any rearrangement takes place.
• If the cloud is too close to full capacity, it may not be possible to rearrange the VMs from their current placement to a more optimal placement without first shutting down some VMs, which could be prohibited by the SLAs.
• Even if the cloud is at low utilization, live migration is not always possible, e.g.
  • the hypervisor may not support it (especially across non-identical CPU architectures)
  • shared storage may be required but unavailable
  • the network may not support it
• Even if the arrangement is achievable purely via a sequence of live migrations, the algorithm must also be sensitive to the performance impact to running workloads when performing multiple live migrations, since live migrations require intensive bursts of network I/O in order to synchronize the VM’s memory contents between the source and target hosts, followed by a momentary freezing of the VM as it flips from the source to the target. This trade-off between optimal resource utilization and service availability means that a sub-optimal final placement may be preferable to an optimal one.
• In the case where the hypervisor is capable of sharing memory pages between VMs (e.g. KSM on KVM), the algorithm should try to place together VMs which are likely to share memory pages (e.g. VMs running the same OS platform, OS version, software libraries, or applications. A research paper published in 2011 demonstrated that VM packing which optimises placement in this fashion can be approximated in polytime, achieving 32% to 50% reduction in servers and a 25% to 57% reduction in memory footprint compared to sharing-oblivious algorithms.

(There is prior art of course. For example, several years ago, Pacemaker implemented a best effort algorithm for VM host selection.)

As noted by the 2011 research paper referenced above, this area of computer science is still evolving. There is one constant however: any rearrangement solution must not only provide a final VM placement optimised according to the chosen constraints, but also a sequence of migrations to it from the current placement. There will often be multiple migration sequences reaching the optimised placement from the current one, and their efficiency can vary widely. In other words, there are two questions which need answering:

1. Given a starting placement A, which is the best (or near-optimal) final placement B to head for?
2. What’s the best way to get from A to B?

The above considerations strongly suggest that the first question is much harder to answer than the second, although the two are not necessarily orthogonal; for example there could be two different final placements B and C which are equally near-optimal, but it may be much harder to reach C from A than B.

I propose that it is worth examining the effectiveness of a divide and conquer approach: solving the second question may simplify the first, and also provide a mechanism for comparatively evaluating the effectiveness of potential answers to the first. Another bonus of this decoupling is that it should be possible for the path-finding algorithm to also discover opportunities for parallelizing live migrations when walking the path, so that the target placement B can be reached more quickly.

I’ve designed an algorithm which solves the second problem, and hacked together a proof of concept implementation in Python. This video shows it solving three randomly generated scenarios:

It’s far from perfect, but as a proof of concept it seems to hold up fairly well under soak testing. I’m pleased to announce that the code is now on github: https://github.com/aspiers/migrationpaths and I would love to hear feedback on it.

If you found this blog interesting, feel free to get in touch! I’m aspiers on FreeNode IRC, and adamspiers on Twitter.

I’m also in Vancouver for the OpenStack summit which starts this Monday, so if you are, come and say hello!

Many thanks to Florian Haas, Steven Hardy, and Christoph Thiel for their input on an earlier draft version of this post, and apologies that it took me SO long to publish it!

You’ll need the following:

• At least three physical or virtual 64-bit x86 machines, each with at least 2GB RAM and 8GB disk, and no valuable data on any of the attached disks:

  • one admin node running the Crowbar deployment tool which will provision the other nodes from scratch,
  • at least one controller node which will run OpenStack infrastructure services, and
  • at least one compute node which will host VM instances within the cloud. (If this compute node is a VM, then the VM instances in the cloud will have to be run either using KVM nested virtualization, or QEMU software virtualization which is slower but good enough for “kicking the tires”.)

• A private IPv4 network which all the machines must be connected to. Setting this up is the only potentially tricky bit of the whole exercise. By default the network in question needs to be 192.168.124.0/24 with no DHCP server enabled, so:

  • if you are installing the appliance in a VM, you should be able to set up a NAT or host-only virtual network and configure your hypervisor so that it does not serve DHCP on that network, or
  • if you are installing on bare metal, ensure there is no DHCP server active on that L2 segment.

• Another machine (physical or virtual) with a modern web browser on the same private IPv4 network.

Attach the bootable USB media (physical or virtual), and boot the machine. This will automatically install a SUSE Cloud Admin Node onto the disk. Caution: this will wipe any pre-existing OS, so only use it on a spare machine or freshly-created VM, with no valuable data on any of the attached disks!

After confirming you are OK to wipe all existing data on the disk, the appliance will be written to disk and then booted. Shortly after, YaST will appear, allowing you to configure:

• which keyboard layout you want,
• what password to use for the root user,

• what hostname and domain name to use (the defaults are fine),

• the network setup (the default of 192.168.124.10/24 is recommended, otherwise you will also have to configure Crowbar prior to installation),
• the clock and time zone, and finally
• the NTP configuration (supplying an upstream NTP server is recommended but not required).

Log in as root (with the password you specified above) either on the console or via ssh root@192.168.124.10 if you have another machine configured to be on the same subnet.

Press q and then y to accept the beta EULA, which highlights that the appliance is partially based on unreleased code. Please do not use it for production deployments!

Crowbar is very powerful and flexible in terms of network configuration. If you have other traffic on the L2 network segment (e.g. if you are using bare metal hardware and a physical network rather than a dedicated virtual network) then you should check that its default networks don’t conflict with your existing traffic. To do this, type yast crowbar:

and select the Networks tab:

From here you can examine and change the networks Crowbar will use. Some are only used when various options are selected later on, but at a minimum, admin, nova_fixed, and nova_floating will all be used. For more information, see the Networking section of the SUSE Cloud Deployment Guide.

From the shell prompt, type screen go to initiate the installation of SUSE Cloud. This will take several minutes.

On another machine on the same network as the machine running the now-installed appliance, start a browser, and navigate to http://192.168.124.10:3000/ (adjust the IP accordingly if you changed the admin network above).

Now simply PXE-boot your other nodes. (Typically this requires PXE-booting to be enabled in the BIOS, and/or manually selecting it from the BIOS boot menu which is often accessible by hitting the F11 key or similar during boot.)

This will run a small inventory ramdisk image on each one to detect its hardware and report the discovery back to Crowbar, without touching the node’s local disk(s). Each node will then appear in the Crowbar web UI, and sit in an idle loop whilst awaiting task allocation via Crowbar:

Clicking on that node will show the results of the automatic hardware inventorying, and give us the option to allocate the node:

When editing the node, we can give it a more human-friendly alias (e.g. node1), and then click Allocate to install a minimal SLES OS:

A full OS will be automatically installed on the node via AutoYaST:

When OS installation has finished, the console looks like this:

and then the node turns green in the web UI indicating that it’s ready to have roles assigned to it:

Multiple nodes can be installed via PXE/AutoYaST at the same time.

By this point you should have at least two freshly-installed nodes managed by Crowbar (excluding the admin node itself which Crowbar runs on), in which case you are ready to deploy OpenStack via Crowbar’s barclamps, which can be found via the Barclamps drop-down in Crowbar’s web interface:

This process is relatively straightforward, and a full explanation can be found in the corresponding chapter of the SUSE Cloud 4 Deployment Guide.

However, it is also possible to automate the whole deployment from this point on, using Crowbar’s batch subcommand and an appropriately crafted .yaml file. There are three sample .yaml files in /root on the admin node. The simplest configuration reflected by these three files is simple-cloud.yaml, which assumes a single controller node and a single compute node, with aliases controller1 and compute1 respectively:

In this case, assuming your nodes are given the above aliases, you could set up the entire OpenStack cloud with this single command run as root on the admin node:

crowbar batch --timeout 1800 build simple-cloud.yaml

It takes a while to apply all the barclamps, as can be seen from the timestamps whilst it’s running:

(The other two sample .yaml files are for a highly available control plane which assumes you have three nodes aliased controller1, controller2, and compute1.)

Once the barclamps are all applied, they should show as green in the Crowbar UI view:

From the main Nodes dashboard in the Crowbar web UI, click the controller1 node (or whichever one you deployed OpenStack’s Dashboard to), and you will see a couple of links to the OpenStack Dashboard (a.k.a “Horizon”):

Click on OpenStack Dashboard (admin) and it will take you to the OpenStack Dashboard, where you can log in as the admin user with a password of (by default) crowbar:

Congratulations! You have set up a full OpenStack cloud from scratch! Now you can start reading the SUSE Cloud Admin and End User guides to learn more about how to use OpenStack.

Whilst this bootable appliance is partially based on unreleased, unsupported code, we are still very interested to hear feedback from our customers and partners. So while we (obviously!) cannot offer unlimited free support for it, if you post any questions / issues to the SUSE Cloud web forum, we will try to respond on a best-effort basis. (And of course full commercial support for the released version of SUSE Cloud is available if you want it 😉 )

As we say in the SUSE world, have a lot of fun!

It seems the tireless Mark McLoughlin, one of the leading lights of the OpenStack project, was already thinking along similar lines, and wrote a tool called palaver, which also aims to streamline the porting process, and also uses git notes to persist state pertaining to backporting projects. Upon first glance, the main differences between our tools seem to be as follows:

• git-cherry-menu is a slightly overgrown shell-script (yuk), whereas palaver is Python (yay).
• palaver has a nice CLI with numerous non-interactive sub-commands like palaver list merged, and palaver review which provides an interactive mode. git-cherry-menu only offers an interactive mode (which is very similar in concept to palaver review); for non-interactive querying of the current state, you have to use a separate command git-icing. Personally I prefer palaver‘s unified approach to UI design, even though it doesn’t yet have the pretty ANSI colour-coding of git-icing and git-cherry-menu.
• Since git-icing wraps around git cherry, it can automatically detect some (but not all) commits which have already been ported to the target branch. As the name suggests, git-cherry-menu enables cherry-picking of commits, whereas that’s still on the TODO list for palaver.
• palaver persists state by writing notes which are JSON blobs, whereas git-cherry-menu uses plain text where lines beginning with skip: are treated specially. Not much difference here: obviously JSON allows more sophisticated structuring of metadata, but has the disadvantage of being less readable if you view the notes branch via standard git tools such as git log or gitk.
• palaver “manually” classifies commits into the following states: pass / defer / pick / proposed / merged, whereas my tools have fewer “manual” states – conceptually just skip and todo. merged is either automatically detected via git cherry, or manually flagged via a skip. I’d say my approach is marginally uglier but the automatic detection is quite a big advantage.
• git-rnotes allows merging of someone else’s notes. That’s also on palaver‘s TODO list.

Perhaps the best way forward might be to port the remaining functionality which git-icing and git-cherry-menu offer to palaver, since the latter has a nicer interface, and is pure Python. None of this is rocket science, so hopefully it wouldn’t take too long.

It probably makes sense to keep git-rnotes separate, since it is a very simple shell-script which is useful for sharing any git notes, not just those generated during a commit porting project.

Secondly, it turns out that Gerrit also uses git-notes, in order to annotate each reviewed change with a bunch of very useful metadata, and GitHub is clever enough to expose this metadata in its web UI, e.g.

In Gerrit 2.6 and newer, this functionality was moved out from the core into a reviewnotes plugin. As a result, AFAICS its documentation seems to be buried inside a separate git repository rather than exposed as a nice web page like it used to be. However, it’s very easy to use, and OpenStack’s Gerrit instance certainly has the feature enabled.

For example, simply cd into any OpenStack git repository you happen to have checked out (or clone one first if not). Then type the following:

git fetch origin refs/notes/review:refs/notes/review
export GIT_NOTES_REF=refs/notes/review
git log

You’ll see that each non-merge commit now shows annotations like the ones in the image above, explaining how the commit made its way through the review process before getting merged by Jenkins. Perhaps the most useful piece of data in the note is the link to the relevant review on http://review.openstack.org, so you can quickly navigate to it in your browser and review the er, review.

If you don’t want git log to default to showing the notes, you can omit the export GIT_NOTES_REF, and instead do:

git log --notes=refs/notes/review

If you don’t want to have to remember the slightly cryptic method for fetching notes, my git-rnotes wrapper offers simpler syntax:

git rnotes fetch origin

However, this currently requires GIT_NOTES_REF to have the right value exported.

However, on the occasions when you need to sift through a larger number of commits on one branch, and port them to another branch, complications can arise:

• If cherry-picking a commit results in changes to its patch context, git patch-id will return a different SHA-1, and subsequent invocations of git cherry will incorrectly tell you that you haven’t yet ported that commit.
• If you mess something up in the middle of a git rebase, recovery can be awkward, and git rebase --abort will land you back at square one, undoing a lot of your hard work.
• If the porting process is big enough, it could take days or even weeks, so you need some way of reliably tracking which commits have already been ported and which still need porting. In this case you may well want to adopt a divide-and-conquer approach by sharing out the porting workload between team-mates.
• The more the two branches have diverged, the more likely it is that conflicts will be encountered during cherry-picking.
• There may be commits within the range you are looking at which after reviewing, you decide should be excluded from the port, or at least porting them needs to be postponed to a later point.

It could be argued that all of these problems can be avoided with the right branch and release management workflows, and I don’t want to debate that in this post. However, this is the real world, and sometimes it just happens that you have to deal with a porting task which is less than trivial. Well, that happened to me and my team not so long ago, so I’m here to tell you that I have written and published some tools to solve these problems. If that’s of interest, then read on!

Since git patch-id doesn’t guarantee to return an SHA-1 which is stable enough for us to always depend on, we need some other way of reliably tracking which commits have been upstreamed, and where upstreamed commits came from. It would also be useful to be able to mark selected commits as blacklisted, i.e. permanently excluded from the current porting process, or at least as deferred, i.e. temporarily excluded.

Of course, we could do that simply by maintaining an out-of-band text file somewhere outside the repositories in question, but that doesn’t integrate at all with existing git workflows. Whilst pondering these challenges, I discovered a little-known and seldom-used feature of git, which is the ability to attach arbitrary text as notes on a given commit, via the git notes command. “This sounds perfect!”, I thought, and actually as it turned out, it served as a pretty nice foundation for some higher-level tools which needed building:

• git-icing – an “interface-candy” wrapper around git cherry which adds a splash of colour, and checks for specially formatted notes attached to commits which indicate that those commits are temporarily or permanently excluded from the upstreaming backlog
• git-cherry-menu – an interactive wrapper around git-icing and git-notes which makes it easy to cherry-pick and/or blacklist non-upstreamed commits
• git-rnotes – a wrapper around git notes which makes it easier to share notes to and from remote repositories

Let’s see how this works in practice, by pretending we need to backport a bunch of commits from OpenStack Nova‘s master branch to its stable/grizzly release branch.

$ git clone git@github.com:openstack/nova.git
Cloning into 'nova'...
remote: Counting objects: 208984, done.
remote: Compressing objects: 100% (54766/54766), done.
remote: Total 208984 (delta 167220), reused 188334 (delta 148279)
Receiving objects: 100% (208984/208984), 124.50 MiB | 4.42 MiB/s, done.
Resolving deltas: 100% (167220/167220), done.
$ cd nova

Now, master has changed a lot since grizzly was released, so let’s keep our pretend project to a manageable size by only attempting to backport only commits which appeared in the first few days after the stable branch was established:

$ git describe $( git merge-base origin/stable/grizzly master )
2013.1.rc1
$ git show 2013.1.rc1 | head -n3
tag 2013.1.rc1
Tagger: Thierry Carrez <thierry@CENSORED.org>
Date:   Wed Mar 20 17:13:32 2013 +0100

OK, so grizzly was branched at the point where the first release candidate was tagged. That makes sense. Now let’s choose a point about 5 days later:

$ git rev-list --no-merges --before='25 March 2013' origin/stable/grizzly..master | head -n1
d6a9d38a0b7488dd77d3a01ae9c4171c271d7314
$ git tag to-backport d6a9d38

… and get ready to backport commits from to-backport back into our local stable/grizzly branch:

$ git checkout stable/grizzly
Branch stable/grizzly set up to track remote branch stable/grizzly from origin.
Switched to a new branch 'stable/grizzly'

(If you are following along at home, your rev-list will give a different result, since origin/stable/grizzly will have changed since I wrote this. Don’t worry – just copy the git tag command above verbatim, and create your local grizzly branch via git checkout -b stable/grizzly fc9af8f0a9 so that it points to the same place mind did when writing this article.)

At this point, two quick clarifications are needed, because things can get a bit confusing.

Firstly, if you read the git cherry man page or -h usage text, it is described as a tool to “find commits not merged upstream”, with the expectation that you would cherry-pick the commits it finds onto the upstream branch. But if you happen to be backporting, then the cherry-picking will happen in the opposite direction, so the “upstream” referred to by the man page is actually the real downstream, and vice-versa. Confused yet? If so, wherever you read the word “upstream” in the git cherry documentation / help text, pretend that you actually read “target”. Then just think of “source” and “target” branches, and remember that the process is always to look for commits in the source which are not yet in the target, and then cherry-pick those into the target (or decide that you don’t want to). The same applies for the git icing and cherry-menu help text.

Secondly, normally we’d be porting from one branch to another, not from a tag to a branch. But in this artificially constructed scenario, it wouldn’t make much sense to create a to-backport branch pointing to d6a9d38, because that branch isn’t going to move; the only branch which will be changing is our stable/grizzly branch, as we cherry-pick stuff into it. In other words, the target has to be a branch, because that will change during the cherry-picking, but the source can be a branch / tag / ref / commit SHA-1 … basically anything which git rev-parse understands.

When embarking on a porting project, the first question is usually “how much work is this going to be?” Fortunately git-icing is here to help!

(By the way, if you want to follow along with the below by trying this yourself, simply download the three scripts either from the links above or

git clone https://github.com/aspiers/git-config

and then copy them to any directory on your $PATH, making sure that they are executable.)

git-icing is designed to be invoked in exactly the same way as git cherry, but it has some extra options, including the convenient --summary option, and a verbosity parameter which controls which categories of commits are displayed:

$ git icing -h
usage: git icing [options] [git cherry args]

    -v, --verbosity [N]              Set verbosity level
    -s, --summary                    Show summary

first | verbosity | description of
field |   level   | classification
------+-----------+-----------------------------------------------------
  +   |     1     | not yet upstream
  !   |     1     | still needs to be upstreamed, tracked on a TODO list
  .?  |     1     | upstream *and* blacklisted?!
  !?  |     1     | upstream *and* marked as a TODO?
  .   |     2     | blacklisted - should not be pushed to this upstream
  ?   |     2     | not yet upstream, with unparseable note
  -   |     3     | already upstream
  #   |     3     | already upstream with an annotation note

(When you run that on a colour-capable terminal, you’ll also see that the classifications are also nicely colour-coded so that commits which require attention naturally capture your eye more than those which don’t.)

Let’s see all commits (basically a colour-coded version of what git cherry would show us), plus a summary of the status quo:

$ git icing -v3 --summary stable/grizzly to-backport
+ 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
+ f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list
+ dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb Enable tox use of site-packages for libvirt.
- 76844433f69a7c29ed4566ad34d7e9740feaf660 Add caching for ec2 mapping ids.
+ def5fa1e0e1c0426d973e4d8e3935c6eb18698dc Set version to 2013.2
+ 3aa80834e6aab2456f0f5229a63c2dcef007cb36 Change type of ssh_port option from Str to Int
+ 26aa01094a79939320d58f2fe2d5731f169987b1 Change arguments to volume_detach()
+ b52a2157bedba693c5da7dcb783b7a151769d6b1 Add placeholder migrations to allow backports.
+ 8d5159dd6dd36d6f3d1aadc1123574c5b0aa0614 Don't actually connect to libvirtd in unit tests.
+ 8cd5f862ac071023d5a7984f07513e5aa91f7d3d Bring back sexy colorized test results.
- 45e65d8c0301da689de1afcbc9f45756e71097ab Update instance network info cache to include vif_type.
- b8f9815c5ab61466009c0447f54abc4c309e4e3f nova-manage vm list fails looking 'instance_type'
+ c199fd6fa39311305b5b5d94c2e3732b826a6414 Remove outdated try except block in ec2 code
- 03a2463be8034ee4764ac97e8020fc3d3a32f1fd Make nova.virt.fake.FakeDriver useable in integration testing
- 67628c56caf9d84588a92448880ecdb33eea08b4 Fixes passing arbitrary conductor_api argument
+ e136de1aea8d1469465272942fa3d2769cbe3a80 xenapi: fix console for rescued instance
+ 5df18f6b915cf49ac8559274a631f1366eab00fd Add a comment to placeholder migrations.
- 9075069098e32b47bd5011e2653a23b61c18d4a3 Initialize compute manager before loading driver.
- 3801a4d2f4c59dbfda49131ddde22fcb3976d651 translate cinder BadRequest exception
- 132a0c1fd1eb127e393e5794ffa6d4a3a4950567 Don't log traceback on rpc timeout.
+ 81204d4020712cf5b5b368b75d934d637b2c001b Sync rpc from oslo-incubator.
+ f665d798234c19cccc148a178e94c3717ba8bd6e Sync everything from oslo-incubator.
- 12f7d6332a731421d6e2a190ac60828c1cf98910 Reset ec2 image cache between S3 tests.
+ 7a9ce26aaf03459a7ea367fbdb91d91626a1ee93 py2.6 doesn't support TextTestRunner resultclass
- 5908b60b0420f1ad528e56b0c147a330e9a1a5d6 Make _downsize_quota_delta() use stashed instance types
+ 91b9c208f0c6e21759c7828aa42b3216853be425 disable colorizer as it swallows fails
+ df0560f0353bc0837cc62cfaa9029d21d45c529e Imported Translations from Transifex
+ d6a9d38a0b7488dd77d3a01ae9c4171c271d7314 set up FakeLogger for root logger.

Summary
=======

11 commits processed:
    11 already upstream

18 commits remaining:
    18 not yet upstream

Progress: 11 / 29 commits (37%)

OK, not bad! We didn’t even do any real work yet, we’re already 37% of the way through. That’s because some of the commits (the ones prefixed by -) have already been cherry-picked into stable/grizzly. Let’s prove this for the first commit prefixed by - in the above output, which was 775e3b02:

$ git log -n1 775e3b02
commit 775e3b02b2afbf101db22b87a1c3b189d68532e1
Author: Kieran Spear <kispear@CENSORED.com>
Date:   Mon Mar 18 17:32:26 2013 +1100

    Pass project id in quantum driver secgroup list

    The quantum driver is always returning security groups from every tenant
    a user has access to, even when the "project" filter is supplied.

    Make sure we pass along the project value when we call
    quantum.list_security_groups() so it's properly filtered.

    Fixes bug 1155381.

    Change-Id: I682c66a1f3f9db18b5f9924a37b45c759ff259f7

This commit appears to be only on master, not on stable/grizzly:

$ git branch --all --contains 775e3b02
  master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master

Let’s try to find the commit which cherry-picked it into stable/grizzly:

$ git log --format=fuller --no-merges --grep='Pass project id' stable/grizzly
commit 989435822e5841bc1355e75bbdb003b10a3baf58
Author:     Kieran Spear <kispear@CENSORED.com>
AuthorDate: Mon Mar 18 17:32:26 2013 +1100
Commit:     Chuck Short <chuck.short@CENSORED.com>
CommitDate: Tue Mar 26 10:09:40 2013 -0500

    Pass project id in quantum driver secgroup list

    The quantum driver is always returning security groups from every tenant
    a user has access to, even when the "project" filter is supplied.

    Make sure we pass along the project value when we call
    quantum.list_security_groups() so it's properly filtered.

    Fixes bug 1155381.

    Change-Id: I682c66a1f3f9db18b5f9924a37b45c759ff259f7
    (cherry picked from commit 775e3b02b2afbf101db22b87a1c3b189d68532e1)
$ git branch --all --contains 9894358
* stable/grizzly
  remotes/origin/stable/grizzly

There it is! Notice that it was committed 6 days after RC1 was tagged, but it has the same patch-id as the original, even though the commit message/date and committer are all different:

$ git patch-id < <(git show 9894358)
42dd410cf36ad1c16ceb1bb7f6ed6e3f01cc28bf 989435822e5841bc1355e75bbdb003b10a3baf58
$ git patch-id < <(git show 775e3b02)
42dd410cf36ad1c16ceb1bb7f6ed6e3f01cc28bf 775e3b02b2afbf101db22b87a1c3b189d68532e1

Now we can actually start porting the remaining commits. git-cherry-menu provides an interactive interface to make this process less tedious:

$ git cherry-menu -h
usage: git [options] cherry-menu <command> [<args>...]
suggested options:

[... snipped ...]

<command> <args> are typically "git icing -v3" or "git cherry", but
can be anything which gives output in the same format.  This allows
more control over which commits constitute the upstreaming backlog,
e.g.

    git icing -v2 $upstream $downstream | grep ... > tmpfile
    # Could edit tmpfile here if we want
    git-cherry-menu cat tmpfile

Provides an interactive wrapper around git-icing (or git cherry).  For
each commit provided on STDIN by COMMAND which has not yet been
upstreamed, asks the user whether they want to cherry-pick the commit,
blacklist it, or skip it.  After a successful cherry-pick, the source
commit will be automatically blacklisted if the patch-id changed.

You can quit the process at any time and safely re-run it later - it
will resume from where you left off.

Invoking icing with "-v2" ensures that previously blacklisted /
upstreamed commits are also processed.

Let’s give it a go …

$ git cherry-menu git icing -v3 stable/grizzly to-backport
commit 348cfedfcadc377fa91029b932b2a93d16475822
Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove redundant 'dest' args

    Includes a hack to calculate 'dest' from the argument name. This hack is
    removed in a later commit.

    Change-Id: I60567ff232ab7699f3234b3bfc1618a17a648976

diff --git a/bin/nova-manage b/bin/nova-manage
index 0fde8ba..4919d88 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -127,10 +127,10 @@ def param2id(object_id):
 class VpnCommands(object):
     """Class for managing VPNs."""

-    @args('--project', dest="project_id", metavar='<Project name>',
+    @args('--project', dest='project_id', metavar='<Project name>',

[... snipped ...]

Cherry-pick / blacklist / skip 348cfedfca, or quit ?

It’s showing us the first commit in the porting backlog, and asking us to decide whether to cherry-pick it, blacklist it (i.e. permanently exclude it from the porting process), or defer the decision by skipping it and moving onto the next commit in backlog. (q quits the whole process, of course.)

Let’s try cherry-picking it:

Cherry-pick / blacklist / skip 348cfedfca, or quit ? c

error: could not apply 348cfed... nova-manage: remove redundant 'dest' args
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

Spawning a shell so you can fix; exit the shell when done.
$

Oh dear, this one’s going to need some further thought. Let’s skip it for now:

$ git reset --hard
HEAD is now at ddb676d Merge "Typo: certicates=>certificates in nova.conf"
$ exit
Warning: HEAD did not change; no action taken.
Press enter to continue ... 

-----------------------------------------------------------------------

commit f16534db22d63ce9f762ee6f7a9245126e9f28ed
Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove unused import

    Unused since commit 9ff3121b

    Change-Id: I76bb49669d1cdfc3bf5b1c20087b4bd77420cd91

diff --git a/bin/nova-manage b/bin/nova-manage
index 4919d88..e93c3d8 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -88,7 +88,6 @@ from nova.openstack.common import timeutils
 from nova import quota
 from nova.scheduler import rpcapi as scheduler_rpcapi
 from nova import servicegroup
-from nova import utils
 from nova import version

 CONF = cfg.CONF

Cherry-pick / blacklist / skip f16534db22, or quit ?

cherry-menu presents the next commit in the backlog, which in this case is removing an unused import from bin/nova-manage. Sounds pretty straight-forward, so let’s cherry-pick:

Cherry-pick / blacklist / skip f16534db22, or quit ? c

[stable/grizzly 3f013e4] nova-manage: remove unused import
 Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
 1 file changed, 1 deletion(-)

Looks like this worked fine. The output continues:

-----------------------------------------------------------------------

Already upstream: 775e3b02b2 - Pass project id in quantum driver secgroup list

Here cherry-menu is telling us it automatically skipped over a commit which is already in our target branch. The output continues with the next commit in the backlog, which we can also successfully pick:

commit dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb
Author: Clark Boylan <clark.boylan@CENSORED.com>
Date:   Thu Feb 7 21:39:31 2013 -0800

    Enable tox use of site-packages for libvirt.

    Enable the use of site-packages in tox which will allow the use of the
    system install of libvirt while testing.

    Hardcode the libvirt host UUID for tests that check this UUID when
    system libvirt is being used. Without this hardcoding eight tests
    would fail when using the system libvirt install.

    Partially fixes bug #1113181

    Change-Id: I59c5fbd45639962c0963298203c39759b6ca2d11

[... snipped ...]

Cherry-pick / blacklist / skip dbbbbf4ecb, or quit ? c

[stable/grizzly b4fdf14] Enable tox use of site-packages for libvirt.
 Author: Clark Boylan <clark.boylan@CENSORED.com>
 2 files changed, 8 insertions(+)

-----------------------------------------------------------------------

Already upstream: 76844433f6 - Add caching for ec2 mapping ids.
commit def5fa1e0e1c0426d973e4d8e3935c6eb18698dc
Author: Thierry Carrez <thierry@CENSORED.org>
Date:   Wed Mar 20 16:21:48 2013 +0100

    Set version to 2013.2

    Open Havana development by setting version to 2013.2.

    Change-Id: I37917d28a1f9e0adc2fe3e382412ebc4ed0f3bee

diff --git a/setup.py b/setup.py
index fd968ee..e4bc7d4 100644
--- a/setup.py
+++ b/setup.py
@@ -25,7 +25,7 @@

 setuptools.setup(
       name=project,
-      version=common_setup.get_version(project, '2013.1'),
+      version=common_setup.get_version(project, '2013.2'),
       description='cloud computing fabric controller',
       author='OpenStack',
       author_email='nova@CENSORED.launchpad.net',

Cherry-pick / blacklist / skip def5fa1e0e, or quit ?

Now, obviously we don’t want to backport this version change, so let’s blacklist it in order to permanently excluded it from the porting backlog:

Cherry-pick / blacklist / skip def5fa1e0e, or quit ? b

At this point, your favourite $EDITOR will be launched via an invocation of git notes edit, and you’ll be presented with a text buffer which looks like this:

skip: all
XXX  (you can optionally change the "all" above to the name of the
XXX  upstream branch if you want to limit blacklisting to that upstream)

XXX  Enter your justification for blacklisting here or
XXX  remove the whole note to cancel blacklisting.

#
# Write/edit the notes for the following object:

# commit def5fa1e0e1c0426d973e4d8e3935c6eb18698dc
# Author: Thierry Carrez <thierry@CENSORED.org>
# Date:   Wed Mar 20 16:21:48 2013 +0100
#
#     Set version to 2013.2
#     
#     Open Havana development by setting version to 2013.2.
#     
#     Change-Id: I37917d28a1f9e0adc2fe3e382412ebc4ed0f3bee
#
#  setup.py | 2 +-
#  1 file changed, 1 insertion(+), 1 deletion(-)

At this point you should remove all the lines beginning XXX, and decide whether you want this commit to be blacklisted from any future porting operation. If so, you can leave the skip: all line as is and simply save the buffer and quit the editor, although it’s best practice to also give a sentence or two justifying why you’re blacklisting the commit.

Alternatively you can change all to the name of the branch to which this commit should never be ported, which in this case is stable/grizzly. It is also permitted to use Ruby regular expressions here, by surrounding the regexp with forward slashes, e.g.

skip: /stable/

It never makes sense to port version number changes back to
stable releases.

Once the editor has been quit, we see:

Blacklisted def5fa1e0e
Press enter to continue ...

Behind the scenes, cherry-menu has attached a note to this commit within the refs/notes/upstreaming namespace:

$ git notes --ref=upstreaming show def5fa
skip: all

and you can even see the complete history of your note-editing:

$ git log notes/upstreaming
commit ed6cf5fa75921f23088656b7a8d8a96faa9d769d
Author: Adam Spiers <aspiers@CENSORED.com>
Date:   Fri Sep 20 15:59:04 2013 +0100

    Notes added by 'git notes edit'

commit c1f6ba13e643de4da354375b479470599f92b272
Author: Adam Spiers <aspiers@CENSORED.com>
Date:   Fri Sep 20 15:44:15 2013 +0100

    Notes added by 'git notes add'

[... snipped ...]

After all this hard work, maybe it’s time for lunch. We can safely quit cherry-menu via Control-C or q, and resume it later. Before tucking into a tasty sandwich, let’s quickly review our progress:

$ git icing -v3 -s stable/grizzly to-backport
+ 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
- f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list
- dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb Enable tox use of site-packages for libvirt.
- 76844433f69a7c29ed4566ad34d7e9740feaf660 Add caching for ec2 mapping ids.
. def5fa1e0e1c0426d973e4d8e3935c6eb18698dc Set version to 2013.2
+ 3aa80834e6aab2456f0f5229a63c2dcef007cb36 Change type of ssh_port option from Str to Int
+ 26aa01094a79939320d58f2fe2d5731f169987b1 Change arguments to volume_detach()

[... snipped ...]

Summary
=======

14 commits processed:
    13 already upstream
     1 blacklisted - should not be pushed to this upstream

15 commits remaining:
    15 not yet upstream

Progress: 14 / 29 commits (48%)

Wow, almost half-way! Notice also how the prefix by def5fa1 has changed to a period sign to indicate that it’s now blacklisted.

After lunch, we can resume the process:

$ git cherry-menu git icing -v3 stable/grizzly to-backport
commit 348cfedfcadc377fa91029b932b2a93d16475822
Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove redundant 'dest' args

[... snipped ...]

Cherry-pick / blacklist / skip 348cfedfca, or quit ?

This is the same commit which caused conflicts last time we tried to pick it. Let’s officially postpone resolving conflicts by adding it to a “TODO” blacklist. Press b, then as before, remove the XXX, and also replace the skip: all line with:

TODO: fix conflicts next week

Blacklisted 348cfedfca
Press enter to continue ... 

-----------------------------------------------------------------------

Already upstream: f16534db22 - nova-manage: remove unused import
Already upstream: 775e3b02b2 - Pass project id in quantum driver secgroup list
Already upstream: dbbbbf4ecb - Enable tox use of site-packages for libvirt.
Already upstream: 76844433f6 - Add caching for ec2 mapping ids.
Blacklisted: def5fa1e0e - Set version to 2013.2
commit 3aa80834e6aab2456f0f5229a63c2dcef007cb36
Author: Devananda van der Veen <devananda.vdv@CENSORED.com>
Date:   Wed Mar 20 09:19:44 2013 -0700

    Change type of ssh_port option from Str to Int

    The type of CONF option virtual_power_ssh_port was incorrectly defaulted
    to Str. This can cause Paramiko to raise when casting to %d.

    Fixes bug 1157824.

    Change-Id: I30ddd1ff0da45f8392085249f1bd2a539b201a7e

Now we are taken immediately to where we left off just before lunch, because git-icing automatically detected the cherry-picks we did earlier, and also that we blacklisted the 2013.2 version change commit.

If we run git-icing again, we’ll see that it understands the significance of the TODO attached to the first commit in the backlog:

$ git icing -v3 -s stable/grizzly to-backport                                                        
! 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
- f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list

[... snipped ...]

Summary
=======

14 commits processed:
    13 already upstream
     1 blacklisted - should not be pushed to this upstream

15 commits remaining:
    14 not yet upstream
     1 still needs to be upstreamed, tracked on a TODO list

Progress: 14 / 29 commits (48%)

On subsequent invocations of git cherry-menu, it will offer the option of editing existing notes. There is also an option to automatically skip any notes including the word TODO:

$ git cherry-menu -h
usage: git [<options>] cherry-menu <command> [<args>...]
suggested options:

[... snipped ...]

-c cherry-menu.skip-todos=true
      Skip commits which have notes including 'TODO'.  This allows
      unresolved upstreaming tasks to be tracked via an external
      issue tracker without getting in the way during repeated
      runs of cherry-menu.

[... snipped ...]

So that’s the majority of cherry-menu‘s functionality explained. It’s not rocket science, just a quick hack of a shell-script, but it gets the job done pretty well.

There’s one more important feature I didn’t cover: if the patch-id changes during cherry-picking, cherry-menu will notice, and automatically add the old commit to the blacklist:

Cherry-pick / blacklist / skip 81204d4020, or quit ? c

[stable/grizzly 65cd526] Sync rpc from oslo-incubator.
 Author: Russell Bryant <rbryant@CENSORED.com>
 4 files changed, 47 insertions(+), 21 deletions(-)
The git patch-id changed during cherry-picking.  This is normal when
the diff context changes or merge conflicts are resolved.
Blacklisted 81204d4020 so future runs won't attempt to duplicate the upstreaming.

Maybe our project manager comes over and says “hey, we need those features backported to stable ASAP – I’ve asked Joe and Fred to help you out, OK?”

So far, all the work has happened in a local repository. Of course we can push our updated stable/grizzly branch to a public repository somewhere, so that Joe and Fred can see what we already cherry-picked. But they’d be still missing some valuable meta-data, i.e. the git notes we created which mark which commits to exclude from porting, and why. Unfortunately, git doesn’t yet natively provide a convenient way for pushing/pulling notes between repositories, because the refs/* namespace hasn’t been sufficiently standardized yet. To work around that, I wrote git-rnotes to avoid having to remember a slightly arcane sequence of git commands.

Usage is very simple, and mimics git fetch/push/merge/pull:

Usage: git-rnotes [options] SUBCOMMAND REMOTE
Options:
  -h, --help     Show this help and exit

Subcommands:

  fetch
  push
  merge
  pull

For example, if you have a remote called public:

export GIT_NOTES_REF=refs/notes/upstreaming
git rnotes push public

or if Joe has updated his notes and pushed them to public whilst you also updated yours, simply do a pull, which is equivalent to a fetch followed by a merge:

export GIT_NOTES_REF=refs/notes/upstreaming
git rnotes pull public

Sorry that was so long. But hopefully you agree it’s relatively simple once you try it out. I’d love to know if anyone finds this useful – if you do, please leave a comment below! Of course pull requests against my git-config repository are always welcome. There are many other git-related hacks in that repository which you may also find useful.

Thanks for reading!