However, there’s some good news on this front. Intel and AMD realised this was a problem, and have both introduced new hardware capabilities to help improve the level to which cloud users can trust the environment in which their workloads are executed, e.g.:

• AMD SEV (Secure Encrypted Virtualization) which can encrypt the memory of a running VM with a key which is only accessible to the owner of that VM. This is done on-chip so that even if you have physical access to the machine, it makes it a lot harder to snoop in on the running VM¹.

  It can also provide the guest owner with an attestation which cryptographically proves that the memory was encrypted correctly and can only be decrypted by the owner.

• Intel MKTME (Multi-Key Total Memory Encryption) which is a similar approach.

But even with that hardware support, there is the question to what degree anyone can trust public clouds run on proprietary technology. There is a growing awareness that Free (Libre) / Open Source Software tends to be inherently more secure and trustworthy, since its transparency enables unlimited peer review, and its openness allows anyone to contribute improvements.

And these days, OpenStack is pretty much the undisputed king of the Open Source cloud infrastructure world.

So I’m delighted to be able to announce a significant step forward in trustworthy cloud computing: as of this week, OpenStack is now able to launch VMs with SEV enabled! (Given the appropriate AMD hardware, of course.)

The core functionality is all merged and will be in the imminent Train release. You can read the documentation, and you will also find it mentioned in the Nova Release Notes.

While this is “only” an MVP and far from the end of the journey (see below), it’s an important milestone in a strong partnership between my employer SUSE and AMD. We started work on adding SEV support into OpenStack around a year ago:

This resulted in one of the most in-depth technical specification documentations I’ve ever had to write, plus many months of intense collaboration on the code and several changes in design along the way.

I’d like to thank not only my colleagues at SUSE and AMD for all their work so far, but also many members of the upstream OpenStack community, especially the Nova team. In particular I enjoyed fantastic support from the PTL (Project Technical Lead) Eric Fried, and several developers at Red Hat, which I think speaks volumes to how well the “coopetition” model works in the Open Source world.

The rest of this post gives a quick tour of the implementation via screenshots and brief explanations, and then concludes with what’s planned next.

OpenStack’s Compute service (nova) will automatically detect the presence of the SEV feature on any compute node which is configured to support it. You can optionally configure how many slots are available on the memory controller for encryption keys. One is used for each guest, so this effectively acts as the maximum number of guest VMs which can concurrently use SEV. Here you can see the configuration of this option, and how nova handles the inventory. Note that it also registers an SEV trait on the compute host, so that in the future if the cloud has a mix of hardware offering different guest memory encryption technologies, you’ll be able to choose which one you want for any given guest, if you need to.

SEV can be enabled by the operator by adding a new hw:mem_encryption “extra spec” which is a property on nova’s flavors. As already shown in the screenshot above, this can be done through Horizon, OpenStack’s web dashboard. However it can also be set per-image via a similarly-named property hw_mem_encryption:

and of course this can all be done via the command-line too:

Notice the presence of a few other image properties which are crucial for SEV to function correctly. (These are explained fully in the documentation.)

Once booted, an SEV VM instance looks and behaves pretty much like any other OpenStack VM:

However there are some limitations, e.g. it cannot yet be live-migrated or suspended:

Behind the scenes, nova takes care of quite a few important details in how the VM is configured in libvirt. Firstly it performs sanity checks on the flavor and image properties. Then it adds a crucial new <launchSecurity> element:

and also enables IOMMU for virtio devices:

This area of technology is new and rapidly evolving, so there is still plenty of work left to be done, especially on the software side.

Of course we’ll be adding this functionality to SUSE OpenStack Cloud, initially as a technical preview for our customers to try out.

Probably the most important feature needed next on the SEV side is the ability to verify the attestation which cryptographically proves that the memory was encrypted correctly and can only be decrypted by the owner. In addition specification of the work required to add support to OpenStack for Intel’s MKTME already started, so I would expect that to continue.

So I’ve built an app called “the SHIMANator” which can convert any music into its negative harmony equivalent, and I’m very excited to finally announce it to the world! Check out the video:

I mainly wrote this app because a) Jacob asked me to, b) it sounded like a fun challenge, and c) the thought of being able to instantly hear the negative harmony equivalent of any music was very appealing.

But in the process of getting it working, it became apparent that this could actually be a useful tool for generating fresh new musical ideas and sounds. For example, I’ve talked to film / TV composers who got excited about using it to quickly generate music which is coherent with and relating to their existing material. For example if a musical motif in a major key represented a character in a film, flipping it about an axis would give you the negative version, which would sound in a minor key and could be used to represent the “dark side” of that character’s personality.

Not yet, but I’m aiming to offer early beta access to a select group of people at some point soon. Please show your interest by taking the following actions!

Do you find this app interesting? Could you imagine using it yourself? If so, would you use it for composition, or as an extra effect in live improvisation, or maybe even for something else?

So I’d love to hear what you think – and the person who gives the best feedback will win a free copy of the software when it’s fully productised and ready to publish!

Please leave comments on the Facebook page or the YouTube video.

At the simplest level, it takes MIDI events in, does some magic to convert to negative harmony, and then outputs the same events with modified pitches. So it should work with pretty much any piece of MIDI-compatible software or hardware under the sun.

The actual algorithm for the conversion is very complicated, so I’ll save the explanation for another time. But you can see from the video how melodic motion is inverted, as is motion around the circle of fifths.

Here’s the history, in case you’re curious.

I was introduced to negative harmony a few years ago by Barak Schmool, who later taught Jacob in his role as professor of jazz and world rhythms at the Royal Academy of Music. (Barak’s awareness of the technique was heavily influenced by his friend Steve Coleman, an incredible jazz saxophonist who has been using negative harmony in his music for many decades.)

At the time I built a really dumb prototype for fun, and fed Mozart’s 40th Symphony through it. The results were promising, but also sounded amusingly terrible due to every high note being converted to a very low note and vice-versa.

It was clear that without some magic octave transposition algorithm, notes would stray too far from their original register and completely screw up the voicing “texture” which the composer had intended. So shortly afterwards, Barak and I basically forgot about it, and instead got distracted experimenting on Giant Steps with some brilliant ideas he had regarding unequally tempered systems of intonation (which later inspired some of Jacob’s great work with microtonal voice-leading).

A year or two later, Barak told Jacob about my prototype. He was interested enough to get in touch, and you already know the rest of the story.

By the way, there is already some great music out there made entirely using negative harmony. For example see Steve Cruickshank’s fantastic YouTube channel, which is full of negative harmony covers of famous music. And I’m not even the first to come up with a negative version of Beethoven’s Moonlight Sonata. But I’m not yet aware of any other software which does what mine does. Hopefully you find its potential interesting.

Funny you should ask! Actually yes: in an earlier collaboration with Barak I built the Scale Matcher – a free app (web page) for finding which scales match a given chord. Check it out

If you read this far, congratulations – you are a most excellent and dedicated music theory nerd. Let’s have a pint some time. But until then, don’t forget to subscribe and let me know what you think!

The first pair are available as patches to git itself:

• git-splice – splices commits onto/into/out of current branch

  Jump to 08:44 in the video to see me introduce and demo this, or look at the slides, or read the man page for example usage.

• git-transplant – transplants commits from current branch onto/into another

  Jump to 32:22 in the video to see me introduce and demo this, or look at the slides, or read the man page for example usage.

The second pair are available as Python modules on PyPI, but both have a command-line interface too:

• git-deps – automatically detects textual dependencies between commits

  Jump to 41:28 in the video to see me introduce and demo this, or look at the slides, or read the README (which has a couple of short YouTube videos demoing the tool including the fancy web interface), or read my previous blog post about this.

• git-explode – explodes a linear sequence of git commits into new independent topic branches

  Jump to 51:10 in the video to see me introduce and demo this, or look at the slides, or read the README.

It’s taken a while to get them to a reasonable degree of polish, during which this work has been mostly under the radar, but now I’m now going to make a more concerted effort to advertise the tools, hence this blog post.

(Jump to 02:27 in the video to see me talk about this, or look at the slides.)

All developers face common problems:

• It’s difficult to predict what you’ll be doing next. One minute you might be working on a feature, the next you find a bug you need to fix, and the next you find some technical debt which requires refactoring.
• Multi-tasking is almost unavoidable.
• We’re human, so we sometimes screw up when trying to follow best practice workflow.

What does this mean when using git? Typically it means we are constantly juggling branches, and often need to manipulate the history of branches for various reasons, for example:

• Maintaining clean separation of topics between branches to facilitate code reviews: one branch for a refactoring, one for a bugfix, one for a feature, and so on.
• Backporting or forward-porting series of patches between branches.

My feeling is that while the git ecosystem has continued to evolve incrementally over the last few years, not much progress has been made in terms of automating this janitorial work, and that most people either do it by hand, or don’t bother doing it at all. This leads to unclean history, and/or more work and frustration for reviewers.

So I wrote these tools (and others) in an attempt to push the boundaries of automating git workflows to higher levels, as I explained on the git mailing list last year. Even if all the use cases they can cover have yet to be discovered, I hope you find them useful, or at least thought-provoking enough that you might join the effort!

I’d love some feedback, so do let me know what you think in the comments below.

The PTG was well-attended with just over 400 people from all over the world. As with previous PTGs this felt like just the right size: enough to have most projects / organizations well represented, but not too big to prevent serendipitous meetings happening many times throughout the week, where I would just be wandering around and bump into exactly the right person I needed to talk to 🙂

All the etherpads are publicly available in case you want to dive into the gory details.

IMHO the ptgbot continued to work really well for adapting to the dynamic and fast-changing agenda. I submitted a few patches to polish it further.

The “extreme” (for Ireland, anyway) weather, which was reportedly the heaviest snow in 40 years, caused a lot of disruption, not just for travel, but also that our conference venue in Croke Park Stadium was forced to shut on Thursday afternoon, rather than Friday evening as originally planned. Luckily our amazing hotel, The Croke Park, stepped up to the plate and allowed us to continue the conference there. It was quite a squash fitting everyone in, but somehow they managed it, and the ensuing organized mayhem had the unexpected side effect of building really strong comradery between everyone.

There was a piano-bar style sing-along night (with OpenStack’s Executive Director Jonathan Bryce doing a great job on the piano), a board games night … I think it was my colleague Keith Berger who shrewdly observed that it was like being on a cruise ship 🙂 Having the best Guinness in the world on tap probably helped too. Check out #SnowpenStack on Twitter for some of the hilarious antics.

We managed a couple of SUSE team dinners and it was great to spend more time with teammates, especially the ones who are normally on a different continent 🙂

• https://etherpad.openstack.org/p/self-healing-ptg-rocky

My main reason for attending the PTG was bootstrapping this new SIG which I founded a few months ago. We had a really good session filling Tuesday afternoon, with about 30 people attending.

There were discussions about various use cases, and some people are already keen to start work on use cases immediately. We also had representatives from the API SIG, including Graham Hayes a.k.a. mugsie who is leading the new health-check API initiative, and from OPNFV, since we are trying to promote better sharing of work on automated testing of self-healing / HA functionality. (If you’re interested, please sign up here!)

We agreed some concrete next steps, some of which I already started working on:

• Populate our self-healing-sig repo with the standard Sphinx stuff and templates for use cases and specs
• Compile a list of contact info for self-healing stakeholders across the various projects / organizations
• Start to visualize the architectural integration points between involved projects.
• Start to document use cases and write specs for proposed work.

You can follow all activity via the StoryBoard project.

The only disappointment with this session was that no operators attended. However I managed to coordinate with some people involved with the Tokyo Ops meetup this week, in order to gather feedback from ops, and we got a session slot booked in that agenda which happened yesterday. Additionally I am speaking on self-healing and leading a discussion session on it at the London OpenStack Meetup next Monday, so hopefully we’ll get some good feedback there too.

• https://etherpad.openstack.org/p/FC_SIG_Rocky_PTG

I joined the contributor guide part of this session briefly to give some feedback and also organise collaboration on writing a quickstart guide for people who are used to GitHub and don’t know Gerrit, since I promised to write that after creating a comparison matrix in Denver, which provides a base for the guide.

The Self-healing SIG previously agreed to use StoryBoard for tracking work, not just to aid collaboration within the SIG, but also to help dogfood StoryBoard and provide feedback. So I spent quite a bit of time with the StoryBoard folks giving feedback and brainstorming ideas for improving usability. I also started work on a “Rosetta stone” matrix which aims to show the key differences between StoryBoard and other issue trackers people are used to, such as JIRA, Trello, GitHub, Bugzilla, Redmine etc. Similarly to the approach with the GitHub / Gerrit matrix above, it could serve as the basis for a document which helps “port” people to StoryBoard from other issue trackers.

• https://etherpad.openstack.org/p/ffu-ptg-rocky

This was a really full session, with some good discussions. Main points:

• It was agreed to form an Upgrade SIG, and this has now happened.
• AFAICS everyone is not yet on the same page about what FFU means exactly, especially regarding online migrations. There is perhaps room for improvement in the governance tags provided for tracking upgradability (assert:supports-upgrade, assert:supports-accessible-upgrade), which would help clarify this.
• IIUC Kolla seem to be going in a slightly different direction based on online rolling upgrades, although I don’t think they necessarily have any strong objections to supporting offline upgrades so maybe things will converge in the long term.
• Noone from openstack-helm present 🙁
• Lots of gotchas, and consequent debates on how best to capture and share these.

• https://etherpad.openstack.org/p/oslo-ptg-rocky

I only had time to pop into the oslo room briefly, to discuss an interesting new initiative in oslo.config to automate migration of config between releases (a bit like DB migrations, but for oslo.config instead of DB schema). This was also mentioned in the FFU session.

• gerrit#I851628c0 (Propose spec for handle config changes in code)
• gerrit#I0a14a4a4 (Handle config mapping changes)
• gerrit#I0b95d3ea ({WIP}{POC} Handle config mapping changes)

I posed some questions about how this would handle config which requires extra per-project logic to migrate, and also how it would handle config which has been explicitly set to the default value. We had some interesting discussions on that but I think ultimately the answer was “let’s cross that bridge when we come to it”.

I also had a quick chat with Ken Giusti about a long-standing oslo.service bug we’ve worked on, but we ran out of time and agreed to take it online.

I demo’d ansible-viz, the Ansible visualization tool of which I recently became the maintainer, to the OpenStack Ansible folks. They seemed really enthusiastic, and are looking into using it for embedding diagrams in their docs. Hopefully this means they will start contributing to make it better!

My colleague Dirk Müller and I also talked to them in general about other areas for potential collaboration. We were both impressed with how open they are to help and collaborate!

As usual with the PTGs, I got huge value from random unplanned conversations in hallways, at lunch tables, in the bar etc.

I spent a lot of time talking to Emmet Hickory (a.k.a. persia) and he gave me some great ideas about new use cases for the git workflow tools I have been building in hackweeks and spare time. I already knew that my git-deps tool allows fairly accurate predictions of how difficult it will be to backport a given commit to a stable branch, but a new idea emerged that this could be added as a non-voting job to Zuul‘s checks pipeline, to facilitate a more proactive / preemptive approach to backporting.

I talked with Tony Breeds (PTL of stable branch maintenance) about this who told me that Neutron is already extremely proactive when it comes to backporting, so they might be interested in this. So I met Miguel Lavalle (Neutron PTL) and he was really interested and requested that I put it on the agenda for a future Neutron IRC meeting. This has now happened and we will discuss during next Tuesday’s meeting.

Emmet helped me realise that my git-explode tool could also help with backporting work, so I need to finish packaging that and publish it. There is a London Git meetup coming up soon, and I might present the tools at that.

I’m getting off-topic now so I’ll finish here 🙂 If you found this report useful please leave a comment below, so that I know it’s worth bothering to do next time 🙂

But then we realised that managing OpenStack itself was a pain, and someone had the idea that rather than building a separate management layer for managing OpenStack, we could just use OpenStack to manage itself! And so OpenStack on OpenStack, or Triple-O as it’s now known, was born.

Within and alongside OpenStack, several other new exciting trends emerged: Software-Defined Networking (SDN), Software-Defined Storage (e.g. Ceph), etc. So the umbrella term Software-Defined Infrastructure was coined to refer to this group of abstraction layers.

Whilst OpenStack was busy growing up and moving past the Peak of Inflated Expectations, all of a sudden Docker and containers burst onto the scene and provided a lot of new buzzwords to get everyone excited again. But after the excitement started to fade, that familiar sinking feeling came back with the realisation that just like VMs, containers need something to manage them.

But then Kubernetes leapt in to the rescue! And all the excitement returned. Except that of course then you need something to manage Kubernetes, but fortunately we already had OpenStack, so we could just use that! And so Magnum was born. And since Kubernetes is so awesome, we realised that we could also use it as the basis for deploying OpenStack. At the recent OpenStack Summit in Sydney, we saw the continued rise in popularity of running both Kubernetes on OpenStack, and OpenStack on Kubernetes.

But that still leaves the pesky job of managing raw hardware to put all this stuff on top. Fortunately there are services you can pay for so that other people do that for you, and they even have APIs you can hook into! This is called public cloud. And even better, many companies use OpenStack to drive their public clouds.

So now we’re running Kubernetes on OpenStack on Kubernetes on OpenStack. And now we’re done! Right?

Well … if we’re to learn anything from this history, it should be that we’ll always find more good reasons for new abstraction layers. I mean, there’s already a huge amount of work going into things like Cloud Foundry on the PaaS layer, NFV in the telco space, serverless computing, … And I hear that Kubernetes is a great platform for running Cloud Foundry, just like it is for OpenStack. And wouldn’t OpenStack be a cool platform to provide inside Cloud Foundry, e.g. for people who just want to quickly try it out? So who knows, maybe in the next few years we’ll have OpenStack on Cloud Foundry on Kubernetes on OpenStack on Kubernetes on OpenStack.

Of course this starts getting a bit unwieldly. Every time we introduce a new abstraction layer there’s extra complexity to deal with. But that’s OK, because we can always deal with complexity by abstracting it away! It’s a bit like the cyber-equivalent of delegating difficult tasks to someone else. So I’d like to propose a new concept, and corresponding meta-component of the overall architecture:

Whenever we realise we need a new abstraction layer, rather than having to deal with the complexity of deploying and managing this layer, we could just invoke APIs to a central service which takes care of this complexity for us. We could call this new concept (drum roll, please…) Abstraction As A Service, or AaaS. (If you’re British or Australian you may prefer to pronounce this as if there was an “r” in the middle, to distinguish from similar-sounding existing words such as “as”. Or maybe we should call it Sweeping Stuff Under The Carpet As A Service? Or SSUtCaaS for short, which can be pronounced “suitcase” (thanks to Florian for pointing this out).

With AaaS, if we wanted say, NFV on serverless on CF on Kubernetes on OpenStack on Kubernetes on OpenStack on public cloud on COBOL, we could simply write some declarative YAML or JSON describing the stack we want, push it to the AaaS REST API endpoint via an HTTP POST, and it would set the whole thing up for us automatically. We could build any number and combination of abstraction layers we would possibly need, so at this point the job could be considered well and truly done, once and for all!

Except, ya know, we’d need a way to deploy and manage our AaaS service, of course. Maybe we could build an AaaSaaS service for that …

P.S. I know, it’s a terrible joke if you have to explain it, but based on previous experiences of my dry British humour being misunderstood (especially given the inevitably international audience), I feel the need to point out that this blog post was intended as nothing more than poking a bit of gentle fun at the cloud software industry. I’m a huge fan of all the technologies mentioned here, and yes, I’m even in favour of multiple abstraction layers, despite occasionally wondering if we’ve all gone a bit insane 😉 Thanks to Florian, Andrew, and Dirk for reviewing an earlier draft, but I take responsibility for any mistakes or any offence unintentionally caused!

To address this, I organised an unofficial kick-off meeting at the PTG in Denver, at which it became clear that there was sufficient interest in this idea from many of the above projects in order to create a new “Self-healing” SIG. However, there were still open questions:

1. What exactly should be the scope of the SIG? Should it be for developers and operators, or also end users?
2. What should the name be? Is “self-healing” good enough, or should it also include, say, non-failure scenarios like optimization?

In an attempt to answer these, I formally proposed the creation of the SIG, asking the community to fill in a short survey to vote on its creation, and to provide their feedback regarding the name and scope. Unfortunately whilst everyone unanimously supported its creation, opinions were split more or less 50%-50% on the name and the scope! So on advice from Thierry, I listed the SIG as “forming”, created the corresponding wiki page, and proposed a session for the Sydney Forum, which was subsequently accepted.

We had around 30 people attend the Sydney Forum session, which was extremely encouraging! You can read more details in the etherpad, but here is the quick summary …

Most importantly, we resolved the naming and scoping issues, concluding that to avoid biting off too much in one go, it was better to be pragmatic and start small:

• Initially focus on cloud infrastructure, and not worry too much about the user-facing impact of failures yet; we can add that concern whenever it makes sense (which is particularly relevant for telcos / NFV).
• Not worry too much about optimization initially; Watcher is possibly the only project focusing on this right now, and again we can expand to include optimization any time we want.

So now that the naming and scoping issues are resolved, I am excited to announce that the Self-healing SIG is officially formed!

Discussion went beyond mere administravia, however:

• We collected a few initial use cases.
• We informally decided the governance of the SIG. I asked if anyone else would like to assume leadership, but noone seemed keen, dashing my hopes of avoiding extra work 😉 But Eric Kao, PTL of Congress, generously offered to act as co-chair.
• We discussed health check APIs, which were mentioned in at least 2 or 3 other Forum sessions this time round.
• We agreed that we wanted an IRC channel, and that it could host bi-weekly meetings. However as usual there was no clean solution to choosing a time which would suit everyone ;-/ I’ll try to figure out what to do about this!

You are warmly invited to join, if this topic interests you:

• Ensure you are subscribed to the openstack-sigs mailing list, and watch out for posts tagged [self-healing].
• Bookmark the wiki page which is the SIG’s official home.

I have sent out a similar announcement to the mailing list, and next will set up the IRC channel, and see if we can make progress on agreeing times for regular IRC meetings.

Other than this administravia, it is of course up to the community to decide in which direction the SIG should go, but my suggestions are:

• Continue to collect use cases. It makes sense to have a very lightweight process for this (at least, initially), so Eric has created a Google Doc and populated it with a suggested template and a first example. Feel free to add your own based on this template.
• Collect links to any existing documentation or other resources which describe how existing services can be combined. This awesome talk on Advanced Fault Management with Vitrage and Mistral is a perfect example, and here is another, but we need to make it easier for operators to understand which combinations like this are possible, and easier for them to be set up.
• Finish the architecture diagram drafted in Denver.
• At a higher level, we could document reference stacks which address multiple self-healing cases.
• Talk more with the OPNFV community to find out what capabilities they have which could be reused within non-NFV OpenStack clouds.
• Perform gaps analysis on the use cases, and liase with specific projects to drive development in directions which can address those gaps.

In case you’re interested in the history …

I first became aware of the need for this SIG while working upstream within the community on OpenStack HA – specifically on compute plane HA, where failures of compute nodes or hypervisors are automatically handled by resurrecting affects VMs on other compute nodes. I saw many groups independently trying to solve the same problem, so I created the #openstack-ha IRC channel, organised weekly meetings, and tried to bring all stakeholders together to converge on a single upstream solution. Progress was gradually made, which we presented in Austin, Boston, and most recently in Tel Aviv at OpenStack Day Israel 2017.

After the talk I had a great conversation with Ifat Afek, who is the PTL of OpenStack Vitrage, which is an awesome project providing RCA (Root Cause Analysis) of faults within OpenStack. Since Vitrage can do things like receive an alert about a fault on a compute node (e.g. from Aodh) and then automatically determine all affected VMs and call out to another service like Mistral to enact appropriate remediation, there was obvious synergy between our work.

However Vitrage goes much further than just compute HA: since it can receive various alerts from multiple types of data source, model relationships between many types of resource, and trigger external services to take action, this kind of combination has tremendous potential for building automatically self-healing cloud infrastructure. And as shown above, there are several other OpenStack projects operating in the same space, which could take this approach further; for example, Congress can be used to specify policies regarding how failures should be handled.

Talking with Ifat resulted in the idea to create a new SIG with the goals of identifying self-healing use cases, establishing and documenting what can already be achieved by combining existing OpenStack services, and enhancing collaboration between the projects and with operators to fill in any remaining gaps. And now you know the rest of the story 🙂

So what are the underlying problems which made this such a frequently requested feature? From reading the various links above, it seems that by far the biggest motivator is that people frequently submit pull requests (or merge requests, in GitLab-speak) which contain multiple commits, and these commits are seen as too “noisy” / fine-grained. In other words there is a desire to not pollute the target/trunk branch (e.g. master) with these fine-grained commits, and instead only have larger, less fine-grained commits merged.

But where does this desire come from? Well, if the fine-grained commits which accumulate on a PR branch are frequently amendments to earlier commits in the same PR (like “oops, fix typo I just made” or “oops, fix bug I just introduced”) then this desire is entirely understandable, because noone wants to see that kind of mess on master. However the real problem here is that that kind of mess should have never made it onto GitHub in the first place – not even onto a PR branch! It should have instead been fixed in the developer’s local repository. That is why there is a whole section in the “Pro Git” book dedicated to explaining how to rewrite local history, and why git-commit(1) and git-rebase(1) have native support for creating and squashing “fixup” commits into commits which they fix.

If an existing PR needs to be amended, make the change and then rewrite local history so that it’s clean. The new version of the branch can then be force-pushed to GitHub via git push -f, which is an operation GitHub understands and in many situations handles reasonably gracefully. I have previously blogged about why this way is better, but one way of quickly summarising it is: don’t wash your dirty linen in public any more than you have to.

In other words, if you do your history rewriting locally, there will be a substantial increase in the quality of what you publish to the PR, because you will only ever publish commits which (at the time) you believed to be the final candidate, ready for merging. (And later, when you benefit from the hindsight of peer reviews and/or your own realisations about how to make things better, you simply force-push a new final candidate.) Consequently, people reviewing your PR don’t have to waste time reviewing anything unless it was a candidate for merging, and nor do CI systems.

Granted, there are some exceptions to this approach, where you really do want to publish a PR which is still work in progress in order to get early feedback. In that case you can explicitly state in the PR description (or via a label) that this PR is WIP so that anyone reviewing it will know to expect some mess, and that’s totally fine.

Unfortunately, the force-push approach is not perfect either: the full history (a.k.a. reflog) of the PR branch is not exposed via GitHub’s UI, nor are there even guarantees that it will be preserved. This is one of GitHub’s serious flaws, which in contrast Gerrit gets right, and services integrating with GitHub such as https://reviewable.io/ and http://gerrithub.io/ can also help fill in the gaps here. The poor man’s workaround is to manually add comments every time you force-push, which explain what changed and how the changes address feedback from reviewers, but this of course requires discipline and extra effort, and does not show the exact changes, so it’s far from ideal.

These drawbacks with GitHub are another reason that some people prefer to append fixup commits to the PR and then squash at merge-time – at least then the history of the PR is visible to all. But this approach only works with PRs which represent a single logical change. If you squash multiple logical changes into a single commit, you lose valuable history. (In fact, this value can extend as far as using the history for literate programming.) So I strongly recommend keeping the GitHub repo configured with the default setting which allows normal merges, so that you don’t force this history loss on merge of PRs.

In response to the previous issue, you could decide a policy where you insist on clean commits and normal merging if the PR contains multiple logical changes, but allow unclean commits and squash merging if the PR contains only a single logical change. However the decision whether to do squash merge or normal merge is left to the merger, and it’s made at merge-time, outside the peer review process. So on a bigger project where many people have merge permissions, that reduces the chance of merges being done in a correct and consistent way.

By the way, I’m not the first person who has highlighted these problems.

OK sorry, that’s enough Star Wars quotes.

But this awkwardness highlights another fundamental issue with GitHub: it does not conveniently support “stacking” one PR on top of another, i.e. where the head of the first is the base of the second, with a dependency modelled between the two. (You can do it, but it’s awkward when the PRs come from different forks, and you also have to change the base branch of the second once the first is merged.)

So the only convenient way to group two dependent commits together is to put them in the same PR. This encourages the creation of PRs with many commits, which suck up more and more review effort each time their history is rewritten, and take longer to merge on average, because controversial and non-controversial commits tend to get mixed up within the same PR, and the former block the latter.

This is another area where Gerrit beats GitHub, since Gerrit natively supports dependencies between reviews, making the need for multiple commits per review redundant. Another advantage of this is that you don’t have to write a separate description for the PR, since all the info the reviewer requires is provided in a single commit message.

Anyway, I digress; back to squash merging. Another issue with it is that the final editing of the squashed commit message is after the review process, not during. If, like me, you work on projects which care deeply about the quality of the commit message, this is a deal-breaker.

Finally, the GitHub web UI doesn’t show when squash merging has been used, which can create confusion.

To wrap up, there is no perfect solution right now. However the following guidelines should help:

• Always adhere to best practice for constructing git commits.
• Try to keep your mess to yourself as much as possible, by rewriting history locally before publishing to or updating a GitHub PR.
• Squash merging will not work well for projects with stricter review policies, or projects which receive PRs comprising of multiple logical changes.
• Squash merging might work OK as a workaround for smaller projects with less stringent review policies, where PRs only ever consist of a single logical change which can reasonably be squashed into a single commit without harm.
• Keep PRs as small as possible, and aim for quick, short review cycles.

And in case the right people are listening:

• GitHub needs to track PR history and expose it through the UI.
• GitHub needs to support stacking of PRs (i.e. tracking dependencies between PRs, and correctly handling when a dependency gets merged).

It is clear from the diversity of the use cases listed above that VM placement policies are likely to vary greatly across clouds, and sometimes even within a single cloud. OpenStack Compute (nova) has fairly sophisticated scheduling capabilities which can be configured to implement some of the above policies on an incremental basis, i.e. every time a VM instance is started or migrated, the destination VM host can be automatically chosen according to filters and weighted cost functions. However, this approach is somewhat limited for migration, because the placement policies are only considered one migration at a time. Like in Tetris, more efficient arrangements can be achieved by thinking further ahead!

OpenStack clouds can already be segregated in various ways via cells, regions, availability zones, host aggregates, and server groups, and so there are various techniques available for implementing different placement policies to different areas of your cloud. For example you could have groups of hosts dedicated to CPU-intensive or I/O-bound workloads using an anti-affinity placement policy, and other groups dedicated to light workloads using an affinity placement policy. But these policies are static in nature (e.g. nova filters and schedulers are configured by the cloud administrator in nova.conf), which somewhat limits their flexibility.

OpenStack is still relatively new, however with Cinder and Neutron rapidly evolving, it is at the point where a VM’s network and storage dependencies can be live migrated along with the workload in a near seamless fashion. So now we should be able to start developing mechanisms for implementing more sophisticated placement policies, where not only is VM rearrangement performed automatically, but the policies themselves can be varied dynamically over time as workload requirements change.

VMware advocates might instinctively recommend running vSphere clusters underneath nova, to harness vSphere’s pre-existing DRS / DPM functionality. However, there are still unresolved problems with this approach, as this article on nova-scheduler and DRS highlights. (Funnily enough, I encountered exactly these same problems a few years ago when I was part of a team adding vSphere integration into the orchestration component of what has since become NetIQ Cloud Manager …) Besides, a vSphere cluster and an OpenStack cloud are very different beasts, and of course not everyone wants to build their cloud exclusively with VMware technology anyway.

Unfortunately developing algorithms to determine optimal placement is distinctly non-trivial. For example, the consolidation scenario above is a complex variant of the bin packing problem, which is NP-hard. The following constraints add significant complexity to the problem:

• A useful real world solution should take into account not only the RAM footprint of the VMs, but also CPU, disk, and network.
• The algorithm needs to ensure that SLAs are maintained whilst any rearrangement takes place.
• If the cloud is too close to full capacity, it may not be possible to rearrange the VMs from their current placement to a more optimal placement without first shutting down some VMs, which could be prohibited by the SLAs.
• Even if the cloud is at low utilization, live migration is not always possible, e.g.
  • the hypervisor may not support it (especially across non-identical CPU architectures)
  • shared storage may be required but unavailable
  • the network may not support it
• Even if the arrangement is achievable purely via a sequence of live migrations, the algorithm must also be sensitive to the performance impact to running workloads when performing multiple live migrations, since live migrations require intensive bursts of network I/O in order to synchronize the VM’s memory contents between the source and target hosts, followed by a momentary freezing of the VM as it flips from the source to the target. This trade-off between optimal resource utilization and service availability means that a sub-optimal final placement may be preferable to an optimal one.
• In the case where the hypervisor is capable of sharing memory pages between VMs (e.g. KSM on KVM), the algorithm should try to place together VMs which are likely to share memory pages (e.g. VMs running the same OS platform, OS version, software libraries, or applications. A research paper published in 2011 demonstrated that VM packing which optimises placement in this fashion can be approximated in polytime, achieving 32% to 50% reduction in servers and a 25% to 57% reduction in memory footprint compared to sharing-oblivious algorithms.

(There is prior art of course. For example, several years ago, Pacemaker implemented a best effort algorithm for VM host selection.)

As noted by the 2011 research paper referenced above, this area of computer science is still evolving. There is one constant however: any rearrangement solution must not only provide a final VM placement optimised according to the chosen constraints, but also a sequence of migrations to it from the current placement. There will often be multiple migration sequences reaching the optimised placement from the current one, and their efficiency can vary widely. In other words, there are two questions which need answering:

1. Given a starting placement A, which is the best (or near-optimal) final placement B to head for?
2. What’s the best way to get from A to B?

The above considerations strongly suggest that the first question is much harder to answer than the second, although the two are not necessarily orthogonal; for example there could be two different final placements B and C which are equally near-optimal, but it may be much harder to reach C from A than B.

I propose that it is worth examining the effectiveness of a divide and conquer approach: solving the second question may simplify the first, and also provide a mechanism for comparatively evaluating the effectiveness of potential answers to the first. Another bonus of this decoupling is that it should be possible for the path-finding algorithm to also discover opportunities for parallelizing live migrations when walking the path, so that the target placement B can be reached more quickly.

I’ve designed an algorithm which solves the second problem, and hacked together a proof of concept implementation in Python. This video shows it solving three randomly generated scenarios:

It’s far from perfect, but as a proof of concept it seems to hold up fairly well under soak testing. I’m pleased to announce that the code is now on github: https://github.com/aspiers/migrationpaths and I would love to hear feedback on it.

If you found this blog interesting, feel free to get in touch! I’m aspiers on FreeNode IRC, and adamspiers on Twitter.

I’m also in Vancouver for the OpenStack summit which starts this Monday, so if you are, come and say hello!

Many thanks to Florian Haas, Steven Hardy, and Christoph Thiel for their input on an earlier draft version of this post, and apologies that it took me SO long to publish it!

A peer review cycle can potentially reveal many issues which make the pull request unready for merging, e.g.

• typos
• bugs in the proposed code changes
• missing features in the proposed code changes
• incomplete test coverage
• incomplete documentation changes
• style inconsistencies (including whitespace issues)
• incorrect or incomplete commit messages
• the commits violate the rule of one logical change per commit
• some changes are outside the scope of the pull request

This is of course what makes peer review of pull requests so valuable: the problems can be addressed even before they hit the master branch, which helps maintain high quality in the development trunk. But then how do we address the issues?

One approach is for the submitter fix the issues in their original local branch, close the pull request, and submit a new one. However in most cases this is a bad idea, because each time you do it, you create an “orphaned” abandoned pull request, and there is no automatic linking between the new revised one and the abandoned one, which makes it a lot harder to follow the history of the review process.

Another approach (the one apparently implied by the GitHub documentation mentioned above) is to append “fixup” commits to the series which fix issues introduced by commits earlier in the series. But that’s almost always a really bad idea, because again it violates the rule of one logical change per commit. In particular:

• It makes life much harder for reviewers, because they have to review a combination of broken commits and subsequent commits which are supposed to fix that brokenness. So for each change, they have follow this process:

  1. Review it as normal.
  2. If they spot brokenness, refrain from immediately commenting on it, and instead look ahead for any subsequent commit in the series which may claim to fix that brokenness.
  3. If such a commit is found, verify that it really does fix the brokenness.
  4. If it does fix it, selectively ignore that brokenness, whilst taking care not to accidentally ignore other brokenness in the same area.

  This problem is significantly compounded each review cycle, i.e. each time fixup commits are appended to the pull request.

  The only workaround is to review via the “Files changed” tab, but that brings considerable disadvantages when the pull request contains more than one commit, because then the reviewer cannot review individual logical changes one at a time.

• After merging the pull request, it pollutes the project’s history with useless noise: fixup commits which are not linked to the commits they are fixing.
• It breaks git bisect, git revert, and git blame.

The good news is that GitHub actually supports a much better alternative which works really nicely. As a maintainer and reviewer of many GitHub repositories, I’ve been continually surprised how many people are unaware of this simple trick; perhaps it’s due to the hole in GitHub’s pull request documentation which I mentioned above? The trick is simply this:

1. Rewrite the history of your local branch corresponding to the pull request, so that the issues revealed during the review of the pull request are addressed.
2. Do a force push (i.e. a git push command which includes the --force-with-lease option, not -f / --force which is dangerous and should never be used).

I won’t go into details here about how to rewrite history. git history rewriting via git rebase (with fixup, squash), git commit --amend etc. is something of an art form, and there are already many great resources out there which there would be no value in me replicating.

So really this whole blog post is a long way of saying:

When you force-push a new head for a pull request branch, GitHub handles it gracefully and automatically updates the pull request in the way you want.

It’s almost like magic. For example, if a reviewer commented on two hunks in the original pull request, and your force-push effectively rewrites one of those hunks but not the other, then the GitHub pull request UI will get updated to say something like “@reviewer commented on an outdated diff” for the rewritten hunk, but any outstanding review comments on the changed hunk will be presented as before the force-push.

Now don’t get me wrong – GitHub is not perfect. In some (but not all) ways, it is inferior to Gerrit’s review system which is better at exposing the complete history of the review cycle. I previously wrote another blog post examining GitHub’s review system in more depth. But in general, it’s usually plenty good enough.

If you’ve previously been fed the mantra “git force-push is evil, don’t ever use it”, or the more specific version “don’t ever force-push to public branches” then hopefully you now realise neither are correct. There are many perfectly valid use cases for force push, otherwise the extremely clever developers who implemented it wouldn’t have bothered (and let’s face it, they know far more about git than you or I ever will). Ultimately the validity / safety of doing a force push to a public branch depends on the audience’s expectations for the stability of that branch. If people expect it to be stable and never get rewritten (only fast-forwarded), then you shouldn’t force push, or at least not without a decent effort to correct those people’s expectations. However there is generally no reason why people should expect GitHub pull request branches to be stable, because people generally don’t base other branches on top of them.

The article entitled “The Dark Side of the Force Push” may help further your understanding.

Note that the title of the previous sections referred to your existing pull requests. GitHub doesn’t let you amend anyone else’s pull requests, and that’s a good thing for obvious reasons. But what if one developer submits a pull request, vanishes off on holiday, and whilst they are away, it gets reviewed and issues are revealed?

In this case it’s easy to effectively take over the process. Simply create a local branch which points to the same commit as the head of the pull request. If you have the totally awesome hub CLI tool installed, then this is as simple as:

git checkout https://github.com/github/hub/pull/134

Then amend your branch as necessary according to the existing reviews, and submit a new pull request which references the old pull request it’s superceding, either by URL or by number. You will probably also want to arrange for the old pull request to be closed with a comment linking to its successor.

Another bias which exists in the western classical tradition for dubious reasons is that everything is bottom-heavy: almost all exercises start at the bottom, go up to the top, and come back down again. This inevitably results in the bottom end being practised more than the top, which is particularly bad because the top end is way harder to play well. So let’s swap it all around!

(By the way, along similar lines I recommend this great blog post on back-to-front learning by Anton Schwarz.)

The next odd thing to note about the classical tradition is that scales are almost always played using a single fingering, and that fingering is typically tonic-biased in another way: the tonic notes are very commonly played using first finger. Perhaps the classical pedagogy evolved this way simply because a single fingering system for each scale was perceived as easier to teach and also for students to memorise. But this approach is hopelessly inconsistent with the real world. If you don’t believe me, take a look at every fragment of a B♭ major scale present in the solo part of Boccherini’s cello concerto in B♭ major, and count for how many of those fragments it would make sense to use the fingering system shown above. If we don’t use those fingering systems when playing actual music, why spend so much time practising them?

So it seems clear to me that practising multiple fingering systems is beneficial for technical reasons, and in the case of jazz for musical reasons too: the improvisatory nature of which means that unless you rely very heavily on licks, you rarely know far in advance what notes you want to play, let alone what fingerings to use. So for any given tonality and position on the instrument, you should practise as many fingering positions as possible, so that when you are in the moment of improvising a solo, whichever hand position you’re in, as many notes are physically accessible from there because you’ve already covered those transitions in practice.

But where do we draw the line? After all there are an enormous number of different possible fingerings for a 3-octave scale, and there’s just not enough time in the day to practise even a fraction of them. Or is there?

Thirdly, it allows freedom to introduce variations on which string is used for a particular group. For example bars 5 and 6 of the above example could be played like this:

I don’t use a strict system to ensure I always cover all possible string variations; instead I like to improvise the choice of string on the spur of the moment, since this matches more closely what can happen when improvising on stage.

Of course this system of 3-note groupings with 3 different starting positions for every scale triples the amount of scales to practise! However since I would normally practise each scale more than once anyway, I aim to seamlessly join up the different offsets in order to cover all combinations without requiring significantly more practice time:

and so on. Actually the hand position “offset” can be changed at any point in the scale, and this is also something I like to improvise on.

So far the examples have all used a major scale. The classical tradition also covers harmonic minor and melodic minor (which is different ascending to descending). In the jazz world (especially for more modern jazz), the ascending form of the melodic minor is particularly important, so I practise that in both directions:

Jazz also uses the harmonic major scale and many other scales, so I mix some of these into my practice, although I find it easier to spend a few weeks/months focusing on one particular scale rather than attempt to cover all scales in every practice routine, which just seems too ambitious. Incidentally, back in 2013 I built the “Scale Matcher”, a web application to show which scales can be used for improvisation in any given chordal context.

Fortunately not all music in the world is written purely in B♭. Unfortunately this means we have to practise other keys too! I try to change key seamlessly without stopping, because this is one of the most fundamental requirements in jazz for improvising over a series of chord changes. Again I like to avoid any tonic bias by changing key on other degrees of the scale. For example, after completing one cycle of all 3 hand positions in B♭ minor, I could shift tonality up a semitone to B minor, changing near the top of the instrument:

or on the lowest note of the instrument:

or even somewhere in the middle:

This is a fun workout for the brain as well as the fingers!

Above I’ve suggested a system for dealing with the cello’s limitation of not being able to play a continuous scale in any key without shifting the hand, which ensures that every possible fingering of a given scale can be methodically covered during practice.

However there’s a completely opposite approach, which is to embrace this limitation and simply fix your hand in a single position and only play the notes of the scale which you can reach from that position, dropping the others! This is particularly useful when improvising at speed, since it drastically reduces the amount of shifting required. I learnt this idea from three of my all-time cello heroes (Erik Friedlander, Mike Block, and Rushad Eggleston), and then extended it to apply to every scale I work on:

I’ve found that improvising within a given “window” can result in some interesting melodic shapes.

Rather than simply ascending or descending, there are an infinite number of patterns to choose from for varying the shapes generated by traversing the scale. Here are a few simple examples:

(The last two were shamelessly stolen from Christian Howes.)

So far I haven’t said anything about rhythm or use of the metronome (which in my opinion is an essential part of scale practice). In fact this topic is so huge I’ll leave it for another blog post at another time! But of course there are an infinite number of ways to vary rhythm, and the western tradition tends not to examine these in much detail. One thing I enjoy doing is freely improvising the rhythm when playing scales, even if everything else (notes, fingerings etc.) is fixed.

How to articulate with the bow is another huge topic, but this post is already way too long!

It’s clear that there are zillions of ways to vary scale practice and keep it interesting whilst challenging yourself in new ways. Perhaps the most important thing for any diligent musician is to constantly invent their own new practice systems, rather than just copying someone else’s verbatim.

If you’ve made it this far, congratulations! This is pretty dry reading, but I hope you found it useful or at least interesting. If you have any feedback of any sort, I’d love to hear it, so please leave a comment!

P.S. For the curious geeks, this blog post was created from a single text file with the combination of some incredible free software: the GNU Emacs editor, GNU Lilypond music typesetter, Org mode, org2blog, and the Org-babel-lilypond backend to Babel.

You’ll need the following:

• At least three physical or virtual 64-bit x86 machines, each with at least 2GB RAM and 8GB disk, and no valuable data on any of the attached disks:

  • one admin node running the Crowbar deployment tool which will provision the other nodes from scratch,
  • at least one controller node which will run OpenStack infrastructure services, and
  • at least one compute node which will host VM instances within the cloud. (If this compute node is a VM, then the VM instances in the cloud will have to be run either using KVM nested virtualization, or QEMU software virtualization which is slower but good enough for “kicking the tires”.)

• A private IPv4 network which all the machines must be connected to. Setting this up is the only potentially tricky bit of the whole exercise. By default the network in question needs to be 192.168.124.0/24 with no DHCP server enabled, so:

  • if you are installing the appliance in a VM, you should be able to set up a NAT or host-only virtual network and configure your hypervisor so that it does not serve DHCP on that network, or
  • if you are installing on bare metal, ensure there is no DHCP server active on that L2 segment.

• Another machine (physical or virtual) with a modern web browser on the same private IPv4 network.

Attach the bootable USB media (physical or virtual), and boot the machine. This will automatically install a SUSE Cloud Admin Node onto the disk. Caution: this will wipe any pre-existing OS, so only use it on a spare machine or freshly-created VM, with no valuable data on any of the attached disks!

After confirming you are OK to wipe all existing data on the disk, the appliance will be written to disk and then booted. Shortly after, YaST will appear, allowing you to configure:

• which keyboard layout you want,
• what password to use for the root user,

• what hostname and domain name to use (the defaults are fine),

• the network setup (the default of 192.168.124.10/24 is recommended, otherwise you will also have to configure Crowbar prior to installation),
• the clock and time zone, and finally
• the NTP configuration (supplying an upstream NTP server is recommended but not required).

Log in as root (with the password you specified above) either on the console or via ssh root@192.168.124.10 if you have another machine configured to be on the same subnet.

Press q and then y to accept the beta EULA, which highlights that the appliance is partially based on unreleased code. Please do not use it for production deployments!

Crowbar is very powerful and flexible in terms of network configuration. If you have other traffic on the L2 network segment (e.g. if you are using bare metal hardware and a physical network rather than a dedicated virtual network) then you should check that its default networks don’t conflict with your existing traffic. To do this, type yast crowbar:

and select the Networks tab:

From here you can examine and change the networks Crowbar will use. Some are only used when various options are selected later on, but at a minimum, admin, nova_fixed, and nova_floating will all be used. For more information, see the Networking section of the SUSE Cloud Deployment Guide.

From the shell prompt, type screen go to initiate the installation of SUSE Cloud. This will take several minutes.

On another machine on the same network as the machine running the now-installed appliance, start a browser, and navigate to http://192.168.124.10:3000/ (adjust the IP accordingly if you changed the admin network above).

Now simply PXE-boot your other nodes. (Typically this requires PXE-booting to be enabled in the BIOS, and/or manually selecting it from the BIOS boot menu which is often accessible by hitting the F11 key or similar during boot.)

This will run a small inventory ramdisk image on each one to detect its hardware and report the discovery back to Crowbar, without touching the node’s local disk(s). Each node will then appear in the Crowbar web UI, and sit in an idle loop whilst awaiting task allocation via Crowbar:

Clicking on that node will show the results of the automatic hardware inventorying, and give us the option to allocate the node:

When editing the node, we can give it a more human-friendly alias (e.g. node1), and then click Allocate to install a minimal SLES OS:

A full OS will be automatically installed on the node via AutoYaST:

When OS installation has finished, the console looks like this:

and then the node turns green in the web UI indicating that it’s ready to have roles assigned to it:

Multiple nodes can be installed via PXE/AutoYaST at the same time.

By this point you should have at least two freshly-installed nodes managed by Crowbar (excluding the admin node itself which Crowbar runs on), in which case you are ready to deploy OpenStack via Crowbar’s barclamps, which can be found via the Barclamps drop-down in Crowbar’s web interface:

This process is relatively straightforward, and a full explanation can be found in the corresponding chapter of the SUSE Cloud 4 Deployment Guide.

However, it is also possible to automate the whole deployment from this point on, using Crowbar’s batch subcommand and an appropriately crafted .yaml file. There are three sample .yaml files in /root on the admin node. The simplest configuration reflected by these three files is simple-cloud.yaml, which assumes a single controller node and a single compute node, with aliases controller1 and compute1 respectively:

In this case, assuming your nodes are given the above aliases, you could set up the entire OpenStack cloud with this single command run as root on the admin node:

crowbar batch --timeout 1800 build simple-cloud.yaml

It takes a while to apply all the barclamps, as can be seen from the timestamps whilst it’s running:

(The other two sample .yaml files are for a highly available control plane which assumes you have three nodes aliased controller1, controller2, and compute1.)

Once the barclamps are all applied, they should show as green in the Crowbar UI view:

From the main Nodes dashboard in the Crowbar web UI, click the controller1 node (or whichever one you deployed OpenStack’s Dashboard to), and you will see a couple of links to the OpenStack Dashboard (a.k.a “Horizon”):

Click on OpenStack Dashboard (admin) and it will take you to the OpenStack Dashboard, where you can log in as the admin user with a password of (by default) crowbar:

Congratulations! You have set up a full OpenStack cloud from scratch! Now you can start reading the SUSE Cloud Admin and End User guides to learn more about how to use OpenStack.

Whilst this bootable appliance is partially based on unreleased, unsupported code, we are still very interested to hear feedback from our customers and partners. So while we (obviously!) cannot offer unlimited free support for it, if you post any questions / issues to the SUSE Cloud web forum, we will try to respond on a best-effort basis. (And of course full commercial support for the released version of SUSE Cloud is available if you want it 😉 )

As we say in the SUSE world, have a lot of fun!

It seems the tireless Mark McLoughlin, one of the leading lights of the OpenStack project, was already thinking along similar lines, and wrote a tool called palaver, which also aims to streamline the porting process, and also uses git notes to persist state pertaining to backporting projects. Upon first glance, the main differences between our tools seem to be as follows:

• git-cherry-menu is a slightly overgrown shell-script (yuk), whereas palaver is Python (yay).
• palaver has a nice CLI with numerous non-interactive sub-commands like palaver list merged, and palaver review which provides an interactive mode. git-cherry-menu only offers an interactive mode (which is very similar in concept to palaver review); for non-interactive querying of the current state, you have to use a separate command git-icing. Personally I prefer palaver‘s unified approach to UI design, even though it doesn’t yet have the pretty ANSI colour-coding of git-icing and git-cherry-menu.
• Since git-icing wraps around git cherry, it can automatically detect some (but not all) commits which have already been ported to the target branch. As the name suggests, git-cherry-menu enables cherry-picking of commits, whereas that’s still on the TODO list for palaver.
• palaver persists state by writing notes which are JSON blobs, whereas git-cherry-menu uses plain text where lines beginning with skip: are treated specially. Not much difference here: obviously JSON allows more sophisticated structuring of metadata, but has the disadvantage of being less readable if you view the notes branch via standard git tools such as git log or gitk.
• palaver “manually” classifies commits into the following states: pass / defer / pick / proposed / merged, whereas my tools have fewer “manual” states – conceptually just skip and todo. merged is either automatically detected via git cherry, or manually flagged via a skip. I’d say my approach is marginally uglier but the automatic detection is quite a big advantage.
• git-rnotes allows merging of someone else’s notes. That’s also on palaver‘s TODO list.

Perhaps the best way forward might be to port the remaining functionality which git-icing and git-cherry-menu offer to palaver, since the latter has a nicer interface, and is pure Python. None of this is rocket science, so hopefully it wouldn’t take too long.

It probably makes sense to keep git-rnotes separate, since it is a very simple shell-script which is useful for sharing any git notes, not just those generated during a commit porting project.

Secondly, it turns out that Gerrit also uses git-notes, in order to annotate each reviewed change with a bunch of very useful metadata, and GitHub is clever enough to expose this metadata in its web UI, e.g.

In Gerrit 2.6 and newer, this functionality was moved out from the core into a reviewnotes plugin. As a result, AFAICS its documentation seems to be buried inside a separate git repository rather than exposed as a nice web page like it used to be. However, it’s very easy to use, and OpenStack’s Gerrit instance certainly has the feature enabled.

For example, simply cd into any OpenStack git repository you happen to have checked out (or clone one first if not). Then type the following:

git fetch origin refs/notes/review:refs/notes/review
export GIT_NOTES_REF=refs/notes/review
git log

You’ll see that each non-merge commit now shows annotations like the ones in the image above, explaining how the commit made its way through the review process before getting merged by Jenkins. Perhaps the most useful piece of data in the note is the link to the relevant review on http://review.openstack.org, so you can quickly navigate to it in your browser and review the er, review.

If you don’t want git log to default to showing the notes, you can omit the export GIT_NOTES_REF, and instead do:

git log --notes=refs/notes/review

If you don’t want to have to remember the slightly cryptic method for fetching notes, my git-rnotes wrapper offers simpler syntax:

git rnotes fetch origin

However, this currently requires GIT_NOTES_REF to have the right value exported.

However, on the occasions when you need to sift through a larger number of commits on one branch, and port them to another branch, complications can arise:

• If cherry-picking a commit results in changes to its patch context, git patch-id will return a different SHA-1, and subsequent invocations of git cherry will incorrectly tell you that you haven’t yet ported that commit.
• If you mess something up in the middle of a git rebase, recovery can be awkward, and git rebase --abort will land you back at square one, undoing a lot of your hard work.
• If the porting process is big enough, it could take days or even weeks, so you need some way of reliably tracking which commits have already been ported and which still need porting. In this case you may well want to adopt a divide-and-conquer approach by sharing out the porting workload between team-mates.
• The more the two branches have diverged, the more likely it is that conflicts will be encountered during cherry-picking.
• There may be commits within the range you are looking at which after reviewing, you decide should be excluded from the port, or at least porting them needs to be postponed to a later point.

It could be argued that all of these problems can be avoided with the right branch and release management workflows, and I don’t want to debate that in this post. However, this is the real world, and sometimes it just happens that you have to deal with a porting task which is less than trivial. Well, that happened to me and my team not so long ago, so I’m here to tell you that I have written and published some tools to solve these problems. If that’s of interest, then read on!

Since git patch-id doesn’t guarantee to return an SHA-1 which is stable enough for us to always depend on, we need some other way of reliably tracking which commits have been upstreamed, and where upstreamed commits came from. It would also be useful to be able to mark selected commits as blacklisted, i.e. permanently excluded from the current porting process, or at least as deferred, i.e. temporarily excluded.

Of course, we could do that simply by maintaining an out-of-band text file somewhere outside the repositories in question, but that doesn’t integrate at all with existing git workflows. Whilst pondering these challenges, I discovered a little-known and seldom-used feature of git, which is the ability to attach arbitrary text as notes on a given commit, via the git notes command. “This sounds perfect!”, I thought, and actually as it turned out, it served as a pretty nice foundation for some higher-level tools which needed building:

• git-icing – an “interface-candy” wrapper around git cherry which adds a splash of colour, and checks for specially formatted notes attached to commits which indicate that those commits are temporarily or permanently excluded from the upstreaming backlog
• git-cherry-menu – an interactive wrapper around git-icing and git-notes which makes it easy to cherry-pick and/or blacklist non-upstreamed commits
• git-rnotes – a wrapper around git notes which makes it easier to share notes to and from remote repositories

Let’s see how this works in practice, by pretending we need to backport a bunch of commits from OpenStack Nova‘s master branch to its stable/grizzly release branch.

$ git clone git@github.com:openstack/nova.git
Cloning into 'nova'...
remote: Counting objects: 208984, done.
remote: Compressing objects: 100% (54766/54766), done.
remote: Total 208984 (delta 167220), reused 188334 (delta 148279)
Receiving objects: 100% (208984/208984), 124.50 MiB | 4.42 MiB/s, done.
Resolving deltas: 100% (167220/167220), done.
$ cd nova

Now, master has changed a lot since grizzly was released, so let’s keep our pretend project to a manageable size by only attempting to backport only commits which appeared in the first few days after the stable branch was established:

$ git describe $( git merge-base origin/stable/grizzly master )
2013.1.rc1
$ git show 2013.1.rc1 | head -n3
tag 2013.1.rc1
Tagger: Thierry Carrez <thierry@CENSORED.org>
Date:   Wed Mar 20 17:13:32 2013 +0100

OK, so grizzly was branched at the point where the first release candidate was tagged. That makes sense. Now let’s choose a point about 5 days later:

$ git rev-list --no-merges --before='25 March 2013' origin/stable/grizzly..master | head -n1
d6a9d38a0b7488dd77d3a01ae9c4171c271d7314
$ git tag to-backport d6a9d38

… and get ready to backport commits from to-backport back into our local stable/grizzly branch:

$ git checkout stable/grizzly
Branch stable/grizzly set up to track remote branch stable/grizzly from origin.
Switched to a new branch 'stable/grizzly'

(If you are following along at home, your rev-list will give a different result, since origin/stable/grizzly will have changed since I wrote this. Don’t worry – just copy the git tag command above verbatim, and create your local grizzly branch via git checkout -b stable/grizzly fc9af8f0a9 so that it points to the same place mind did when writing this article.)

At this point, two quick clarifications are needed, because things can get a bit confusing.

Firstly, if you read the git cherry man page or -h usage text, it is described as a tool to “find commits not merged upstream”, with the expectation that you would cherry-pick the commits it finds onto the upstream branch. But if you happen to be backporting, then the cherry-picking will happen in the opposite direction, so the “upstream” referred to by the man page is actually the real downstream, and vice-versa. Confused yet? If so, wherever you read the word “upstream” in the git cherry documentation / help text, pretend that you actually read “target”. Then just think of “source” and “target” branches, and remember that the process is always to look for commits in the source which are not yet in the target, and then cherry-pick those into the target (or decide that you don’t want to). The same applies for the git icing and cherry-menu help text.

Secondly, normally we’d be porting from one branch to another, not from a tag to a branch. But in this artificially constructed scenario, it wouldn’t make much sense to create a to-backport branch pointing to d6a9d38, because that branch isn’t going to move; the only branch which will be changing is our stable/grizzly branch, as we cherry-pick stuff into it. In other words, the target has to be a branch, because that will change during the cherry-picking, but the source can be a branch / tag / ref / commit SHA-1 … basically anything which git rev-parse understands.

When embarking on a porting project, the first question is usually “how much work is this going to be?” Fortunately git-icing is here to help!

(By the way, if you want to follow along with the below by trying this yourself, simply download the three scripts either from the links above or

git clone https://github.com/aspiers/git-config

and then copy them to any directory on your $PATH, making sure that they are executable.)

git-icing is designed to be invoked in exactly the same way as git cherry, but it has some extra options, including the convenient --summary option, and a verbosity parameter which controls which categories of commits are displayed:

$ git icing -h
usage: git icing [options] [git cherry args]

    -v, --verbosity [N]              Set verbosity level
    -s, --summary                    Show summary

first | verbosity | description of
field |   level   | classification
------+-----------+-----------------------------------------------------
  +   |     1     | not yet upstream
  !   |     1     | still needs to be upstreamed, tracked on a TODO list
  .?  |     1     | upstream *and* blacklisted?!
  !?  |     1     | upstream *and* marked as a TODO?
  .   |     2     | blacklisted - should not be pushed to this upstream
  ?   |     2     | not yet upstream, with unparseable note
  -   |     3     | already upstream
  #   |     3     | already upstream with an annotation note

(When you run that on a colour-capable terminal, you’ll also see that the classifications are also nicely colour-coded so that commits which require attention naturally capture your eye more than those which don’t.)

Let’s see all commits (basically a colour-coded version of what git cherry would show us), plus a summary of the status quo:

$ git icing -v3 --summary stable/grizzly to-backport
+ 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
+ f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list
+ dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb Enable tox use of site-packages for libvirt.
- 76844433f69a7c29ed4566ad34d7e9740feaf660 Add caching for ec2 mapping ids.
+ def5fa1e0e1c0426d973e4d8e3935c6eb18698dc Set version to 2013.2
+ 3aa80834e6aab2456f0f5229a63c2dcef007cb36 Change type of ssh_port option from Str to Int
+ 26aa01094a79939320d58f2fe2d5731f169987b1 Change arguments to volume_detach()
+ b52a2157bedba693c5da7dcb783b7a151769d6b1 Add placeholder migrations to allow backports.
+ 8d5159dd6dd36d6f3d1aadc1123574c5b0aa0614 Don't actually connect to libvirtd in unit tests.
+ 8cd5f862ac071023d5a7984f07513e5aa91f7d3d Bring back sexy colorized test results.
- 45e65d8c0301da689de1afcbc9f45756e71097ab Update instance network info cache to include vif_type.
- b8f9815c5ab61466009c0447f54abc4c309e4e3f nova-manage vm list fails looking 'instance_type'
+ c199fd6fa39311305b5b5d94c2e3732b826a6414 Remove outdated try except block in ec2 code
- 03a2463be8034ee4764ac97e8020fc3d3a32f1fd Make nova.virt.fake.FakeDriver useable in integration testing
- 67628c56caf9d84588a92448880ecdb33eea08b4 Fixes passing arbitrary conductor_api argument
+ e136de1aea8d1469465272942fa3d2769cbe3a80 xenapi: fix console for rescued instance
+ 5df18f6b915cf49ac8559274a631f1366eab00fd Add a comment to placeholder migrations.
- 9075069098e32b47bd5011e2653a23b61c18d4a3 Initialize compute manager before loading driver.
- 3801a4d2f4c59dbfda49131ddde22fcb3976d651 translate cinder BadRequest exception
- 132a0c1fd1eb127e393e5794ffa6d4a3a4950567 Don't log traceback on rpc timeout.
+ 81204d4020712cf5b5b368b75d934d637b2c001b Sync rpc from oslo-incubator.
+ f665d798234c19cccc148a178e94c3717ba8bd6e Sync everything from oslo-incubator.
- 12f7d6332a731421d6e2a190ac60828c1cf98910 Reset ec2 image cache between S3 tests.
+ 7a9ce26aaf03459a7ea367fbdb91d91626a1ee93 py2.6 doesn't support TextTestRunner resultclass
- 5908b60b0420f1ad528e56b0c147a330e9a1a5d6 Make _downsize_quota_delta() use stashed instance types
+ 91b9c208f0c6e21759c7828aa42b3216853be425 disable colorizer as it swallows fails
+ df0560f0353bc0837cc62cfaa9029d21d45c529e Imported Translations from Transifex
+ d6a9d38a0b7488dd77d3a01ae9c4171c271d7314 set up FakeLogger for root logger.

Summary
=======

11 commits processed:
    11 already upstream

18 commits remaining:
    18 not yet upstream

Progress: 11 / 29 commits (37%)

OK, not bad! We didn’t even do any real work yet, we’re already 37% of the way through. That’s because some of the commits (the ones prefixed by -) have already been cherry-picked into stable/grizzly. Let’s prove this for the first commit prefixed by - in the above output, which was 775e3b02:

$ git log -n1 775e3b02
commit 775e3b02b2afbf101db22b87a1c3b189d68532e1
Author: Kieran Spear <kispear@CENSORED.com>
Date:   Mon Mar 18 17:32:26 2013 +1100

    Pass project id in quantum driver secgroup list

    The quantum driver is always returning security groups from every tenant
    a user has access to, even when the "project" filter is supplied.

    Make sure we pass along the project value when we call
    quantum.list_security_groups() so it's properly filtered.

    Fixes bug 1155381.

    Change-Id: I682c66a1f3f9db18b5f9924a37b45c759ff259f7

This commit appears to be only on master, not on stable/grizzly:

$ git branch --all --contains 775e3b02
  master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master

Let’s try to find the commit which cherry-picked it into stable/grizzly:

$ git log --format=fuller --no-merges --grep='Pass project id' stable/grizzly
commit 989435822e5841bc1355e75bbdb003b10a3baf58
Author:     Kieran Spear <kispear@CENSORED.com>
AuthorDate: Mon Mar 18 17:32:26 2013 +1100
Commit:     Chuck Short <chuck.short@CENSORED.com>
CommitDate: Tue Mar 26 10:09:40 2013 -0500

    Pass project id in quantum driver secgroup list

    The quantum driver is always returning security groups from every tenant
    a user has access to, even when the "project" filter is supplied.

    Make sure we pass along the project value when we call
    quantum.list_security_groups() so it's properly filtered.

    Fixes bug 1155381.

    Change-Id: I682c66a1f3f9db18b5f9924a37b45c759ff259f7
    (cherry picked from commit 775e3b02b2afbf101db22b87a1c3b189d68532e1)
$ git branch --all --contains 9894358
* stable/grizzly
  remotes/origin/stable/grizzly

There it is! Notice that it was committed 6 days after RC1 was tagged, but it has the same patch-id as the original, even though the commit message/date and committer are all different:

$ git patch-id < <(git show 9894358)
42dd410cf36ad1c16ceb1bb7f6ed6e3f01cc28bf 989435822e5841bc1355e75bbdb003b10a3baf58
$ git patch-id < <(git show 775e3b02)
42dd410cf36ad1c16ceb1bb7f6ed6e3f01cc28bf 775e3b02b2afbf101db22b87a1c3b189d68532e1

Now we can actually start porting the remaining commits. git-cherry-menu provides an interactive interface to make this process less tedious:

$ git cherry-menu -h
usage: git [options] cherry-menu <command> [<args>...]
suggested options:

[... snipped ...]

<command> <args> are typically "git icing -v3" or "git cherry", but
can be anything which gives output in the same format.  This allows
more control over which commits constitute the upstreaming backlog,
e.g.

    git icing -v2 $upstream $downstream | grep ... > tmpfile
    # Could edit tmpfile here if we want
    git-cherry-menu cat tmpfile

Provides an interactive wrapper around git-icing (or git cherry).  For
each commit provided on STDIN by COMMAND which has not yet been
upstreamed, asks the user whether they want to cherry-pick the commit,
blacklist it, or skip it.  After a successful cherry-pick, the source
commit will be automatically blacklisted if the patch-id changed.

You can quit the process at any time and safely re-run it later - it
will resume from where you left off.

Invoking icing with "-v2" ensures that previously blacklisted /
upstreamed commits are also processed.

Let’s give it a go …

$ git cherry-menu git icing -v3 stable/grizzly to-backport
commit 348cfedfcadc377fa91029b932b2a93d16475822
Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove redundant 'dest' args

    Includes a hack to calculate 'dest' from the argument name. This hack is
    removed in a later commit.

    Change-Id: I60567ff232ab7699f3234b3bfc1618a17a648976

diff --git a/bin/nova-manage b/bin/nova-manage
index 0fde8ba..4919d88 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -127,10 +127,10 @@ def param2id(object_id):
 class VpnCommands(object):
     """Class for managing VPNs."""

-    @args('--project', dest="project_id", metavar='<Project name>',
+    @args('--project', dest='project_id', metavar='<Project name>',

[... snipped ...]

Cherry-pick / blacklist / skip 348cfedfca, or quit ?

It’s showing us the first commit in the porting backlog, and asking us to decide whether to cherry-pick it, blacklist it (i.e. permanently exclude it from the porting process), or defer the decision by skipping it and moving onto the next commit in backlog. (q quits the whole process, of course.)

Let’s try cherry-picking it:

Cherry-pick / blacklist / skip 348cfedfca, or quit ? c

error: could not apply 348cfed... nova-manage: remove redundant 'dest' args
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

Spawning a shell so you can fix; exit the shell when done.
$

Oh dear, this one’s going to need some further thought. Let’s skip it for now:

$ git reset --hard
HEAD is now at ddb676d Merge "Typo: certicates=>certificates in nova.conf"
$ exit
Warning: HEAD did not change; no action taken.
Press enter to continue ... 

-----------------------------------------------------------------------

commit f16534db22d63ce9f762ee6f7a9245126e9f28ed
Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove unused import

    Unused since commit 9ff3121b

    Change-Id: I76bb49669d1cdfc3bf5b1c20087b4bd77420cd91

diff --git a/bin/nova-manage b/bin/nova-manage
index 4919d88..e93c3d8 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -88,7 +88,6 @@ from nova.openstack.common import timeutils
 from nova import quota
 from nova.scheduler import rpcapi as scheduler_rpcapi
 from nova import servicegroup
-from nova import utils
 from nova import version

 CONF = cfg.CONF

Cherry-pick / blacklist / skip f16534db22, or quit ?

cherry-menu presents the next commit in the backlog, which in this case is removing an unused import from bin/nova-manage. Sounds pretty straight-forward, so let’s cherry-pick:

Cherry-pick / blacklist / skip f16534db22, or quit ? c

[stable/grizzly 3f013e4] nova-manage: remove unused import
 Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
 1 file changed, 1 deletion(-)

Looks like this worked fine. The output continues:

-----------------------------------------------------------------------

Already upstream: 775e3b02b2 - Pass project id in quantum driver secgroup list

Here cherry-menu is telling us it automatically skipped over a commit which is already in our target branch. The output continues with the next commit in the backlog, which we can also successfully pick:

commit dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb
Author: Clark Boylan <clark.boylan@CENSORED.com>
Date:   Thu Feb 7 21:39:31 2013 -0800

    Enable tox use of site-packages for libvirt.

    Enable the use of site-packages in tox which will allow the use of the
    system install of libvirt while testing.

    Hardcode the libvirt host UUID for tests that check this UUID when
    system libvirt is being used. Without this hardcoding eight tests
    would fail when using the system libvirt install.

    Partially fixes bug #1113181

    Change-Id: I59c5fbd45639962c0963298203c39759b6ca2d11

[... snipped ...]

Cherry-pick / blacklist / skip dbbbbf4ecb, or quit ? c

[stable/grizzly b4fdf14] Enable tox use of site-packages for libvirt.
 Author: Clark Boylan <clark.boylan@CENSORED.com>
 2 files changed, 8 insertions(+)

-----------------------------------------------------------------------

Already upstream: 76844433f6 - Add caching for ec2 mapping ids.
commit def5fa1e0e1c0426d973e4d8e3935c6eb18698dc
Author: Thierry Carrez <thierry@CENSORED.org>
Date:   Wed Mar 20 16:21:48 2013 +0100

    Set version to 2013.2

    Open Havana development by setting version to 2013.2.

    Change-Id: I37917d28a1f9e0adc2fe3e382412ebc4ed0f3bee

diff --git a/setup.py b/setup.py
index fd968ee..e4bc7d4 100644
--- a/setup.py
+++ b/setup.py
@@ -25,7 +25,7 @@

 setuptools.setup(
       name=project,
-      version=common_setup.get_version(project, '2013.1'),
+      version=common_setup.get_version(project, '2013.2'),
       description='cloud computing fabric controller',
       author='OpenStack',
       author_email='nova@CENSORED.launchpad.net',

Cherry-pick / blacklist / skip def5fa1e0e, or quit ?

Now, obviously we don’t want to backport this version change, so let’s blacklist it in order to permanently excluded it from the porting backlog:

Cherry-pick / blacklist / skip def5fa1e0e, or quit ? b

At this point, your favourite $EDITOR will be launched via an invocation of git notes edit, and you’ll be presented with a text buffer which looks like this:

skip: all
XXX  (you can optionally change the "all" above to the name of the
XXX  upstream branch if you want to limit blacklisting to that upstream)

XXX  Enter your justification for blacklisting here or
XXX  remove the whole note to cancel blacklisting.

#
# Write/edit the notes for the following object:

# commit def5fa1e0e1c0426d973e4d8e3935c6eb18698dc
# Author: Thierry Carrez <thierry@CENSORED.org>
# Date:   Wed Mar 20 16:21:48 2013 +0100
#
#     Set version to 2013.2
#     
#     Open Havana development by setting version to 2013.2.
#     
#     Change-Id: I37917d28a1f9e0adc2fe3e382412ebc4ed0f3bee
#
#  setup.py | 2 +-
#  1 file changed, 1 insertion(+), 1 deletion(-)

At this point you should remove all the lines beginning XXX, and decide whether you want this commit to be blacklisted from any future porting operation. If so, you can leave the skip: all line as is and simply save the buffer and quit the editor, although it’s best practice to also give a sentence or two justifying why you’re blacklisting the commit.

Alternatively you can change all to the name of the branch to which this commit should never be ported, which in this case is stable/grizzly. It is also permitted to use Ruby regular expressions here, by surrounding the regexp with forward slashes, e.g.

skip: /stable/

It never makes sense to port version number changes back to
stable releases.

Once the editor has been quit, we see:

Blacklisted def5fa1e0e
Press enter to continue ...

Behind the scenes, cherry-menu has attached a note to this commit within the refs/notes/upstreaming namespace:

$ git notes --ref=upstreaming show def5fa
skip: all

and you can even see the complete history of your note-editing:

$ git log notes/upstreaming
commit ed6cf5fa75921f23088656b7a8d8a96faa9d769d
Author: Adam Spiers <aspiers@CENSORED.com>
Date:   Fri Sep 20 15:59:04 2013 +0100

    Notes added by 'git notes edit'

commit c1f6ba13e643de4da354375b479470599f92b272
Author: Adam Spiers <aspiers@CENSORED.com>
Date:   Fri Sep 20 15:44:15 2013 +0100

    Notes added by 'git notes add'

[... snipped ...]

After all this hard work, maybe it’s time for lunch. We can safely quit cherry-menu via Control-C or q, and resume it later. Before tucking into a tasty sandwich, let’s quickly review our progress:

$ git icing -v3 -s stable/grizzly to-backport
+ 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
- f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list
- dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb Enable tox use of site-packages for libvirt.
- 76844433f69a7c29ed4566ad34d7e9740feaf660 Add caching for ec2 mapping ids.
. def5fa1e0e1c0426d973e4d8e3935c6eb18698dc Set version to 2013.2
+ 3aa80834e6aab2456f0f5229a63c2dcef007cb36 Change type of ssh_port option from Str to Int
+ 26aa01094a79939320d58f2fe2d5731f169987b1 Change arguments to volume_detach()

[... snipped ...]

Summary
=======

14 commits processed:
    13 already upstream
     1 blacklisted - should not be pushed to this upstream

15 commits remaining:
    15 not yet upstream

Progress: 14 / 29 commits (48%)

Wow, almost half-way! Notice also how the prefix by def5fa1 has changed to a period sign to indicate that it’s now blacklisted.

After lunch, we can resume the process:

$ git cherry-menu git icing -v3 stable/grizzly to-backport
commit 348cfedfcadc377fa91029b932b2a93d16475822
Author: Zhiteng Huang <zhiteng.huang@CENSORED.com>
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove redundant 'dest' args

[... snipped ...]

Cherry-pick / blacklist / skip 348cfedfca, or quit ?

This is the same commit which caused conflicts last time we tried to pick it. Let’s officially postpone resolving conflicts by adding it to a “TODO” blacklist. Press b, then as before, remove the XXX, and also replace the skip: all line with:

TODO: fix conflicts next week

Blacklisted 348cfedfca
Press enter to continue ... 

-----------------------------------------------------------------------

Already upstream: f16534db22 - nova-manage: remove unused import
Already upstream: 775e3b02b2 - Pass project id in quantum driver secgroup list
Already upstream: dbbbbf4ecb - Enable tox use of site-packages for libvirt.
Already upstream: 76844433f6 - Add caching for ec2 mapping ids.
Blacklisted: def5fa1e0e - Set version to 2013.2
commit 3aa80834e6aab2456f0f5229a63c2dcef007cb36
Author: Devananda van der Veen <devananda.vdv@CENSORED.com>
Date:   Wed Mar 20 09:19:44 2013 -0700

    Change type of ssh_port option from Str to Int

    The type of CONF option virtual_power_ssh_port was incorrectly defaulted
    to Str. This can cause Paramiko to raise when casting to %d.

    Fixes bug 1157824.

    Change-Id: I30ddd1ff0da45f8392085249f1bd2a539b201a7e

Now we are taken immediately to where we left off just before lunch, because git-icing automatically detected the cherry-picks we did earlier, and also that we blacklisted the 2013.2 version change commit.

If we run git-icing again, we’ll see that it understands the significance of the TODO attached to the first commit in the backlog:

$ git icing -v3 -s stable/grizzly to-backport                                                        
! 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
- f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list

[... snipped ...]

Summary
=======

14 commits processed:
    13 already upstream
     1 blacklisted - should not be pushed to this upstream

15 commits remaining:
    14 not yet upstream
     1 still needs to be upstreamed, tracked on a TODO list

Progress: 14 / 29 commits (48%)

On subsequent invocations of git cherry-menu, it will offer the option of editing existing notes. There is also an option to automatically skip any notes including the word TODO:

$ git cherry-menu -h
usage: git [<options>] cherry-menu <command> [<args>...]
suggested options:

[... snipped ...]

-c cherry-menu.skip-todos=true
      Skip commits which have notes including 'TODO'.  This allows
      unresolved upstreaming tasks to be tracked via an external
      issue tracker without getting in the way during repeated
      runs of cherry-menu.

[... snipped ...]

So that’s the majority of cherry-menu‘s functionality explained. It’s not rocket science, just a quick hack of a shell-script, but it gets the job done pretty well.

There’s one more important feature I didn’t cover: if the patch-id changes during cherry-picking, cherry-menu will notice, and automatically add the old commit to the blacklist:

Cherry-pick / blacklist / skip 81204d4020, or quit ? c

[stable/grizzly 65cd526] Sync rpc from oslo-incubator.
 Author: Russell Bryant <rbryant@CENSORED.com>
 4 files changed, 47 insertions(+), 21 deletions(-)
The git patch-id changed during cherry-picking.  This is normal when
the diff context changes or merge conflicts are resolved.
Blacklisted 81204d4020 so future runs won't attempt to duplicate the upstreaming.

Maybe our project manager comes over and says “hey, we need those features backported to stable ASAP – I’ve asked Joe and Fred to help you out, OK?”

So far, all the work has happened in a local repository. Of course we can push our updated stable/grizzly branch to a public repository somewhere, so that Joe and Fred can see what we already cherry-picked. But they’d be still missing some valuable meta-data, i.e. the git notes we created which mark which commits to exclude from porting, and why. Unfortunately, git doesn’t yet natively provide a convenient way for pushing/pulling notes between repositories, because the refs/* namespace hasn’t been sufficiently standardized yet. To work around that, I wrote git-rnotes to avoid having to remember a slightly arcane sequence of git commands.

Usage is very simple, and mimics git fetch/push/merge/pull:

Usage: git-rnotes [options] SUBCOMMAND REMOTE
Options:
  -h, --help     Show this help and exit

Subcommands:

  fetch
  push
  merge
  pull

For example, if you have a remote called public:

export GIT_NOTES_REF=refs/notes/upstreaming
git rnotes push public

or if Joe has updated his notes and pushed them to public whilst you also updated yours, simply do a pull, which is equivalent to a fetch followed by a merge:

export GIT_NOTES_REF=refs/notes/upstreaming
git rnotes pull public

Sorry that was so long. But hopefully you agree it’s relatively simple once you try it out. I’d love to know if anyone finds this useful – if you do, please leave a comment below! Of course pull requests against my git-config repository are always welcome. There are many other git-related hacks in that repository which you may also find useful.

Thanks for reading!