upstreaming Archives – Structured Procrastination

git branch auto-magic: git-splice, git-transplant, git-deps, and announcing git-explode!

Adam — Thu, 14 Jun 2018 22:00:00 +0000

For the last few years I’ve been enjoying the luxury of SUSE’s generous HackWeek policy to work on building four tools supporting higher-level workflows on top of git. I’d already (quietly) announced three of them: git-splice, git-transplant, git-deps (more details below). But I’m now excited to announce that I’ve just released the fourth: git-explode !

git-explode automatically explodes a large topic branch into a set of smaller, independent topic branches. It does this by harnessing git-deps to automatically detect inter-dependencies between commits in the large source branch and using that dependency tree to construct the smaller topic branches.

I recently presented all four tools at a Meetup of the Git London User Group, and thanks to the awesome services of the host venue Skills Matter, I’m delighted to announce that the talk is now online:

If you don’t have time to watch the whole thing, you can look at the slides, or just keep on reading to see which ones you might be interested in. I’ll list them first, and then talk about the motivation for writing them.

The tools

The first pair are available as patches to git itself:

git-splice – splices commits onto/into/out of current branch

Jump to 08:44 in the video to see me introduce and demo this, or look at the slides, or read the man page for example usage.
git-transplant – transplants commits from current branch onto/into another

Jump to 32:22 in the video to see me introduce and demo this, or look at the slides, or read the man page for example usage.

The second pair are available as Python modules on PyPI, but both have a command-line interface too:

git-deps – automatically detects textual dependencies between commits

Jump to 41:28 in the video to see me introduce and demo this, or look at the slides, or read the README (which has a couple of short YouTube videos demoing the tool including the fancy web interface), or read my previous blog post about this.
git-explode – explodes a linear sequence of git commits into new independent topic branches

Jump to 51:10 in the video to see me introduce and demo this, or look at the slides, or read the README.

It’s taken a while to get them to a reasonable degree of polish, during which this work has been mostly under the radar, but now I’m now going to make a more concerted effort to advertise the tools, hence this blog post.

The Big Idea

(Jump to 02:27 in the video to see me talk about this, or look at the slides.)

All developers face common problems:

It’s difficult to predict what you’ll be doing next. One minute you might be working on a feature, the next you find a bug you need to fix, and the next you find some technical debt which requires refactoring.
Multi-tasking is almost unavoidable.
We’re human, so we sometimes screw up when trying to follow best practice workflow.

What does this mean when using git? Typically it means we are constantly juggling branches, and often need to manipulate the history of branches for various reasons, for example:

Maintaining clean separation of topics between branches to facilitate code reviews: one branch for a refactoring, one for a bugfix, one for a feature, and so on.
Backporting or forward-porting series of patches between branches.

My feeling is that while the git ecosystem has continued to evolve incrementally over the last few years, not much progress has been made in terms of automating this janitorial work, and that most people either do it by hand, or don’t bother doing it at all. This leads to unclean history, and/or more work and frustration for reviewers.

So I wrote these tools (and others) in an attempt to push the boundaries of automating git workflows to higher levels, as I explained on the git mailing list last year. Even if all the use cases they can cover have yet to be discovered, I hope you find them useful, or at least thought-provoking enough that you might join the effort!

I’d love some feedback, so do let me know what you think in the comments below.

The post git branch auto-magic: git-splice, git-transplant, git-deps, and announcing git-explode! appeared first on Structured Procrastination.

Easier upstreaming / back-porting of patch series with git

Adam — Thu, 19 Sep 2013 20:22:00 +0000

Have you ever needed to port a selection of commits from one git branch to another, but without doing a full merge? This is a common challenge, e.g.

forward-porting / upstreaming bugfixes from a stable release branch to a development branch, or
back-porting features from a development branch to a stable release branch.

Of course, git already goes quite some way to making this possible:

git cherry-pick can port individual commits, or even a range of commits (since git 1.7.2) from anywhere, into the current branch.
git cherry can compare a branch with its upstream branch and find which commits have been upstreamed and which haven’t. This command is particularly clever because, thanks to git patch-id, it can correctly spot when a commit has been upstreamed, even when the upstreaming process resulted in changes to the commit message, line numbers, or whitespace.
git rebase --onto can transplant a contiguous series of commits onto another branch.

It’s not always that easy …

However, on the occasions when you need to sift through a larger number of commits on one branch, and port them to another branch, complications can arise:

If cherry-picking a commit results in changes to its patch context, git patch-id will return a different SHA-1, and subsequent invocations of git cherry will incorrectly tell you that you haven’t yet ported that commit.
If you mess something up in the middle of a git rebase, recovery can be awkward, and git rebase --abort will land you back at square one, undoing a lot of your hard work.
If the porting process is big enough, it could take days or even weeks, so you need some way of reliably tracking which commits have already been ported and which still need porting. In this case you may well want to adopt a divide-and-conquer approach by sharing out the porting workload between team-mates.
The more the two branches have diverged, the more likely it is that conflicts will be encountered during cherry-picking.
There may be commits within the range you are looking at which after reviewing, you decide should be excluded from the port, or at least porting them needs to be postponed to a later point.

It could be argued that all of these problems can be avoided with the right branch and release management workflows, and I don’t want to debate that in this post. However, this is the real world, and sometimes it just happens that you have to deal with a porting task which is less than trivial. Well, that happened to me and my team not so long ago, so I’m here to tell you that I have written and published some tools to solve these problems. If that’s of interest, then read on!

Building tools around `git notes`

Since git patch-id doesn’t guarantee to return an SHA-1 which is stable enough for us to always depend on, we need some other way of reliably tracking which commits have been upstreamed, and where upstreamed commits came from. It would also be useful to be able to mark selected commits as blacklisted, i.e. permanently excluded from the current porting process, or at least as deferred, i.e. temporarily excluded.

Of course, we could do that simply by maintaining an out-of-band text file somewhere outside the repositories in question, but that doesn’t integrate at all with existing git workflows. Whilst pondering these challenges, I discovered a little-known and seldom-used feature of git, which is the ability to attach arbitrary text as notes on a given commit, via the git notes command. “This sounds perfect!”, I thought, and actually as it turned out, it served as a pretty nice foundation for some higher-level tools which needed building:

git-icing – an “interface-candy” wrapper around git cherry which adds a splash of colour, and checks for specially formatted notes attached to commits which indicate that those commits are temporarily or permanently excluded from the upstreaming backlog
git-cherry-menu – an interactive wrapper around git-icing and git-notes which makes it easy to cherry-pick and/or blacklist non-upstreamed commits
git-rnotes – a wrapper around git notes which makes it easier to share notes to and from remote repositories

A “real world” example

Let’s see how this works in practice, by pretending we need to backport a bunch of commits from OpenStack Nova‘s master branch to its stable/grizzly release branch.

$ git clone git@github.com:openstack/nova.git
Cloning into 'nova'...
remote: Counting objects: 208984, done.
remote: Compressing objects: 100% (54766/54766), done.
remote: Total 208984 (delta 167220), reused 188334 (delta 148279)
Receiving objects: 100% (208984/208984), 124.50 MiB | 4.42 MiB/s, done.
Resolving deltas: 100% (167220/167220), done.
$ cd nova

Now, master has changed a lot since grizzly was released, so let’s keep our pretend project to a manageable size by only attempting to backport only commits which appeared in the first few days after the stable branch was established:

$ git describe $( git merge-base origin/stable/grizzly master )
2013.1.rc1
$ git show 2013.1.rc1 | head -n3
tag 2013.1.rc1
Tagger: Thierry Carrez 
Date:   Wed Mar 20 17:13:32 2013 +0100

OK, so grizzly was branched at the point where the first release candidate was tagged. That makes sense. Now let’s choose a point about 5 days later:

$ git rev-list --no-merges --before='25 March 2013' origin/stable/grizzly..master | head -n1
d6a9d38a0b7488dd77d3a01ae9c4171c271d7314
$ git tag to-backport d6a9d38

… and get ready to backport commits from to-backport back into our local stable/grizzly branch:

$ git checkout stable/grizzly
Branch stable/grizzly set up to track remote branch stable/grizzly from origin.
Switched to a new branch 'stable/grizzly'

(If you are following along at home, your rev-list will give a different result, since origin/stable/grizzly will have changed since I wrote this. Don’t worry – just copy the git tag command above verbatim, and create your local grizzly branch via git checkout -b stable/grizzly fc9af8f0a9 so that it points to the same place mind did when writing this article.)

At this point, two quick clarifications are needed, because things can get a bit confusing.

A quick aside: some observations on terminology

Firstly, if you read the git cherry man page or -h usage text, it is described as a tool to “find commits not merged upstream”, with the expectation that you would cherry-pick the commits it finds onto the upstream branch. But if you happen to be backporting, then the cherry-picking will happen in the opposite direction, so the “upstream” referred to by the man page is actually the real downstream, and vice-versa. Confused yet? If so, wherever you read the word “upstream” in the git cherry documentation / help text, pretend that you actually read “target”. Then just think of “source” and “target” branches, and remember that the process is always to look for commits in the source which are not yet in the target, and then cherry-pick those into the target (or decide that you don’t want to). The same applies for the git icing and cherry-menu help text.

Secondly, normally we’d be porting from one branch to another, not from a tag to a branch. But in this artificially constructed scenario, it wouldn’t make much sense to create a to-backport branch pointing to d6a9d38, because that branch isn’t going to move; the only branch which will be changing is our stable/grizzly branch, as we cherry-pick stuff into it. In other words, the target has to be a branch, because that will change during the cherry-picking, but the source can be a branch / tag / ref / commit SHA-1 … basically anything which git rev-parse understands.

How far behind?

When embarking on a porting project, the first question is usually “how much work is this going to be?” Fortunately git-icing is here to help!

(By the way, if you want to follow along with the below by trying this yourself, simply download the three scripts either from the links above or

git clone https://github.com/aspiers/git-config

and then copy them to any directory on your $PATH, making sure that they are executable.)

git-icing is designed to be invoked in exactly the same way as git cherry, but it has some extra options, including the convenient --summary option, and a verbosity parameter which controls which categories of commits are displayed:

$ git icing -h
usage: git icing [options] [git cherry args]

    -v, --verbosity [N]              Set verbosity level
    -s, --summary                    Show summary

first | verbosity | description of
field |   level   | classification
------+-----------+-----------------------------------------------------
  +   |     1     | not yet upstream
  !   |     1     | still needs to be upstreamed, tracked on a TODO list
  .?  |     1     | upstream *and* blacklisted?!
  !?  |     1     | upstream *and* marked as a TODO?
  .   |     2     | blacklisted - should not be pushed to this upstream
  ?   |     2     | not yet upstream, with unparseable note
  -   |     3     | already upstream
  #   |     3     | already upstream with an annotation note

(When you run that on a colour-capable terminal, you’ll also see that the classifications are also nicely colour-coded so that commits which require attention naturally capture your eye more than those which don’t.)

Let’s see all commits (basically a colour-coded version of what git cherry would show us), plus a summary of the status quo:

$ git icing -v3 --summary stable/grizzly to-backport
+ 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
+ f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list
+ dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb Enable tox use of site-packages for libvirt.
- 76844433f69a7c29ed4566ad34d7e9740feaf660 Add caching for ec2 mapping ids.
+ def5fa1e0e1c0426d973e4d8e3935c6eb18698dc Set version to 2013.2
+ 3aa80834e6aab2456f0f5229a63c2dcef007cb36 Change type of ssh_port option from Str to Int
+ 26aa01094a79939320d58f2fe2d5731f169987b1 Change arguments to volume_detach()
+ b52a2157bedba693c5da7dcb783b7a151769d6b1 Add placeholder migrations to allow backports.
+ 8d5159dd6dd36d6f3d1aadc1123574c5b0aa0614 Don't actually connect to libvirtd in unit tests.
+ 8cd5f862ac071023d5a7984f07513e5aa91f7d3d Bring back sexy colorized test results.
- 45e65d8c0301da689de1afcbc9f45756e71097ab Update instance network info cache to include vif_type.
- b8f9815c5ab61466009c0447f54abc4c309e4e3f nova-manage vm list fails looking 'instance_type'
+ c199fd6fa39311305b5b5d94c2e3732b826a6414 Remove outdated try except block in ec2 code
- 03a2463be8034ee4764ac97e8020fc3d3a32f1fd Make nova.virt.fake.FakeDriver useable in integration testing
- 67628c56caf9d84588a92448880ecdb33eea08b4 Fixes passing arbitrary conductor_api argument
+ e136de1aea8d1469465272942fa3d2769cbe3a80 xenapi: fix console for rescued instance
+ 5df18f6b915cf49ac8559274a631f1366eab00fd Add a comment to placeholder migrations.
- 9075069098e32b47bd5011e2653a23b61c18d4a3 Initialize compute manager before loading driver.
- 3801a4d2f4c59dbfda49131ddde22fcb3976d651 translate cinder BadRequest exception
- 132a0c1fd1eb127e393e5794ffa6d4a3a4950567 Don't log traceback on rpc timeout.
+ 81204d4020712cf5b5b368b75d934d637b2c001b Sync rpc from oslo-incubator.
+ f665d798234c19cccc148a178e94c3717ba8bd6e Sync everything from oslo-incubator.
- 12f7d6332a731421d6e2a190ac60828c1cf98910 Reset ec2 image cache between S3 tests.
+ 7a9ce26aaf03459a7ea367fbdb91d91626a1ee93 py2.6 doesn't support TextTestRunner resultclass
- 5908b60b0420f1ad528e56b0c147a330e9a1a5d6 Make _downsize_quota_delta() use stashed instance types
+ 91b9c208f0c6e21759c7828aa42b3216853be425 disable colorizer as it swallows fails
+ df0560f0353bc0837cc62cfaa9029d21d45c529e Imported Translations from Transifex
+ d6a9d38a0b7488dd77d3a01ae9c4171c271d7314 set up FakeLogger for root logger.

Summary
=======

11 commits processed:
    11 already upstream

18 commits remaining:
    18 not yet upstream

Progress: 11 / 29 commits (37%)

OK, not bad! We didn’t even do any real work yet, we’re already 37% of the way through. That’s because some of the commits (the ones prefixed by -) have already been cherry-picked into stable/grizzly. Let’s prove this for the first commit prefixed by - in the above output, which was 775e3b02:

$ git log -n1 775e3b02
commit 775e3b02b2afbf101db22b87a1c3b189d68532e1
Author: Kieran Spear 
Date:   Mon Mar 18 17:32:26 2013 +1100

    Pass project id in quantum driver secgroup list

    The quantum driver is always returning security groups from every tenant
    a user has access to, even when the "project" filter is supplied.

    Make sure we pass along the project value when we call
    quantum.list_security_groups() so it's properly filtered.

    Fixes bug 1155381.

    Change-Id: I682c66a1f3f9db18b5f9924a37b45c759ff259f7

This commit appears to be only on master, not on stable/grizzly:

$ git branch --all --contains 775e3b02
  master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master

Let’s try to find the commit which cherry-picked it into stable/grizzly:

$ git log --format=fuller --no-merges --grep='Pass project id' stable/grizzly
commit 989435822e5841bc1355e75bbdb003b10a3baf58
Author:     Kieran Spear 
AuthorDate: Mon Mar 18 17:32:26 2013 +1100
Commit:     Chuck Short 
CommitDate: Tue Mar 26 10:09:40 2013 -0500

    Pass project id in quantum driver secgroup list

    The quantum driver is always returning security groups from every tenant
    a user has access to, even when the "project" filter is supplied.

    Make sure we pass along the project value when we call
    quantum.list_security_groups() so it's properly filtered.

    Fixes bug 1155381.

    Change-Id: I682c66a1f3f9db18b5f9924a37b45c759ff259f7
    (cherry picked from commit 775e3b02b2afbf101db22b87a1c3b189d68532e1)
$ git branch --all --contains 9894358
* stable/grizzly
  remotes/origin/stable/grizzly

There it is! Notice that it was committed 6 days after RC1 was tagged, but it has the same patch-id as the original, even though the commit message/date and committer are all different:

$ git patch-id < <(git show 9894358)
42dd410cf36ad1c16ceb1bb7f6ed6e3f01cc28bf 989435822e5841bc1355e75bbdb003b10a3baf58
$ git patch-id < <(git show 775e3b02)
42dd410cf36ad1c16ceb1bb7f6ed6e3f01cc28bf 775e3b02b2afbf101db22b87a1c3b189d68532e1

`git cherry-menu` – let the picking commence

Now we can actually start porting the remaining commits. git-cherry-menu provides an interactive interface to make this process less tedious:

$ git cherry-menu -h
usage: git [options] cherry-menu  [...]
suggested options:

[... snipped ...]

  are typically "git icing -v3" or "git cherry", but
can be anything which gives output in the same format.  This allows
more control over which commits constitute the upstreaming backlog,
e.g.

    git icing -v2 $upstream $downstream | grep ... > tmpfile
    # Could edit tmpfile here if we want
    git-cherry-menu cat tmpfile

Provides an interactive wrapper around git-icing (or git cherry).  For
each commit provided on STDIN by COMMAND which has not yet been
upstreamed, asks the user whether they want to cherry-pick the commit,
blacklist it, or skip it.  After a successful cherry-pick, the source
commit will be automatically blacklisted if the patch-id changed.

You can quit the process at any time and safely re-run it later - it
will resume from where you left off.

Invoking icing with "-v2" ensures that previously blacklisted /
upstreamed commits are also processed.

Let’s give it a go …

$ git cherry-menu git icing -v3 stable/grizzly to-backport
commit 348cfedfcadc377fa91029b932b2a93d16475822
Author: Zhiteng Huang 
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove redundant 'dest' args

    Includes a hack to calculate 'dest' from the argument name. This hack is
    removed in a later commit.

    Change-Id: I60567ff232ab7699f3234b3bfc1618a17a648976

diff --git a/bin/nova-manage b/bin/nova-manage
index 0fde8ba..4919d88 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -127,10 +127,10 @@ def param2id(object_id):
 class VpnCommands(object):
     """Class for managing VPNs."""

-    @args('--project', dest="project_id", metavar='',
+    @args('--project', dest='project_id', metavar='',

[... snipped ...]

Cherry-pick / blacklist / skip 348cfedfca, or quit ?

It’s showing us the first commit in the porting backlog, and asking us to decide whether to cherry-pick it, blacklist it (i.e. permanently exclude it from the porting process), or defer the decision by skipping it and moving onto the next commit in backlog. (q quits the whole process, of course.)

Let’s try cherry-picking it:

Cherry-pick / blacklist / skip 348cfedfca, or quit ? c

error: could not apply 348cfed... nova-manage: remove redundant 'dest' args
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add ' or 'git rm '
hint: and commit the result with 'git commit'

Spawning a shell so you can fix; exit the shell when done.
$

Oh dear, this one’s going to need some further thought. Let’s skip it for now:

$ git reset --hard
HEAD is now at ddb676d Merge "Typo: certicates=>certificates in nova.conf"
$ exit
Warning: HEAD did not change; no action taken.
Press enter to continue ... 

-----------------------------------------------------------------------

commit f16534db22d63ce9f762ee6f7a9245126e9f28ed
Author: Zhiteng Huang 
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove unused import

    Unused since commit 9ff3121b

    Change-Id: I76bb49669d1cdfc3bf5b1c20087b4bd77420cd91

diff --git a/bin/nova-manage b/bin/nova-manage
index 4919d88..e93c3d8 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -88,7 +88,6 @@ from nova.openstack.common import timeutils
 from nova import quota
 from nova.scheduler import rpcapi as scheduler_rpcapi
 from nova import servicegroup
-from nova import utils
 from nova import version

 CONF = cfg.CONF

Cherry-pick / blacklist / skip f16534db22, or quit ?

cherry-menu presents the next commit in the backlog, which in this case is removing an unused import from bin/nova-manage. Sounds pretty straight-forward, so let’s cherry-pick:

Cherry-pick / blacklist / skip f16534db22, or quit ? c

[stable/grizzly 3f013e4] nova-manage: remove unused import
 Author: Zhiteng Huang 
 1 file changed, 1 deletion(-)

Looks like this worked fine. The output continues:

-----------------------------------------------------------------------

Already upstream: 775e3b02b2 - Pass project id in quantum driver secgroup list

Here cherry-menu is telling us it automatically skipped over a commit which is already in our target branch. The output continues with the next commit in the backlog, which we can also successfully pick:

commit dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb
Author: Clark Boylan 
Date:   Thu Feb 7 21:39:31 2013 -0800

    Enable tox use of site-packages for libvirt.

    Enable the use of site-packages in tox which will allow the use of the
    system install of libvirt while testing.

    Hardcode the libvirt host UUID for tests that check this UUID when
    system libvirt is being used. Without this hardcoding eight tests
    would fail when using the system libvirt install.

    Partially fixes bug #1113181

    Change-Id: I59c5fbd45639962c0963298203c39759b6ca2d11

[... snipped ...]

Cherry-pick / blacklist / skip dbbbbf4ecb, or quit ? c

[stable/grizzly b4fdf14] Enable tox use of site-packages for libvirt.
 Author: Clark Boylan 
 2 files changed, 8 insertions(+)

-----------------------------------------------------------------------

Already upstream: 76844433f6 - Add caching for ec2 mapping ids.
commit def5fa1e0e1c0426d973e4d8e3935c6eb18698dc
Author: Thierry Carrez 
Date:   Wed Mar 20 16:21:48 2013 +0100

    Set version to 2013.2

    Open Havana development by setting version to 2013.2.

    Change-Id: I37917d28a1f9e0adc2fe3e382412ebc4ed0f3bee

diff --git a/setup.py b/setup.py
index fd968ee..e4bc7d4 100644
--- a/setup.py
+++ b/setup.py
@@ -25,7 +25,7 @@

 setuptools.setup(
       name=project,
-      version=common_setup.get_version(project, '2013.1'),
+      version=common_setup.get_version(project, '2013.2'),
       description='cloud computing fabric controller',
       author='OpenStack',
       author_email='nova@CENSORED.launchpad.net',

Cherry-pick / blacklist / skip def5fa1e0e, or quit ?

Now, obviously we don’t want to backport this version change, so let’s blacklist it in order to permanently excluded it from the porting backlog:

Cherry-pick / blacklist / skip def5fa1e0e, or quit ? b

At this point, your favourite $EDITOR will be launched via an invocation of git notes edit, and you’ll be presented with a text buffer which looks like this:

skip: all
XXX  (you can optionally change the "all" above to the name of the
XXX  upstream branch if you want to limit blacklisting to that upstream)

XXX  Enter your justification for blacklisting here or
XXX  remove the whole note to cancel blacklisting.

#
# Write/edit the notes for the following object:

# commit def5fa1e0e1c0426d973e4d8e3935c6eb18698dc
# Author: Thierry Carrez 
# Date:   Wed Mar 20 16:21:48 2013 +0100
#
#     Set version to 2013.2
#     
#     Open Havana development by setting version to 2013.2.
#     
#     Change-Id: I37917d28a1f9e0adc2fe3e382412ebc4ed0f3bee
#
#  setup.py | 2 +-
#  1 file changed, 1 insertion(+), 1 deletion(-)

At this point you should remove all the lines beginning XXX, and decide whether you want this commit to be blacklisted from any future porting operation. If so, you can leave the skip: all line as is and simply save the buffer and quit the editor, although it’s best practice to also give a sentence or two justifying why you’re blacklisting the commit.

Alternatively you can change all to the name of the branch to which this commit should never be ported, which in this case is stable/grizzly. It is also permitted to use Ruby regular expressions here, by surrounding the regexp with forward slashes, e.g.

skip: /stable/

It never makes sense to port version number changes back to
stable releases.

Once the editor has been quit, we see:

Blacklisted def5fa1e0e
Press enter to continue ...

Behind the scenes, cherry-menu has attached a note to this commit within the refs/notes/upstreaming namespace:

$ git notes --ref=upstreaming show def5fa
skip: all

and you can even see the complete history of your note-editing:

$ git log notes/upstreaming
commit ed6cf5fa75921f23088656b7a8d8a96faa9d769d
Author: Adam Spiers 
Date:   Fri Sep 20 15:59:04 2013 +0100

    Notes added by 'git notes edit'

commit c1f6ba13e643de4da354375b479470599f92b272
Author: Adam Spiers 
Date:   Fri Sep 20 15:44:15 2013 +0100

    Notes added by 'git notes add'

[... snipped ...]

After all this hard work, maybe it’s time for lunch. We can safely quit cherry-menu via Control-C or q, and resume it later. Before tucking into a tasty sandwich, let’s quickly review our progress:

$ git icing -v3 -s stable/grizzly to-backport
+ 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
- f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list
- dbbbbf4ecb2c7b03ebb30fa11e077d35b2186cbb Enable tox use of site-packages for libvirt.
- 76844433f69a7c29ed4566ad34d7e9740feaf660 Add caching for ec2 mapping ids.
. def5fa1e0e1c0426d973e4d8e3935c6eb18698dc Set version to 2013.2
+ 3aa80834e6aab2456f0f5229a63c2dcef007cb36 Change type of ssh_port option from Str to Int
+ 26aa01094a79939320d58f2fe2d5731f169987b1 Change arguments to volume_detach()

[... snipped ...]

Summary
=======

14 commits processed:
    13 already upstream
     1 blacklisted - should not be pushed to this upstream

15 commits remaining:
    15 not yet upstream

Progress: 14 / 29 commits (48%)

Wow, almost half-way! Notice also how the prefix by def5fa1 has changed to a period sign to indicate that it’s now blacklisted.

After lunch, we can resume the process:

$ git cherry-menu git icing -v3 stable/grizzly to-backport
commit 348cfedfcadc377fa91029b932b2a93d16475822
Author: Zhiteng Huang 
Date:   Tue Feb 26 09:23:13 2013 +0800

    nova-manage: remove redundant 'dest' args

[... snipped ...]

Cherry-pick / blacklist / skip 348cfedfca, or quit ?

This is the same commit which caused conflicts last time we tried to pick it. Let’s officially postpone resolving conflicts by adding it to a “TODO” blacklist. Press b, then as before, remove the XXX, and also replace the skip: all line with:

TODO: fix conflicts next week

Blacklisted 348cfedfca
Press enter to continue ... 

-----------------------------------------------------------------------

Already upstream: f16534db22 - nova-manage: remove unused import
Already upstream: 775e3b02b2 - Pass project id in quantum driver secgroup list
Already upstream: dbbbbf4ecb - Enable tox use of site-packages for libvirt.
Already upstream: 76844433f6 - Add caching for ec2 mapping ids.
Blacklisted: def5fa1e0e - Set version to 2013.2
commit 3aa80834e6aab2456f0f5229a63c2dcef007cb36
Author: Devananda van der Veen 
Date:   Wed Mar 20 09:19:44 2013 -0700

    Change type of ssh_port option from Str to Int

    The type of CONF option virtual_power_ssh_port was incorrectly defaulted
    to Str. This can cause Paramiko to raise when casting to %d.

    Fixes bug 1157824.

    Change-Id: I30ddd1ff0da45f8392085249f1bd2a539b201a7e

Now we are taken immediately to where we left off just before lunch, because git-icing automatically detected the cherry-picks we did earlier, and also that we blacklisted the 2013.2 version change commit.

If we run git-icing again, we’ll see that it understands the significance of the TODO attached to the first commit in the backlog:

$ git icing -v3 -s stable/grizzly to-backport                                                        
! 348cfedfcadc377fa91029b932b2a93d16475822 nova-manage: remove redundant 'dest' args
- f16534db22d63ce9f762ee6f7a9245126e9f28ed nova-manage: remove unused import
- 775e3b02b2afbf101db22b87a1c3b189d68532e1 Pass project id in quantum driver secgroup list

[... snipped ...]

Summary
=======

14 commits processed:
    13 already upstream
     1 blacklisted - should not be pushed to this upstream

15 commits remaining:
    14 not yet upstream
     1 still needs to be upstreamed, tracked on a TODO list

Progress: 14 / 29 commits (48%)

On subsequent invocations of git cherry-menu, it will offer the option of editing existing notes. There is also an option to automatically skip any notes including the word TODO:

$ git cherry-menu -h
usage: git [] cherry-menu  [...]
suggested options:

[... snipped ...]

-c cherry-menu.skip-todos=true
      Skip commits which have notes including 'TODO'.  This allows
      unresolved upstreaming tasks to be tracked via an external
      issue tracker without getting in the way during repeated
      runs of cherry-menu.

[... snipped ...]

So that’s the majority of cherry-menu‘s functionality explained. It’s not rocket science, just a quick hack of a shell-script, but it gets the job done pretty well.

There’s one more important feature I didn’t cover: if the patch-id changes during cherry-picking, cherry-menu will notice, and automatically add the old commit to the blacklist:

Cherry-pick / blacklist / skip 81204d4020, or quit ? c

[stable/grizzly 65cd526] Sync rpc from oslo-incubator.
 Author: Russell Bryant 
 4 files changed, 47 insertions(+), 21 deletions(-)
The git patch-id changed during cherry-picking.  This is normal when
the diff context changes or merge conflicts are resolved.
Blacklisted 81204d4020 so future runs won't attempt to duplicate the upstreaming.

`git rnotes` – sharing the fruits of your labours

Maybe our project manager comes over and says “hey, we need those features backported to stable ASAP – I’ve asked Joe and Fred to help you out, OK?”

So far, all the work has happened in a local repository. Of course we can push our updated stable/grizzly branch to a public repository somewhere, so that Joe and Fred can see what we already cherry-picked. But they’d be still missing some valuable meta-data, i.e. the git notes we created which mark which commits to exclude from porting, and why. Unfortunately, git doesn’t yet natively provide a convenient way for pushing/pulling notes between repositories, because the refs/* namespace hasn’t been sufficiently standardized yet. To work around that, I wrote git-rnotes to avoid having to remember a slightly arcane sequence of git commands.

Usage is very simple, and mimics git fetch/push/merge/pull:

Usage: git-rnotes [options] SUBCOMMAND REMOTE
Options:
  -h, --help     Show this help and exit

Subcommands:

  fetch
  push
  merge
  pull

For example, if you have a remote called public:

export GIT_NOTES_REF=refs/notes/upstreaming
git rnotes push public

or if Joe has updated his notes and pushed them to public whilst you also updated yours, simply do a pull, which is equivalent to a fetch followed by a merge:

export GIT_NOTES_REF=refs/notes/upstreaming
git rnotes pull public

Wrapping up

Sorry that was so long. But hopefully you agree it’s relatively simple once you try it out. I’d love to know if anyone finds this useful – if you do, please leave a comment below! Of course pull requests against my git-config repository are always welcome. There are many other git-related hacks in that repository which you may also find useful.

Thanks for reading!

The post Easier upstreaming / back-porting of patch series with git appeared first on Structured Procrastination.