Comments (10)

By Adam, June 14, 2018 11:00 pm

For the last few years I’ve been enjoying the luxury of SUSE’s generous HackWeek policy to work on building four tools supporting higher-level workflows on top of git. I’d already (quietly) announced three of them: git-splice, git-transplant, git-deps (more details below). But I’m now excited to announce that I’ve just released the fourth: git-explode !

git-explode automatically explodes a large topic branch into a set of smaller, independent topic branches. It does this by harnessing git-deps to automatically detect inter-dependencies between commits in the large source branch and using that dependency tree to construct the smaller topic branches.

I recently presented all four tools at a Meetup of the Git London User Group, and thanks to the awesome services of the host venue Skills Matter, I’m delighted to announce that the talk is now online:

If you don’t have time to watch the whole thing, you can look at the slides, or just keep on reading to see which ones you might be interested in. I’ll list them first, and then talk about the motivation for writing them.

Continue reading 'git branch auto-magic: git-splice, git-transplant, git-deps, and announcing git-explode!'»

Comments (1)

By Adam, May 17, 2015 4:42 am

In a populated compute cloud, there are several scenarios in which it’s beneficial to be able to rearrange VM guest instances into a different placement across the hypervisor hosts via migration (live or otherwise). These use cases typically fall into three categories:

1. Rebalancing – spread the VMs evenly across as many physical VM host machines as possible (conceptually similar to vSphere DRS). Example use cases:
   • Optimise workloads for performance, by reducing CPU / I/O hotspots.
   • Maximise headroom on each physical machine.
   • Reduce thermal hotspots in order to reduce power consumption; for example, back in 2003, HP showed that intelligent workload placement can reduce energy consumption by more than 14%. (Here’s an old slidedeck I made years ago when researching this use case.)
2. Consolidation – condense VMs onto fewer physical VM host machines (conceptually similar to vSphere DPM). Typically involves some degree of defragmentation. Example use cases:
   • Increase CPU / RAM / I/O utilization. CERN blogged last summer about this Tetris-like challenge, and it can be taken even further over-committing CPU/RAM and/or memory page sharing.
   • Free up physical servers to reduce power consumption.
3. Evacuation – free up physical servers:
   • for repurposing or decommissioning
   • for maintenance (BIOS upgrades, re-cabling etc.) or repair
   • to protect SLAs, e.g. when monitors indicate potential imminent hardware failure, or when HVAC failures are likely to cause servers to shutdown due to over-heating. (This is different to the somewhat confusingly named nova evacuate functionality which moves VMs to a new host after the server has already failed.)

Whilst one-shot manual or semi-automatic rearrangement can bring immediate benefits, the biggest wins often come when continual rearrangement is automated. The approaches can also be combined, e.g. first evacuate and/or consolidate, then rebalance on the remaining physical servers.

Other custom rearrangements may be required according to other IT- or business-driven policies, e.g. only rearrange VM instances relating to a specific workload, in order to increase locality of reference, reduce latency, respect availability zones, or facilitate other out-of-band workflows or policies (such as data privacy or other legalities).

In the rest of this post I will expand this topic in the context of OpenStack, talk about the computer science behind it, propose a possible way forward, and offer a working prototype in Python.

If you’re in Vancouver for the OpenStack summit which starts this Monday and you find this post interesting, ping me for a face-to-face chat!

Continue reading 'Cloud rearrangement for fun and profit'»

Comments (1)

By Adam, October 5, 2014 1:59 pm

Like many F/OSS developers, I’m a heavy user of GitHub. This means I interact with other developers via GitHub multiple times a day. GitHub has a very nice notifications system which lets me know when there has been some activity on a project I’m collaborating on.

I’m a fan of David Allen’s GTD (“Getting Things Done”) system, and in my experience I get the best results by minimising the number of inboxes I have to look at every day. So I use another great feature of GitHub, which is the ability to have notification emails delivered directly to your email inbox. This means I don’t have to keep checking https://github.com/notifications in addition to my email inbox.

However, this means that I receive GitHub notifications in two places. Wouldn’t it be nice if when I read them in my email inbox, GitHub could somehow realise and mark them read at https://github.com/notifications too, so that when I look there, I don’t end up getting reminded about notifications I’ve already seen in my inbox? Happily the folks at GitHub already thought of this too, and come up with a solution:

If you read a notification email, it’ll automatically be marked as read in the Notifications section. An invisible image is embedded in each mail message to enable this, which means that you must allow viewing images from notifications@github.com in order for this feature to work.

– https://help.github.com/articles/configuring-notification-emails/#shared-read-state

But there’s a catch! Like many Linux geeks, I use mutt for reading and writing email. In fact, I’ve been using it since 1997 and I’m still waiting for another MUA to appear which is more powerful and lets me crunch through email faster. However mutt is primarily text-based, which means by default it doesn’t download images when displaying HTML-based email. Of course, it can. But do I want it to automatically open a new tab in my browser every time I encounter an HTML attachment? No! That would slow me down horribly. Even launching a terminal-based HTML viewer such as w3m or links or lynx would be too slow.

So I figured out a better solution. mutt has a nice message-hook feature where you can configure it to automatically execute mutt functions for any message matching specific criteria just before it displays the message. So we can use that to pipe the whole email to a script whenever a message is being read for the first time:

message-hook "(~N|~O) ~f notifications@github.com" "push '<pipe-message>read-github-notification\n'"

(~N|~O) matches mails which have the N flag (meaning new unread email) or O (meaning old unread email) set.

The read-github-notifications script reads the email on STDIN, extracts the URL of the 1-pixel read notification beacon <img> embedded in the HTML attachment, and sends an HTTP request for that image, so that github knows the notification has been read.

This means an extra delay of 0.5 seconds or so when viewing a notification email, but for me it’s a worthwhile sacrifice.

If you want to try it, simply download the script and stick it somewhere on your $PATH, and then add the above line to your ~/.muttrc file.

UPDATE 2017/08/29: Unfortunately I have since discovered that the above solution breaks when you try to save one of these notifications before reading it. Suggestions on how to fix this are most welcome!

Comments (15)

By Adam, September 19, 2013 9:22 pm

Have you ever needed to port a selection of commits from one git branch to another, but without doing a full merge? This is a common challenge, e.g.

• forward-porting / upstreaming bugfixes from a stable release branch to a development branch, or
• back-porting features from a development branch to a stable release branch.

Of course, git already goes quite some way to making this possible:

• git cherry-pick can port individual commits, or even a range of commits (since git 1.7.2) from anywhere, into the current branch.
• git cherry can compare a branch with its upstream branch and find which commits have been upstreamed and which haven’t. This command is particularly clever because, thanks to git patch-id, it can correctly spot when a commit has been upstreamed, even when the upstreaming process resulted in changes to the commit message, line numbers, or whitespace.
• git rebase --onto can transplant a contiguous series of commits onto another branch.

Comments (6)

By Adam, August 23, 2013 7:00 pm

I’ve been a bit of a hermit the last few weeks, burning the candle both ends and spending the majority of my spare time building a new toy … well actually it started out as a toy, but now I think it’s good enough for musicians to use as a serious tool for improving their improvisation / compositional skills, and harmonic understanding.

So I’m very pleased (and relieved) to be able to announce … <drum roll> … the Scale Matcher!  It should work equally well on your computer, phone, and tablet.  Please try it out and let me know what you think!  You can also click the About and FAQ buttons to find out more.

Thanks to Barak Schmool for providing the original inspiration to do this, and for the time he spent testing it out and suggesting improvements.