Currently showing posts tagged: OpenStack

Report from the OpenStack PTG in Dublin

By , March 9, 2018 7:30 pm

Last week I attended OpenStack’s PTG (Project Teams Gathering) in Dublin. This event happens every 6 months in a different city, and is a fantastic opportunity for OpenStack developers and upstream contributors to get together and turbo-charge the next phase of collaboration.

I wrote a private report for my SUSE colleagues summarising my experience, but then Colleen posted her report publicly, which made me realise that it would be far more in keeping with OpenStack’s Four Opens to publish mine online. So here it is!

Continue reading 'Report from the OpenStack PTG in Dublin'»


Abstraction As A Service

By , December 19, 2017 7:55 pm

The birth of abstraction layers

The last five decades of computing have seen a gradual progression of architectural abstraction layers. Around 50 years ago, IBM mainframes gained virtualization capabilities. Despite explosive progress in the sophistication of hardware following Moore’s Law, there wasn’t too much further innovation in abstraction layers in server computing until well after the dawn of the microcomputer era, in the early 2000s, when virtualization suddenly became all the rage again. (I heard a rumour that this was due to certain IBM patents expiring, but maybe that’s an urban myth.) Different types of hypervisors emerged, including early forms of containers.

Then we started to realise that a hypervisor wasn’t enough, and we needed a whole management layer to keep control of the new “VM sprawl” problem which had arisen. A whole bunch of solutions appeared, including the concept of “cloud”, but many were proprietary, and so after a few years OpenStack came along to the rescue!

The cloud era

But then we realised that managing OpenStack itself was a pain, and someone had the idea that rather than building a separate management layer for managing OpenStack, we could just use OpenStack to manage itself! And so OpenStack on OpenStack, or Triple-O as it’s now known, was born.

Within and alongside OpenStack, several other new exciting trends emerged: Software-Defined Networking (SDN), Software-Defined Storage (e.g. Ceph), etc. So the umbrella term Software-Defined Infrastructure was coined to refer to this group of abstraction layers.

Continue reading 'Abstraction As A Service'»


Announcing OpenStack’s Self-healing SIG

By , November 24, 2017 4:15 pm

One of the biggest promises of the cloud vision was the idea that all infrastructure could be managed in a policy-driven fashion, reacting to failures and other events by automatically healing and optimising services.

In OpenStack, most of the components required to implement such an architecture already exist, and are nicely scoped, for the most part without too much overlap:

However, there is not yet a clear strategy within the community for how these should all tie together. (The OPNFV community is arguably further ahead in this respect, but hopefully some of their work could be applied outside NFV-specific environments.)

Designing a new SIG

To address this, I organised an unofficial kick-off meeting at the PTG in Denver, at which it became clear that there was sufficient interest in this idea from many of the above projects in order to create a new “Self-healing” SIG. However, there were still open questions:

  1. What exactly should be the scope of the SIG? Should it be for developers and operators, or also end users?
  2. What should the name be? Is “self-healing” good enough, or should it also include, say, non-failure scenarios like optimization?

Continue reading 'Announcing OpenStack’s Self-healing SIG'»


Cloud rearrangement for fun and profit

By , May 17, 2015 4:42 am

In a populated compute cloud, there are several scenarios in which it’s beneficial to be able to rearrange VM guest instances into a different placement across the hypervisor hosts via migration (live or otherwise). These use cases typically fall into three categories:

  1. Rebalancing – spread the VMs evenly across as many physical VM host machines as possible (conceptually similar to vSphere DRS). Example use cases:
  2. Consolidation – condense VMs onto fewer physical VM host machines (conceptually similar to vSphere DPM). Typically involves some degree of defragmentation. Example use cases:
  3. Evacuation – free up physical servers:

Whilst one-shot manual or semi-automatic rearrangement can bring immediate benefits, the biggest wins often come when continual rearrangement is automated. The approaches can also be combined, e.g. first evacuate and/or consolidate, then rebalance on the remaining physical servers.

Other custom rearrangements may be required according to other IT- or business-driven policies, e.g. only rearrange VM instances relating to a specific workload, in order to increase locality of reference, reduce latency, respect availability zones, or facilitate other out-of-band workflows or policies (such as data privacy or other legalities).

In the rest of this post I will expand this topic in the context of OpenStack, talk about the computer science behind it, propose a possible way forward, and offer a working prototype in Python.

If you’re in Vancouver for the OpenStack summit which starts this Monday and you find this post interesting, ping me for a face-to-face chat!

Continue reading 'Cloud rearrangement for fun and profit'»


Announcing git-deps: commit dependency analysis / visualization tool

By , January 19, 2015 12:15 am

I’m happy to announce a new tool called git-deps which performs automatic analysis and visualization of dependencies between commits in a git repository. Here’s a screencast demonstration!

Back in 2013 I blogged about some tools I wrote which harness the notes feature of git to help with the process of porting commits from one branch to another. These are mostly useful in the cases where porting is more complex than just cherry-picking a small number of commits.

However, even in the case where there are a small number of desired commits, sometimes those commits have hidden dependencies on other commits which you didn’t particularly want, but need to pull in anyway, e.g. in order to avoid conflicts during cherry-picking. Of course those secondary commits may in turn require other commits, and before you know it, you’re in dependency hell, which is only supposed to happen if you’re trying to install Linux packages and it’s still 1998 … but in fact that’s exactly what happened to me at SUSEcon 2013, when I attempted to help a colleague backport a bugfix in OpenStack Nova from the master branch to a stable release branch.

At first sight it looked like it would only require a trivial git cherry-pick, but that immediately revealed conflicts due to related code having changed in master since the release was made. I manually found the underlying commit which the bugfix required by using git blame, and tried another cherry-pick. The same thing happened again. Very soon I found myself in a quagmire of dependencies between commits, with no idea whether the end was in sight.

So wouldn’t it be nice if you could see the dependency tree ahead of time, rather than spending a whole bunch of time resolving unexpected conflicts due to missing dependencies, only to realise that the tree’s way deeper than you expected, and that actually a totally different approach is needed? Well, I thought it would, and so git-deps was born!

In coffee breaks during the ensuing openSUSE conference at the same venue, I feverishly hacked together a prototype and it seemed to work. Then normal life intervened, and no progress was made for another year.

However thanks to SUSE’s generous Hack Week policy, I have had the luxury of being able to spending some of early January 2015 working to bring this tool to the next level. I submitted a Hack Week project page, announced my intentions on the git mailing list, started hacking, missed quite a bit of sleep, and finally recorded the above screencast.

The tool is available here:

Please give it a go and let me know what you think! I’m particularly interested in hearing ideas for use cases I didn’t think of yet, and proposals for integration with other git web front-ends.


Panorama Theme by Themocracy