However, there’s some good news on this front. Intel and AMD realised this was a problem, and have both introduced new hardware capabilities to help improve the level to which cloud users can trust the environment in which their workloads are executed, e.g.:

• AMD SEV (Secure Encrypted Virtualization) which can encrypt the memory of a running VM with a key which is only accessible to the owner of that VM. This is done on-chip so that even if you have physical access to the machine, it makes it a lot harder to snoop in on the running VM¹.

  It can also provide the guest owner with an attestation which cryptographically proves that the memory was encrypted correctly and can only be decrypted by the owner.

• Intel MKTME (Multi-Key Total Memory Encryption) which is a similar approach.

But even with that hardware support, there is the question to what degree anyone can trust public clouds run on proprietary technology. There is a growing awareness that Free (Libre) / Open Source Software tends to be inherently more secure and trustworthy, since its transparency enables unlimited peer review, and its openness allows anyone to contribute improvements.

And these days, OpenStack is pretty much the undisputed king of the Open Source cloud infrastructure world.

So I’m delighted to be able to announce a significant step forward in trustworthy cloud computing: as of this week, OpenStack is now able to launch VMs with SEV enabled! (Given the appropriate AMD hardware, of course.)

The core functionality is all merged and will be in the imminent Train release. You can read the documentation, and you will also find it mentioned in the Nova Release Notes.

While this is “only” an MVP and far from the end of the journey (see below), it’s an important milestone in a strong partnership between my employer SUSE and AMD. We started work on adding SEV support into OpenStack around a year ago:

This resulted in one of the most in-depth technical specification documentations I’ve ever had to write, plus many months of intense collaboration on the code and several changes in design along the way.

I’d like to thank not only my colleagues at SUSE and AMD for all their work so far, but also many members of the upstream OpenStack community, especially the Nova team. In particular I enjoyed fantastic support from the PTL (Project Technical Lead) Eric Fried, and several developers at Red Hat, which I think speaks volumes to how well the “coopetition” model works in the Open Source world.

The rest of this post gives a quick tour of the implementation via screenshots and brief explanations, and then concludes with what’s planned next.

OpenStack’s Compute service (nova) will automatically detect the presence of the SEV feature on any compute node which is configured to support it. You can optionally configure how many slots are available on the memory controller for encryption keys. One is used for each guest, so this effectively acts as the maximum number of guest VMs which can concurrently use SEV. Here you can see the configuration of this option, and how nova handles the inventory. Note that it also registers an SEV trait on the compute host, so that in the future if the cloud has a mix of hardware offering different guest memory encryption technologies, you’ll be able to choose which one you want for any given guest, if you need to.

SEV can be enabled by the operator by adding a new hw:mem_encryption “extra spec” which is a property on nova’s flavors. As already shown in the screenshot above, this can be done through Horizon, OpenStack’s web dashboard. However it can also be set per-image via a similarly-named property hw_mem_encryption:

and of course this can all be done via the command-line too:

Notice the presence of a few other image properties which are crucial for SEV to function correctly. (These are explained fully in the documentation.)

Once booted, an SEV VM instance looks and behaves pretty much like any other OpenStack VM:

However there are some limitations, e.g. it cannot yet be live-migrated or suspended:

Behind the scenes, nova takes care of quite a few important details in how the VM is configured in libvirt. Firstly it performs sanity checks on the flavor and image properties. Then it adds a crucial new <launchSecurity> element:

and also enables IOMMU for virtio devices:

This area of technology is new and rapidly evolving, so there is still plenty of work left to be done, especially on the software side.

Of course we’ll be adding this functionality to SUSE OpenStack Cloud, initially as a technical preview for our customers to try out.

Probably the most important feature needed next on the SEV side is the ability to verify the attestation which cryptographically proves that the memory was encrypted correctly and can only be decrypted by the owner. In addition specification of the work required to add support to OpenStack for Intel’s MKTME already started, so I would expect that to continue.

The PTG was well-attended with just over 400 people from all over the world. As with previous PTGs this felt like just the right size: enough to have most projects / organizations well represented, but not too big to prevent serendipitous meetings happening many times throughout the week, where I would just be wandering around and bump into exactly the right person I needed to talk to 🙂

All the etherpads are publicly available in case you want to dive into the gory details.

IMHO the ptgbot continued to work really well for adapting to the dynamic and fast-changing agenda. I submitted a few patches to polish it further.

The “extreme” (for Ireland, anyway) weather, which was reportedly the heaviest snow in 40 years, caused a lot of disruption, not just for travel, but also that our conference venue in Croke Park Stadium was forced to shut on Thursday afternoon, rather than Friday evening as originally planned. Luckily our amazing hotel, The Croke Park, stepped up to the plate and allowed us to continue the conference there. It was quite a squash fitting everyone in, but somehow they managed it, and the ensuing organized mayhem had the unexpected side effect of building really strong comradery between everyone.

There was a piano-bar style sing-along night (with OpenStack’s Executive Director Jonathan Bryce doing a great job on the piano), a board games night … I think it was my colleague Keith Berger who shrewdly observed that it was like being on a cruise ship 🙂 Having the best Guinness in the world on tap probably helped too. Check out #SnowpenStack on Twitter for some of the hilarious antics.

We managed a couple of SUSE team dinners and it was great to spend more time with teammates, especially the ones who are normally on a different continent 🙂

• https://etherpad.openstack.org/p/self-healing-ptg-rocky

My main reason for attending the PTG was bootstrapping this new SIG which I founded a few months ago. We had a really good session filling Tuesday afternoon, with about 30 people attending.

There were discussions about various use cases, and some people are already keen to start work on use cases immediately. We also had representatives from the API SIG, including Graham Hayes a.k.a. mugsie who is leading the new health-check API initiative, and from OPNFV, since we are trying to promote better sharing of work on automated testing of self-healing / HA functionality. (If you’re interested, please sign up here!)

We agreed some concrete next steps, some of which I already started working on:

• Populate our self-healing-sig repo with the standard Sphinx stuff and templates for use cases and specs
• Compile a list of contact info for self-healing stakeholders across the various projects / organizations
• Start to visualize the architectural integration points between involved projects.
• Start to document use cases and write specs for proposed work.

You can follow all activity via the StoryBoard project.

The only disappointment with this session was that no operators attended. However I managed to coordinate with some people involved with the Tokyo Ops meetup this week, in order to gather feedback from ops, and we got a session slot booked in that agenda which happened yesterday. Additionally I am speaking on self-healing and leading a discussion session on it at the London OpenStack Meetup next Monday, so hopefully we’ll get some good feedback there too.

• https://etherpad.openstack.org/p/FC_SIG_Rocky_PTG

I joined the contributor guide part of this session briefly to give some feedback and also organise collaboration on writing a quickstart guide for people who are used to GitHub and don’t know Gerrit, since I promised to write that after creating a comparison matrix in Denver, which provides a base for the guide.

The Self-healing SIG previously agreed to use StoryBoard for tracking work, not just to aid collaboration within the SIG, but also to help dogfood StoryBoard and provide feedback. So I spent quite a bit of time with the StoryBoard folks giving feedback and brainstorming ideas for improving usability. I also started work on a “Rosetta stone” matrix which aims to show the key differences between StoryBoard and other issue trackers people are used to, such as JIRA, Trello, GitHub, Bugzilla, Redmine etc. Similarly to the approach with the GitHub / Gerrit matrix above, it could serve as the basis for a document which helps “port” people to StoryBoard from other issue trackers.

• https://etherpad.openstack.org/p/ffu-ptg-rocky

This was a really full session, with some good discussions. Main points:

• It was agreed to form an Upgrade SIG, and this has now happened.
• AFAICS everyone is not yet on the same page about what FFU means exactly, especially regarding online migrations. There is perhaps room for improvement in the governance tags provided for tracking upgradability (assert:supports-upgrade, assert:supports-accessible-upgrade), which would help clarify this.
• IIUC Kolla seem to be going in a slightly different direction based on online rolling upgrades, although I don’t think they necessarily have any strong objections to supporting offline upgrades so maybe things will converge in the long term.
• Noone from openstack-helm present 🙁
• Lots of gotchas, and consequent debates on how best to capture and share these.

• https://etherpad.openstack.org/p/oslo-ptg-rocky

I only had time to pop into the oslo room briefly, to discuss an interesting new initiative in oslo.config to automate migration of config between releases (a bit like DB migrations, but for oslo.config instead of DB schema). This was also mentioned in the FFU session.

• gerrit#I851628c0 (Propose spec for handle config changes in code)
• gerrit#I0a14a4a4 (Handle config mapping changes)
• gerrit#I0b95d3ea ({WIP}{POC} Handle config mapping changes)

I posed some questions about how this would handle config which requires extra per-project logic to migrate, and also how it would handle config which has been explicitly set to the default value. We had some interesting discussions on that but I think ultimately the answer was “let’s cross that bridge when we come to it”.

I also had a quick chat with Ken Giusti about a long-standing oslo.service bug we’ve worked on, but we ran out of time and agreed to take it online.

I demo’d ansible-viz, the Ansible visualization tool of which I recently became the maintainer, to the OpenStack Ansible folks. They seemed really enthusiastic, and are looking into using it for embedding diagrams in their docs. Hopefully this means they will start contributing to make it better!

My colleague Dirk Müller and I also talked to them in general about other areas for potential collaboration. We were both impressed with how open they are to help and collaborate!

As usual with the PTGs, I got huge value from random unplanned conversations in hallways, at lunch tables, in the bar etc.

I spent a lot of time talking to Emmet Hickory (a.k.a. persia) and he gave me some great ideas about new use cases for the git workflow tools I have been building in hackweeks and spare time. I already knew that my git-deps tool allows fairly accurate predictions of how difficult it will be to backport a given commit to a stable branch, but a new idea emerged that this could be added as a non-voting job to Zuul‘s checks pipeline, to facilitate a more proactive / preemptive approach to backporting.

I talked with Tony Breeds (PTL of stable branch maintenance) about this who told me that Neutron is already extremely proactive when it comes to backporting, so they might be interested in this. So I met Miguel Lavalle (Neutron PTL) and he was really interested and requested that I put it on the agenda for a future Neutron IRC meeting. This has now happened and we will discuss during next Tuesday’s meeting.

Emmet helped me realise that my git-explode tool could also help with backporting work, so I need to finish packaging that and publish it. There is a London Git meetup coming up soon, and I might present the tools at that.

I’m getting off-topic now so I’ll finish here 🙂 If you found this report useful please leave a comment below, so that I know it’s worth bothering to do next time 🙂

But then we realised that managing OpenStack itself was a pain, and someone had the idea that rather than building a separate management layer for managing OpenStack, we could just use OpenStack to manage itself! And so OpenStack on OpenStack, or Triple-O as it’s now known, was born.

Within and alongside OpenStack, several other new exciting trends emerged: Software-Defined Networking (SDN), Software-Defined Storage (e.g. Ceph), etc. So the umbrella term Software-Defined Infrastructure was coined to refer to this group of abstraction layers.

Whilst OpenStack was busy growing up and moving past the Peak of Inflated Expectations, all of a sudden Docker and containers burst onto the scene and provided a lot of new buzzwords to get everyone excited again. But after the excitement started to fade, that familiar sinking feeling came back with the realisation that just like VMs, containers need something to manage them.

But then Kubernetes leapt in to the rescue! And all the excitement returned. Except that of course then you need something to manage Kubernetes, but fortunately we already had OpenStack, so we could just use that! And so Magnum was born. And since Kubernetes is so awesome, we realised that we could also use it as the basis for deploying OpenStack. At the recent OpenStack Summit in Sydney, we saw the continued rise in popularity of running both Kubernetes on OpenStack, and OpenStack on Kubernetes.

But that still leaves the pesky job of managing raw hardware to put all this stuff on top. Fortunately there are services you can pay for so that other people do that for you, and they even have APIs you can hook into! This is called public cloud. And even better, many companies use OpenStack to drive their public clouds.

So now we’re running Kubernetes on OpenStack on Kubernetes on OpenStack. And now we’re done! Right?

Well … if we’re to learn anything from this history, it should be that we’ll always find more good reasons for new abstraction layers. I mean, there’s already a huge amount of work going into things like Cloud Foundry on the PaaS layer, NFV in the telco space, serverless computing, … And I hear that Kubernetes is a great platform for running Cloud Foundry, just like it is for OpenStack. And wouldn’t OpenStack be a cool platform to provide inside Cloud Foundry, e.g. for people who just want to quickly try it out? So who knows, maybe in the next few years we’ll have OpenStack on Cloud Foundry on Kubernetes on OpenStack on Kubernetes on OpenStack.

Of course this starts getting a bit unwieldly. Every time we introduce a new abstraction layer there’s extra complexity to deal with. But that’s OK, because we can always deal with complexity by abstracting it away! It’s a bit like the cyber-equivalent of delegating difficult tasks to someone else. So I’d like to propose a new concept, and corresponding meta-component of the overall architecture:

Whenever we realise we need a new abstraction layer, rather than having to deal with the complexity of deploying and managing this layer, we could just invoke APIs to a central service which takes care of this complexity for us. We could call this new concept (drum roll, please…) Abstraction As A Service, or AaaS. (If you’re British or Australian you may prefer to pronounce this as if there was an “r” in the middle, to distinguish from similar-sounding existing words such as “as”. Or maybe we should call it Sweeping Stuff Under The Carpet As A Service? Or SSUtCaaS for short, which can be pronounced “suitcase” (thanks to Florian for pointing this out).

With AaaS, if we wanted say, NFV on serverless on CF on Kubernetes on OpenStack on Kubernetes on OpenStack on public cloud on COBOL, we could simply write some declarative YAML or JSON describing the stack we want, push it to the AaaS REST API endpoint via an HTTP POST, and it would set the whole thing up for us automatically. We could build any number and combination of abstraction layers we would possibly need, so at this point the job could be considered well and truly done, once and for all!

Except, ya know, we’d need a way to deploy and manage our AaaS service, of course. Maybe we could build an AaaSaaS service for that …

P.S. I know, it’s a terrible joke if you have to explain it, but based on previous experiences of my dry British humour being misunderstood (especially given the inevitably international audience), I feel the need to point out that this blog post was intended as nothing more than poking a bit of gentle fun at the cloud software industry. I’m a huge fan of all the technologies mentioned here, and yes, I’m even in favour of multiple abstraction layers, despite occasionally wondering if we’ve all gone a bit insane 😉 Thanks to Florian, Andrew, and Dirk for reviewing an earlier draft, but I take responsibility for any mistakes or any offence unintentionally caused!

To address this, I organised an unofficial kick-off meeting at the PTG in Denver, at which it became clear that there was sufficient interest in this idea from many of the above projects in order to create a new “Self-healing” SIG. However, there were still open questions:

1. What exactly should be the scope of the SIG? Should it be for developers and operators, or also end users?
2. What should the name be? Is “self-healing” good enough, or should it also include, say, non-failure scenarios like optimization?

In an attempt to answer these, I formally proposed the creation of the SIG, asking the community to fill in a short survey to vote on its creation, and to provide their feedback regarding the name and scope. Unfortunately whilst everyone unanimously supported its creation, opinions were split more or less 50%-50% on the name and the scope! So on advice from Thierry, I listed the SIG as “forming”, created the corresponding wiki page, and proposed a session for the Sydney Forum, which was subsequently accepted.

We had around 30 people attend the Sydney Forum session, which was extremely encouraging! You can read more details in the etherpad, but here is the quick summary …

Most importantly, we resolved the naming and scoping issues, concluding that to avoid biting off too much in one go, it was better to be pragmatic and start small:

• Initially focus on cloud infrastructure, and not worry too much about the user-facing impact of failures yet; we can add that concern whenever it makes sense (which is particularly relevant for telcos / NFV).
• Not worry too much about optimization initially; Watcher is possibly the only project focusing on this right now, and again we can expand to include optimization any time we want.

So now that the naming and scoping issues are resolved, I am excited to announce that the Self-healing SIG is officially formed!

Discussion went beyond mere administravia, however:

• We collected a few initial use cases.
• We informally decided the governance of the SIG. I asked if anyone else would like to assume leadership, but noone seemed keen, dashing my hopes of avoiding extra work 😉 But Eric Kao, PTL of Congress, generously offered to act as co-chair.
• We discussed health check APIs, which were mentioned in at least 2 or 3 other Forum sessions this time round.
• We agreed that we wanted an IRC channel, and that it could host bi-weekly meetings. However as usual there was no clean solution to choosing a time which would suit everyone ;-/ I’ll try to figure out what to do about this!

You are warmly invited to join, if this topic interests you:

• Ensure you are subscribed to the openstack-sigs mailing list, and watch out for posts tagged [self-healing].
• Bookmark the wiki page which is the SIG’s official home.

I have sent out a similar announcement to the mailing list, and next will set up the IRC channel, and see if we can make progress on agreeing times for regular IRC meetings.

Other than this administravia, it is of course up to the community to decide in which direction the SIG should go, but my suggestions are:

• Continue to collect use cases. It makes sense to have a very lightweight process for this (at least, initially), so Eric has created a Google Doc and populated it with a suggested template and a first example. Feel free to add your own based on this template.
• Collect links to any existing documentation or other resources which describe how existing services can be combined. This awesome talk on Advanced Fault Management with Vitrage and Mistral is a perfect example, and here is another, but we need to make it easier for operators to understand which combinations like this are possible, and easier for them to be set up.
• Finish the architecture diagram drafted in Denver.
• At a higher level, we could document reference stacks which address multiple self-healing cases.
• Talk more with the OPNFV community to find out what capabilities they have which could be reused within non-NFV OpenStack clouds.
• Perform gaps analysis on the use cases, and liase with specific projects to drive development in directions which can address those gaps.

In case you’re interested in the history …

I first became aware of the need for this SIG while working upstream within the community on OpenStack HA – specifically on compute plane HA, where failures of compute nodes or hypervisors are automatically handled by resurrecting affects VMs on other compute nodes. I saw many groups independently trying to solve the same problem, so I created the #openstack-ha IRC channel, organised weekly meetings, and tried to bring all stakeholders together to converge on a single upstream solution. Progress was gradually made, which we presented in Austin, Boston, and most recently in Tel Aviv at OpenStack Day Israel 2017.

After the talk I had a great conversation with Ifat Afek, who is the PTL of OpenStack Vitrage, which is an awesome project providing RCA (Root Cause Analysis) of faults within OpenStack. Since Vitrage can do things like receive an alert about a fault on a compute node (e.g. from Aodh) and then automatically determine all affected VMs and call out to another service like Mistral to enact appropriate remediation, there was obvious synergy between our work.

However Vitrage goes much further than just compute HA: since it can receive various alerts from multiple types of data source, model relationships between many types of resource, and trigger external services to take action, this kind of combination has tremendous potential for building automatically self-healing cloud infrastructure. And as shown above, there are several other OpenStack projects operating in the same space, which could take this approach further; for example, Congress can be used to specify policies regarding how failures should be handled.

Talking with Ifat resulted in the idea to create a new SIG with the goals of identifying self-healing use cases, establishing and documenting what can already be achieved by combining existing OpenStack services, and enhancing collaboration between the projects and with operators to fill in any remaining gaps. And now you know the rest of the story 🙂

It is clear from the diversity of the use cases listed above that VM placement policies are likely to vary greatly across clouds, and sometimes even within a single cloud. OpenStack Compute (nova) has fairly sophisticated scheduling capabilities which can be configured to implement some of the above policies on an incremental basis, i.e. every time a VM instance is started or migrated, the destination VM host can be automatically chosen according to filters and weighted cost functions. However, this approach is somewhat limited for migration, because the placement policies are only considered one migration at a time. Like in Tetris, more efficient arrangements can be achieved by thinking further ahead!

OpenStack clouds can already be segregated in various ways via cells, regions, availability zones, host aggregates, and server groups, and so there are various techniques available for implementing different placement policies to different areas of your cloud. For example you could have groups of hosts dedicated to CPU-intensive or I/O-bound workloads using an anti-affinity placement policy, and other groups dedicated to light workloads using an affinity placement policy. But these policies are static in nature (e.g. nova filters and schedulers are configured by the cloud administrator in nova.conf), which somewhat limits their flexibility.

OpenStack is still relatively new, however with Cinder and Neutron rapidly evolving, it is at the point where a VM’s network and storage dependencies can be live migrated along with the workload in a near seamless fashion. So now we should be able to start developing mechanisms for implementing more sophisticated placement policies, where not only is VM rearrangement performed automatically, but the policies themselves can be varied dynamically over time as workload requirements change.

VMware advocates might instinctively recommend running vSphere clusters underneath nova, to harness vSphere’s pre-existing DRS / DPM functionality. However, there are still unresolved problems with this approach, as this article on nova-scheduler and DRS highlights. (Funnily enough, I encountered exactly these same problems a few years ago when I was part of a team adding vSphere integration into the orchestration component of what has since become NetIQ Cloud Manager …) Besides, a vSphere cluster and an OpenStack cloud are very different beasts, and of course not everyone wants to build their cloud exclusively with VMware technology anyway.

Unfortunately developing algorithms to determine optimal placement is distinctly non-trivial. For example, the consolidation scenario above is a complex variant of the bin packing problem, which is NP-hard. The following constraints add significant complexity to the problem:

• A useful real world solution should take into account not only the RAM footprint of the VMs, but also CPU, disk, and network.
• The algorithm needs to ensure that SLAs are maintained whilst any rearrangement takes place.
• If the cloud is too close to full capacity, it may not be possible to rearrange the VMs from their current placement to a more optimal placement without first shutting down some VMs, which could be prohibited by the SLAs.
• Even if the cloud is at low utilization, live migration is not always possible, e.g.
  • the hypervisor may not support it (especially across non-identical CPU architectures)
  • shared storage may be required but unavailable
  • the network may not support it
• Even if the arrangement is achievable purely via a sequence of live migrations, the algorithm must also be sensitive to the performance impact to running workloads when performing multiple live migrations, since live migrations require intensive bursts of network I/O in order to synchronize the VM’s memory contents between the source and target hosts, followed by a momentary freezing of the VM as it flips from the source to the target. This trade-off between optimal resource utilization and service availability means that a sub-optimal final placement may be preferable to an optimal one.
• In the case where the hypervisor is capable of sharing memory pages between VMs (e.g. KSM on KVM), the algorithm should try to place together VMs which are likely to share memory pages (e.g. VMs running the same OS platform, OS version, software libraries, or applications. A research paper published in 2011 demonstrated that VM packing which optimises placement in this fashion can be approximated in polytime, achieving 32% to 50% reduction in servers and a 25% to 57% reduction in memory footprint compared to sharing-oblivious algorithms.

(There is prior art of course. For example, several years ago, Pacemaker implemented a best effort algorithm for VM host selection.)

As noted by the 2011 research paper referenced above, this area of computer science is still evolving. There is one constant however: any rearrangement solution must not only provide a final VM placement optimised according to the chosen constraints, but also a sequence of migrations to it from the current placement. There will often be multiple migration sequences reaching the optimised placement from the current one, and their efficiency can vary widely. In other words, there are two questions which need answering:

1. Given a starting placement A, which is the best (or near-optimal) final placement B to head for?
2. What’s the best way to get from A to B?

The above considerations strongly suggest that the first question is much harder to answer than the second, although the two are not necessarily orthogonal; for example there could be two different final placements B and C which are equally near-optimal, but it may be much harder to reach C from A than B.

I propose that it is worth examining the effectiveness of a divide and conquer approach: solving the second question may simplify the first, and also provide a mechanism for comparatively evaluating the effectiveness of potential answers to the first. Another bonus of this decoupling is that it should be possible for the path-finding algorithm to also discover opportunities for parallelizing live migrations when walking the path, so that the target placement B can be reached more quickly.

I’ve designed an algorithm which solves the second problem, and hacked together a proof of concept implementation in Python. This video shows it solving three randomly generated scenarios:

It’s far from perfect, but as a proof of concept it seems to hold up fairly well under soak testing. I’m pleased to announce that the code is now on github: https://github.com/aspiers/migrationpaths and I would love to hear feedback on it.

If you found this blog interesting, feel free to get in touch! I’m aspiers on FreeNode IRC, and adamspiers on Twitter.

I’m also in Vancouver for the OpenStack summit which starts this Monday, so if you are, come and say hello!

Many thanks to Florian Haas, Steven Hardy, and Christoph Thiel for their input on an earlier draft version of this post, and apologies that it took me SO long to publish it!