Comments (3)

By Adam, November 5, 2014 7:32 pm

Today at the OpenStack summit here in Paris, we continued SUSE’s winning streak at the “Ruler of the Stack” competition 🙂

This time, the challenge from the OpenStack Foundation was to deploy an OpenStack cloud as quickly as possible which could withstand “attacks” such as one of the controller nodes being killed, and still keep the OpenStack services and VM instances functional.

It was considerably more complex than the challenge posed by Intel at the Atlanta summit in May which my teammate Dirk Müller (pictured right) won, since now we needed a minimum of two controller nodes clustered together, two compute nodes, and some shared or replicated storage. So this time an approach deploying to a single node via a “quickstart” script was not an option, and we knew it would take a lot longer than 3 minutes 14 seconds.

However as soon as I saw the rules I thought we had a good chance to win with SUSE Cloud, for four reasons:

1. We already released the capability to automatically deploy highly available OpenStack clouds back in May, as an update to SUSE Cloud 3.¹
2. With openSUSE’s KIWI Image System and the Open Build Service, our ability to build custom appliance images of product packages for rapid deployment is excellent, as already proved by Dirk’s victory in Atlanta. KIWI avoids costly reboots by using kexec.
3. I have been working hard the last couple of months on reducing the amount of manual interaction during deployment to the absolute minimum, and this challenge was a perfect use case for harnessing these new capabilities.²
4. The solution had to work within a prescribed set of VLANs and IP subnets, and Crowbar (the deployment technology embedded in SUSE Cloud) is unique amongst current OpenStack deployment tools for its very flexible approach to network configuration.

I worked with Dirk late in our hotel the night before, preparing and remotely testing two bootable USB images which were custom-built especially for the challenge: one for the Crowbar admin server, and one for the controller and compute nodes.

The clock was started when we booted the first node for installation, and stopped when we declared the nodes ready for stress-testing by the judges. It took us 53 minutes to prepare 6 servers: two admin servers (one as standby), two controllers, and two compute nodes running the KVM hypervisor.³ However we lost ~15 minutes simply through not realising that plugging a USB device directly into the server is far more performant than presenting virtual media through the remote console! And there were several other optimizations we didn’t have time to make, so I think in future we could manage the whole thing under 30 minutes.

However the exercise was a lot of fun and also highlighted several areas where we can make the product better.

At least three other well-known combinations of OpenStack distribution and deployment tools were used by other competitors. No-one else managed to finish deploying a cloud, so we don’t know how they would have fared against the HA tests. Perhaps everyone was too distracted by all the awesome sessions going on at the same time 😉

I have to send a huge thank-you to Dirk whose expertise in several areas, especially building custom appliances for rapid deployment, was a massive help. Also to Bernhard, Tom, and several other members of our kick-ass SUSE Cloud engineering team 🙂 And of course to Intel for arranging a really fun challenge. I’m sure next time they’ll think of some great new ways to stretch our brains!

Comments (0)

By Adam, November 1, 2014 7:39 pm

6 months after its debut in Atlanta, the HA workshop happening again in Paris, this Monday (16:20, Room 241)! If you plan on attending and didn’t already see Florian’s blog post, please get downloading and installing the prerequisite files quick, because downloading them over hotel or conference wifi is likely to be painful. (N.B. Unless you have a really strong reason not to, please go with the VirtualBox option, because there are more pitfalls when using KVM+libvirt.)

However if you’re already on the way to Paris, don’t despair, because we’ll also do our best to make the files available at the SUSE booth. And failing that, you can still just turn up, enjoy the show, and then try the hands-on exercise any time later at your own leisure!

In case you have no idea what I’m talking about, here’s some brief history:

Back before the last OpenStack summit in Atlanta in May, I managed to persuade Florian Haas to join me in an endeavour which some might view as slightly insane: run a 90 minute hands-on workshop in which we’d have every attendee build a Highly Available OpenStack cloud on their laptop, from scratch.

With so many moving parts and a vast range of different hardware involved, it certainly wasn’t plain sailing, but by the power of Vagrant and VirtualBox, I think overall it was a success.

Since then, we’ve been working extremely hard to improve the workshop materials based on lessons learnt last time. So what’s new? Well, firstly the workshop environment has been upgraded from Havana Icehouse (SUSE Cloud 4 vs. version 3 in May), and quite a lot of polish has been applied, since in May the HA code was still relatively new.

Secondly, we’ve added support for building the cloud using KVM+libvirt (although VirtualBox is still recommended for a smoother ride).

Thirdly, the documentation is way more comprehensive, and should help you avoid many common pitfalls.

Hope to see you at the workshop, and please come and say hello to us at the SUSE booth!