Comments (0)

By Adam, March 9, 2018 7:30 pm

Last week I attended OpenStack’s PTG (Project Teams Gathering) in Dublin. This event happens every 6 months in a different city, and is a fantastic opportunity for OpenStack developers and upstream contributors to get together and turbo-charge the next phase of collaboration.

I wrote a private report for my SUSE colleagues summarising my experience, but then Colleen posted her report publicly, which made me realise that it would be far more in keeping with OpenStack’s Four Opens to publish mine online. So here it is!

Continue reading 'Report from the OpenStack PTG in Dublin'»

Comments (0)

By Adam, November 24, 2017 4:15 pm

One of the biggest promises of the cloud vision was the idea that all infrastructure could be managed in a policy-driven fashion, reacting to failures and other events by automatically healing and optimising services.

In OpenStack, most of the components required to implement such an architecture already exist, and are nicely scoped, for the most part without too much overlap:

• Monasca: monitoring
• Aodh: alarming
• Congress: policy-based governance
• Mistral: workflow
• Senlin: clustering service
• Vitrage: root cause analysis
• Watcher: optimization
• Masakari: compute plane HA
• Freezer-dr: compute plane HA
• Heat: orchestration (normally used for cloud applications, but could also potentially auto-heal cloud infrastructure via TripleO)
• Doctor: fault management and maintenance for OPNFV
• Fault Genes Working Group: Fault classification & Recovery Strategy
• Craton: Fleet management (currently stalled)

However, there is not yet a clear strategy within the community for how these should all tie together. (The OPNFV community is arguably further ahead in this respect, but hopefully some of their work could be applied outside NFV-specific environments.)

Comments (2)

By Adam, August 16, 2017 6:45 pm

(Thanks to Ben North, Colleen Murphy, and Nicolas Bock for reviewing earlier drafts of this post.)

In April 2016, GitHub announced a new feature supporting squashing of multiple commits in a PR at merge-time (announced on April 1st, but it was actually bona-fide 😉 ).

I appreciate that there was high demand for this feature (and similarly on GitLab), and apparently that many projects have a “squash before submitting a PR” policy, but I’d like to contend that this is a poor-man’s workaround for the lack of a real solution to the underlying problems.

Comments (1)

By Adam, May 17, 2015 4:42 am

In a populated compute cloud, there are several scenarios in which it’s beneficial to be able to rearrange VM guest instances into a different placement across the hypervisor hosts via migration (live or otherwise). These use cases typically fall into three categories:

1. Rebalancing – spread the VMs evenly across as many physical VM host machines as possible (conceptually similar to vSphere DRS). Example use cases:
   • Optimise workloads for performance, by reducing CPU / I/O hotspots.
   • Maximise headroom on each physical machine.
   • Reduce thermal hotspots in order to reduce power consumption; for example, back in 2003, HP showed that intelligent workload placement can reduce energy consumption by more than 14%. (Here’s an old slidedeck I made years ago when researching this use case.)
2. Consolidation – condense VMs onto fewer physical VM host machines (conceptually similar to vSphere DPM). Typically involves some degree of defragmentation. Example use cases:
   • Increase CPU / RAM / I/O utilization. CERN blogged last summer about this Tetris-like challenge, and it can be taken even further over-committing CPU/RAM and/or memory page sharing.
   • Free up physical servers to reduce power consumption.
3. Evacuation – free up physical servers:
   • for repurposing or decommissioning
   • for maintenance (BIOS upgrades, re-cabling etc.) or repair
   • to protect SLAs, e.g. when monitors indicate potential imminent hardware failure, or when HVAC failures are likely to cause servers to shutdown due to over-heating. (This is different to the somewhat confusingly named nova evacuate functionality which moves VMs to a new host after the server has already failed.)

Whilst one-shot manual or semi-automatic rearrangement can bring immediate benefits, the biggest wins often come when continual rearrangement is automated. The approaches can also be combined, e.g. first evacuate and/or consolidate, then rebalance on the remaining physical servers.

Other custom rearrangements may be required according to other IT- or business-driven policies, e.g. only rearrange VM instances relating to a specific workload, in order to increase locality of reference, reduce latency, respect availability zones, or facilitate other out-of-band workflows or policies (such as data privacy or other legalities).

In the rest of this post I will expand this topic in the context of OpenStack, talk about the computer science behind it, propose a possible way forward, and offer a working prototype in Python.

If you’re in Vancouver for the OpenStack summit which starts this Monday and you find this post interesting, ping me for a face-to-face chat!

Continue reading 'Cloud rearrangement for fun and profit'»