In a populated compute cloud, there are several scenarios in which it’s beneficial to be able to rearrange VM guest instances into a different placement across the hypervisor hosts via migration (live or otherwise). These use cases typically fall into three categories:
- Rebalancing – spread the VMs evenly across as many physical VM host machines as possible (conceptually similar to vSphere DRS). Example use cases:
- Optimise workloads for performance, by reducing CPU / I/O hotspots.
- Maximise headroom on each physical machine.
- Reduce thermal hotspots in order to reduce power consumption; for example, back in 2003, HP showed that intelligent workload placement can reduce energy consumption by more than 14%. (Here’s an old slidedeck I made years ago when researching this use case.)
- Consolidation – condense VMs onto fewer physical VM host machines (conceptually similar to vSphere DPM). Typically involves some degree of defragmentation. Example use cases:
- Increase CPU / RAM / I/O utilization. CERN blogged last summer about this Tetris-like challenge, and it can be taken even further over-committing CPU/RAM and/or memory page sharing.
- Free up physical servers to reduce power consumption.
- Evacuation – free up physical servers:
- for repurposing or decommissioning
- for maintenance (BIOS upgrades, re-cabling etc.) or repair
- to protect SLAs, e.g. when monitors indicate potential imminent hardware failure, or when HVAC failures are likely to cause servers to shutdown due to over-heating. (This is different to the somewhat confusingly named
nova evacuatefunctionality which moves VMs to a new host after the server has already failed.)
Whilst one-shot manual or semi-automatic rearrangement can bring immediate benefits, the biggest wins often come when continual rearrangement is automated. The approaches can also be combined, e.g. first evacuate and/or consolidate, then rebalance on the remaining physical servers.
Other custom rearrangements may be required according to other IT- or business-driven policies, e.g. only rearrange VM instances relating to a specific workload, in order to increase locality of reference, reduce latency, respect availability zones, or facilitate other out-of-band workflows or policies (such as data privacy or other legalities).
In the rest of this post I will expand this topic in the context of OpenStack, talk about the computer science behind it, propose a possible way forward, and offer a working prototype in Python.
If you’re in Vancouver for the OpenStack summit which starts this Monday and you find this post interesting, ping me for a face-to-face chat!