Yesterday, I went to the VMWare best Practices Forum to learn more about people using virtualization in the real world. I learned some new stuff:
VMotion complexity. Man, it’s complex! To use VMotion, you need a machine with Fiber Channel SAN (no iSCSI yet) and at least 2 (3 is recommended) network interfaces. Plus, not all the hardware that can run ESX is compatible with VMotion. Contrary to the expectations, you cannot always move a VM from one ESX machine to another ESX machine – they must have compatible chipsets.
ESX Overhead. VMWare customers were invited to tell a success story. They consolidated their old servers, running 1 app per CPU with barely 5% utilization, and achieved 40-60% utilization. The resultant density is 3-4 apps per CPU on the target machine. Well, 4 apps at 5% each accounts only for 20% of CPU utilization. That means that VMWare itself used 20-40% of the capacity. VMWare takes up more resources than the useful workload. Am I missing something?
Performance monitoring does not work. It turned out that performance monitoring tools (Windows performance counters and PERFMON) do not work correctly inside VM if you try to use them to monitor your virtual hardware. You need to use different tools. Overall, from what I learned, there are a lot of cases, when virtual machine needs to be handled differently than physical machine. A virtualization abstraction, at least as implemented by VMWare, is a fairly leaky abstraction – be careful with your assumptions.
Well, in any case, VMWare is still a very respectable mature technology. Even if it requires more resources than the useful workload that it runs, it still provides very significant savings to its users. Naturally, I wonder how much money could be saved using Virtuozzo, which provides much lower overhead and much better manageability.