February 7th, 2007

About benchmarking

As far as I know, VMware still does not allow 3rd parties to benchmark their product. Fortunately, VMware has benchmarked their own product and made the results publicly available. Here (http://blogs.vmware.com/performance/2007/02/a_performance_c.html) and here (http://www.vmware.com/pdf/hypervisor_performance.pdf) you can find a comparison of VMware ESX and Xen.

I applaud VMware for publishing industry-first comparative virtualization benchmark. Here are my comments that I hope will help VMware to make the comparison more accurate:

·         Take the latest version of Xen – Xen 3.0.4 has been around for a while.

·         Include Linux guests. Xen has never claimed any decent support for Windows. As all of virtualization-savvy people know, under Xen approach (para-virtualization), achieving maximum performance would require modifying source code of Windows, which only Microsoft can do. The only version of Xen that is supposed to provide decent Windows performance is Xen Enterprise 3.1 running on VT-enabled CPUs.

·         Use Xen Enterprise on VT-enabled hardware. It is Xen Enterprise (not the open source version) and ESX (not VMware Server or Workstation or Player) that are positioned for enterprise workloads.

·         Use multiple guests and Virtual SMP. The paper says that Xen could not boot SMP Windows and it was not possible to run multiple guests under Xen. Well, I’m sure if Xen Enterprise was used, some of the problems would go away. I’m also sure that they could call Xen tech support and ask for help J.

·         Limit the “performance comparison of the hypervisors” to performance (not features or manageability) and hypervisors (the lowest-level components in V-stack). It may be true that Xen suffers from “the lack of such RAS, scalability, management, and distributed virtualization capabilities”, but it’s not really relevant to the subject of the paper.

·         Add Virtuozzo to the pack? There is nothing in the workloads that would prevent running ESX against Virtuozzo. I’m quite sure that both multiple guests and SMP guests would workJ

The very last question I have – if I run the very same tests myself on ESX and then Virtuozzo and Parallels – will VMware allow me to post the results?