July 13th, 2006

Virtualization benchmark

After some thinking, we decided that SWsoft needs to take a lead and help create a virtualization benchmark that would allow IT organizations to assess different aspects of server virtualization technologies.

Here is the current thinking about the benchmark, and as usual I’m looking forward to comments:

·         The benchmark will be focused on server virtualization. When we have desktop product, we’ll focus on a benchmark for desktop technologies, but for now, it’s just servers.

·         The benchmark will be comprehensive and will measure several aspects of the technology.

·         SWsoft will work with all interested parties – virtualization technologies vendors, hardware and software vendors and, of course, benchmark vendors – to ensure that the benchmark is accepted by the entire industry.

Here is the current thinking on the metrics of the benchmark:

·         Isolation - We’ll determine how strong and complete the isolation between virtual environments is. Hereafter, by “virtual environment” I mean not just Virtuozzo VE but rather a system partition created by a virtualization technology.

·         Security - We’ll perform several attacks on the target system, trying to measure how robust the system is.

·         Manageability - We’ll perform typical management operations – such as provisioning, starting and stopping of virtual environments and applications – and measure how long does it take to do those as well as how easy or confusing it is for an admin to do them.

·         Performance - We’ll run a series of well-known performance tests, measuring performance of a single environment as well as performance of multiple concurrently running environments.

·         Scalability - We’ll measure how well a technology can use the resources provided by the hardware.

·         Density - We’ll measure how many concurrent environments it is possible to run with acceptable performance.

·         Resource management - We’ll determine if resource management capabilities provided by a virtualization technology are adequate for the test scenarios.

Tests won’t correspond to metrics 1:1. For example, performance will probably include several tests, while density will probably be measured in the same performance tests.

That’s it for now, but there will be more. I’ll do a separate post for each benchmarking metric, trying to communicate:

·         Why it’s an important metric

·         How we’re going to measure it

·         My speculation on results

Meanwhile, please, send me your thoughts (or post a comment) on

·         What metrics should be added/removed from the benchmark

·         What real-world scenarios we should try to emulate

·         What are specific benchmarks you think we should re-run in virtualize environment?

Looking forward to your comments!