I/O performance: Not all hypervisors deliver the same results

by Alex Miroshnichenko, CTO

Posted Sunday, October 03, 2010 in Alternative Solutions          2 comments

Tags: bloggers & media, Hyper-V, no dupe, performance, storage sprawl, Virsto One, VM I/O blender, VMware, Xen

Over on the Technet site, VMware’s Eric Gray asked a couple followup questions to my guest post on the Microsoft Virtualization team blog. I can’t give a complete response in a brief comment, so I’m moving the discussion over here to the Virsto blog.

My post extolled the virtues of Dynamic Memory, a feature of the upcoming SP1 release of Microsoft Windows Server 2008 R2. Why would a storage guy love dynamic memory? Because better memory efficiency further exposes virtualization users to another big VM performance bottleneck, namely storage I/O. Since that’s one of the problems Virsto One deals with so well, at Virsto we are fans of efficient memory performance!

Now, since certain vendors (I didn’t name names) like to crow about how much higher performance their platform is than Microsoft Hyper-V, and since Virsto had recently run some storage I/O performance tests, I couldn’t resist pointing how Hyper-V plus Virsto One can be vastly faster than alternatives.

Eric, who I had never met (nice to meet you Eric), responded by asking me to explain the benchmark upon which I based my assertion.

“The first rule of VMware benchmarks is that you don’t talk about VMware benchmarks”

  (That's a quote from another response to Eric's comment.)

Now Eric, being the unofficial chief competitive blogger for a top virtualization vendor, knew very well that if I was referring to his employer’s product, I couldn’t answer his question without violating the EULA, which prohibits publishing benchmarks without the vendor's approval.

As much as I think EULA benchmarking restrictions are deplorable, I do believe in software licenses. More importantly, Virsto’s lawyers do, so I was stuck. I couldn’t publish benchmark results without the hypervisor vendor’s approval.

Trust me, this vendor’s marketing department is not going to like the numbers. So I’m not even going to think about getting on bended knee to ask for their approval.

A little side rant

I hope you’ll forgive me, dear reader, for taking a moment to echo the comments of my friend Simon Crosby and others.

Not allowing users (or other vendors) to publish benchmarks using your software is not a sign of strength. Such practices are unfortunate for end users and the industry at large.

I’ve worked for large vendors who are defensive about the performance of their products being compared publicly in uncontrolled (by the vendor, I mean) benchmarks. I understand all the arguments against allowing a free-for-all information exchange out in the light of day. Frankly, I find those defenses unconvincing.

In my experience, most of the people who are employed by vendors that put benchmarking handcuffs in their EULAs are embarrassed by such restrictions, but of course they can never say so. The first rule of EULA benchmark restrictions is you don’t talk about EULA benchmark restrictions.

One “good” reason not to allow benchmarks

Having said that, I will make one concession.

Every sophisticated software vendor knows that letting people without adequate expertise run their own benchmark is asking for trouble. Few people really have the ability to run meaningful benchmarks. It is hard to do, requires a lot of rarified knowledge, takes a lot of time, and can cost a lot of money.

In the storage world, a common example is the IT administrator who benchmarks I/O performance with the dreaded dd command to copy a big file from here to there. This is probably the most ineffective, misleading, and non-real world benchmark for just about any application workload. It is a really bad benchmark for hypervisors. Yet people use it all the time.

It is painful to be a vendor having to defend benchmarks that are run by people who don’t know what they are doing. So I do have empathy.

However, we at Virsto know a thing or two about storage performance benchmarking. Some of us have been doing it since long before VMware existed.

Back to Eric’s query

On TechNet, I said “I would be happy to show any virtualization user how they can reproduce the results themselves”. When I said that, I wasn’t meaning to say that I’d show Eric. But hey, in addition to working for a virtualization vendor, Eric is probably a user too, so I may as well just answer his question.

So how could a user recreate performance benchmarks that support my claim? It’s pretty easy. I’ll give you just two examples.

There are a couple notable weaknesses that many storage solutions have in VM environments.

One is thin provisioning performance. Thin provisioning, in which storage space is not allocated until data is actually written, is particularly useful in VM environments where VM sprawl results in storage sprawl. There are a lot of thin provisioning products out there, and some perform well. Some don’t. A benchmark tool like Iometer can be used to spew random writes to unexpanded thinly provisioned virtual disk files. Why random? Because that’s the kind of writes that a hypervisor has to deal with in the real world.

Second, you might want to check out how your storage solution performs when you turn on space efficient cloning, which shares common data blocks across multiple VM disk images. To avoid VM storage sprawl, space efficient cloning is an essential tool. Several vendors we know of have a cloning feature (the name of one vendor’s rhymes with “distinct groans”), but caution their customers against using them in production environments because of known performance weaknesses. What is the point, I wonder, of features that you’re not supposed to use in production? That same Iometer benchmarking tool can be used to test I/O performance of any vendor’s clones against Virsto clones.

If you want detailed instructions on how to run the benchmarks, please contact us and we’ll be happy to walk you through it. We’re pretty confident that, if the benchmarks are run properly, you’ll be interested in using Virsto One in your VM environment.

We stand by our unpublished benchmarks

We, our customers, and our partners have run a number of performance tests to benchmark Virsto One against some industry leaders. The bar chart at the top of this post shows the results of one test. Even though the axes aren’t labeled for fear of getting on the wrong side of any licensing police, I will say this: The magnitude of the bars is precisely accurate. Yes, the golden bar shows the performance of Hyper-V with Virsto One in a specific test. The gray bar is really how much I/O we got on the same hardware through an alternative solution from a certain vendor. But I’m not naming names.

Tweet