A hypervisor IO performance benchmarking recipe
Posted Friday, October 08, 2010 in Alternative Solutions 5 comments
This post is in response to a comment from Eric Gray to my post about hypervisor IO performance. I'd have put this all in a comment to his comment, but our blogging software doesn't like responses with rich formatting (arrgh) and my answer is a bit hard to read without some formatting.
Eric asked for more details about how to run certain IO benchmarks, so I'll provide that here. Eric also asked Virsto so submit our benchmarks to VMware for approval; we'll save our response to that for another post.
As you undoubtedly already know Eric, disk IO workloads in virtualized environments are typically random, have a higher proportion of smaller request sizes, and are more write intensive. So the benchmarking of a hypervisor IO subsystem is relatively straightforward for those with the right expertise. I thought I gave enough hints in my earlier post, but here is a more detailed recipe:
- Take your favorite disk IO benchmark utility, say Iometer, and configure it to run a server style profile load (something like 8k to 64k block size, 50/50 read/write, random IO pattern).
- Configure your ESX host to run multiple VM instances, each containing an identical instance of the configured benchmark.
- Now run 1, 2, 4, 8 and so on VMs simultaneously with active benchmarks in each VM.
- Observe how the COMBINED throughput of ALL benchmark instances degrades as you increase the number of VMs running. This is called the VM IO Blender, which effectively limits the density of VMs that can be run per server.
-
Go back to step 2, and reconfigure your VMs to run on thinly provisioned VMDKs.
-
Repeat the runs and observe the performance of the thinly provisioned VMDKs.
(Hint: the results won't be impressive.)
-
Repeat the runs and observe the performance of the thinly provisioned VMDKs.
-
Go back to step 3 and activate the linked clones on your VMDK.
-
Repeat the runs and observe linked clone performance.
(Hint: the results will be even worse.)
-
Repeat the runs and observe linked clone performance.
-
On the same hardware, repeat the steps 1-6 above with Hyper-V as the hypervisor and Virsto as the storage layer; collect the data; draw your own conclusions.
- Note: Virsto virtual disks are always thinly provisioned, even though they perform as if thickly provisioned, so that will save you one step.
- Use Virsto clones when comparing performance to VMDK linked clones.
- Go step 1 and replace the benchmark profile with something like 80% to 90% writes, 8k random IO which is more characteristic of VDI workloads. Repeat steps 1-7.
- Draw your own conclusions.
Of course this is an outline, and it will take some time, maybe a few days to go through the complete exercise. But I think the results awaiting you at the end justify the effort.
I will be happy to work with you on proving the point in your environment.
Thank you again, and best regards,
Alex.
Comments
Eric Gray [VMware] 1:30pm PST on October 8th, 2010
Alex,
Many thanks for providing these details—you are a real stand-up guy for doing that.
Are these comparisons valid for both local and SAN (iSCSI vs. FC?) storage? Is one particularly better for Virsto?
I’d say this is exactly the transparency desired when working with the VMware benchmark approval team.
Best regards,
Eric
alexmiro 5:25pm PST on October 12th, 2010
Eric,
We have validated our performance claims across a wide range of configurations. The particular comparison has been performed on a Fibre Channel disk array.
Thank you,
Alex.
Jason 11:00am PST on November 1st, 2010
Alex,
Currently, recomended practice for Exchange in a virtualized environment is to use Pass-through disks from the host to the guest VM. I suspect that the reason our recommendations read that way are two fold:
1) In the original release of Hyper-V, VHD (fixed and dynamic) performance for high I/O applications was not in the same ball park as using a pass through disk.
2) To help alleviate the I/O blender issue that you have been talking about.
I’m not 100% sure on the 2nd one, does using a pass through configuration (storage exposed to hypervisor host, passed through to guest), alleviate some of the I/O blender issues you are talking about?
If they do not, does your software level the performance playing field when it comes to VHD fixed files verses pass-through disks? This is a very important question in the context of Exchange, using pass through disks helps performance, but impacts (or complicates might be a better term) portability.
Any feedback greatly appreciated.
J.
Alex Miroshnichenko 4:36pm PST on November 2nd, 2010
Jason,
You are correct, the recommendation for using pass through disks for Exchange has it origins in the poor implementation of the VHD IO stack in the initial release of Hyper-V. I think that in Hyper-V R2, fixed VHDs and passthrough disks have virtually identical performance. Dynamic and differencing VHDs are still much slower on writes.
Limiting your configurations to pass through disks does very little to alleviate the VM IO Blender. In the modern world a “disk” is rarely a separate physical disk. Often it is a LUN carved out of a group of disks by a RAID controller. Even if you configure a dedicated pass through disk, there is a great chance that you’d still be sharing the spindles with other virtual machines (on the same server and other hosts) and therefore will suffer the negative effects of the IO Blender. Besides, pass through disks are inflexible, hard to manage and maintain, and limit what you can do with your Exchange VM; for example, Live Migration will not be supported.
Virsto One eliminates the IO Blender independent of the application or back end storage hardware, and delivers better IO performance than any variety of native VHD, at the same time as it occupies less physical disk space and enables advanced functions like Live Migration and fast snapshotting.
There are certain things about Exchange that Virsto One is particularly suited for. This subject deserves a separate detailed post. Thank you for the great idea!
Alex.
Jason 5:47pm PST on November 2nd, 2010
Excellent, thank you for your reply and I would love to read a more Exchange specific blog article concerning your software and Hyper-V. If you don’t mind, might I suggest one talking point that you might give some attention to in your next post.
Please do not take this as gospel, I am a guru on Exchange, but am only medicore on Hyper-V, but I think that Hyper-V balances IO requeses from guests to storage via a fairly simply algorithm that ensures every guest gets access to the disk subsystem. In my mind, I think of this balancer like it’s a Windows NLB cluster. This is good in that it does ensure equal IO access to all guests, but it is also bad in that it also maximizes spindle movment by making all access 100% random essentially.
Again, don’t quote me on that. That is the picture of how it works in my mind, but my real intrest is learning a little bit about how your software does what it does (fixes the “IO Blender”) while still ensuring that all guest VM’s continue to have some sort of equality of access to the storage subsystems. Does that make sense?
If you have any Exchange specific information requests, please let me know.
Respectfully,
J.
Leave a Comment