Real-world virtualization benchmarking: the best server CPUs compared
by Johan De Gelas on May 21, 2009 3:00 AM EST- Posted in
- IT Computing
Inquisitive Minds Want to Know
Tynopik, a nickname for one of our readers, commented: "Is Nehalem better at virtualization simply because it's a faster CPU? Or are the VM-specific enhancements making a difference?" For some IT professionals that might not matter, but many of our readers are very keen (rightfully so!) to understand the "why" and "how". Which characteristics make a certain CPU a winner in vApus Mark I? What about as we make further progress with our stress testing, profiling, and benchmarking research for virtualization in general?
Understanding how the individual applications behave would be very interesting, but this is close to impossible with our current stress test scenario. We give each of the four VMs four virtual CPUs, and there are only eight physical CPUs available. The result is that the VMs steal time from each other and thus influence each other's results. It is therefore easier to zoom in on the total scores rather than the individual scores. We measured the following numbers with ESXtop:
| Dual Opteron 8389 2.9GHz CPU Usage | |
| Percentage of CPU Time | |
| Web portal VM1 | 19.8 |
| Web portal VM2 | 19.425 |
| OLAP VM | 27.2125 |
| OLTP VM | 27.0625 |
| Total "Work" | 93.5 |
| "Pure" Hypervisor | 1.9375 |
| Idle | 4.5625 |
The "pure" hypervisor percentage is calculated as what is left after subtracting the work that is done in the VMs and the "idle worlds". The work done in the VMs includes the VMM, which is part of the hypervisor. It is impossible, as far as we know, to determine the exact amount of time spent in the guest OS and in the hypervisor. That is the reason why we speak of "pure" hypervisor work: it does not include all the hypervisor work, but it is the part that happens in the address space of the hypervisor kernel.
Notice how the scheduler of ESX is pretty smart as it gives the more intensive OLAP and OLTP VMs more physical CPU time. You could say that those VMs "steal" a bit of time from the web portal VMs. The Nehalem based Xeons shows very similar numbers when it comes to CPU usage:
| Dual Xeon X5570 CPU Usage (no Hyper-Threading) | |
| Percentage of CPU time | |
| Web portal VM1 | 18.5 |
| Web portal VM2 | 17.88 |
| OLAP VM | 27.88 |
| OLTP VM | 27.89 |
| Total "Work" | 92.14 |
| "Pure" Hypervisor | 1.2 |
| Idle | 6.66 |
With Hyper-Threading, we see something interesting. VMware ESXtop does not count the "Hyper-Threading CPUs" as real CPUs but does see that the CPUs are utilized better:
| Dual Xeon X5570 CPU Usage (Hyper-Threading Enabled) | |
| Percentage of CPU time | |
| Web portal VM1 | 20.13 |
| Web portal VM2 | 20.32 |
| OLAP VM | 28.91 |
| OLTP VM | 28.28 |
| Total "Work" | 97.64 |
| "Pure" Hypervisor | 1.04 |
| Idle | 1.32 |
Idle time is reduced from 6.7% to 1.3%.
The Xeon 54XX: no longer a virtualization wretch
It's also interesting that VMmark tells us that the Shanghais and Nehalems are running circles around the relatively young Xeon 54xx platform, while our vApus Mark I tells us that while the Xeon 54xx might not be the first choice for virtualization, it is nevertheless a viable platform for consolidation. The ESXtop numbers you just saw gives us some valuable clues, and the Xeon 54xx "virtualization revival" is a result of the way we test now. Allow us to explain.
In our case, we have eight physical cores with four VMs and four vCPUs each. So on average the hypervisor has to allocate two physical CPUs to each virtual machine. ESXtop shows us that the scheduler plays it smart. In many cases, a VM gets one dual-core die on the Xeon 54xx, and cache coherency messages are exchanged via a very fast shared L2 cache. ESXtop indicates quite a few "core migrations" but never "socket migrations". In other words, the ESX scheduler keeps the virtual machines on the same cores as much as possible, keeping the L2 cache "warm". In this scenario, the Xeon 5450 can leverage a formidable weapon: the very fast and large 6MB that each two cores share. In contrast, two cores working on the same VM have to content themselves with a tiny 512KB L2 and a slower and a smaller L3 cache (4MB per two cores) on Nehalem. The way we tested right now is probably the best case for the Xeon 54xx Harpertown. We'll update with two and three tile results later.
Quad Opteron: room for more
Our current benchmark scenario is not taxing enough for a quad Opteron server:
| Quad Opteron 8389 CPU Usage | |
| Percentage of CPU time | |
| Web portal VM1 | 14.70625 |
| Web portal VM2 | 14.93125 |
| OLAP VM | 23.75 |
| OLTP VM | 23.625 |
| Total "Work" | 77.0125 |
| "Pure" Hypervisor | 2.85 |
| Idle | 21.5625 |
Still, we were curious how a quad machine would handle our virtualization workload, even at 77% CPU load. Be warned that the numbers below are not accurate, but give some initial ideas.
Despite the fact that we are only using 77% of the four CPUs compared to the 94-97% on Intel, the quad socket machine remains out of reach of the dual CPU systems. The quad Shanghai server outperforms the best dual socket Intel by 31% and improves performance by 58% over its dual socket sibling. We expect that once we run with two or three "tiles" (8 or 12 VMs), the quad socket machine will probably outperform the dual shanghai by -- roughly estimated -- 90%. Again, this is a completely different picture than what we see in VMmark.
Facebook
Twitter
RSS
66 Comments
View All Comments
tshen83 - Thursday, May 21, 2009 - link
Jarred:Let's not fool each other. Johan's AMD bias is disgusting.
My assertion that HardOCP killed the GPU market is simply trying to show you the effect of invalidating industry standard benchmarks. Architecturally, Nvidia's GPU bigger monolithic cores are far more advanced than ATI's cores right now. In GPGPU applications, it is not even close. The problem with gaming FPS benchmark as I have said is that developers are typically happy once the FPS reaches parity. It does not show architectural superiority.
vApus? There are a ton of questions unanswered.
1. Who wrote the software?(I assume European)
2. Does the software scale linearly? And does the software scale on both AMD and Intel architecuture?
3. Why benchmark 4 Core Virtual machines when we know that VMware doesn't really scale that well themselves in SMP setup?
4. Seriously? Nieuws.be OLAP database? How many real world people run Nieuws.be?
I usually don't respond to Anandtech articles unless the article is disgustingly stupid. I also don't understand why you guys can't accept the fact that Nehalem is in fact 100% performance/watt improved vs the previous generation Xeon. It is backed by data from more than one industry standard benchmark.
Is AMD worth a look today? No, absolutely not. If you are still considering anything AMD today, you are an idiot. (The world is full of idiots) AMD's only chance is if they can release the G34 socket platform within a TDP range that is acceptable before they run out of cash.
Before you call me a troll, remind yourself this: usually the troll is smarter than the people he/she is trolling. So ask yourself this question: did Johan deserve the negative critism?
JarredWalton - Thursday, May 21, 2009 - link
You criticize every one of his articles, often because I'm not sure your reading comprehension is up to snuff. His "AMD bias" is not disgusting, though I'm quite sure your Intel bias is far worse than his AMD bias. The reason 3DMark has been largely invalidated is that it doesn't show realistic performance - though some of the latest versions scale similarly to some games, at best 3DMark measures 3DMark performance. Similarly, VMmark measures VMmark performance. Unless your workload is the same as VMmark, it doesn't really tell you much.1 - Who wrote the software? According to the article, "vApus or Virtual Application Unique Stresstest is a stress test developed by Dieter Vandroemme, lead developer of the Sizing Server Lab at the University College of West-Flanders." His being European has nothing to do with anything at all, unless you're a racist, bigoted fool.
2 - 2-tile and 3-tile testing is in the works. It will take time.
3 - Perhaps because there are companies looking for exactly that sort of solution. I guess we should only test situations where VMware performs optimally?
4 - The source of the database is not so critical as the fact that it is a real-world database. Whether Johan uses a DB from Nieuws.be, AnandTech.com, Cnet.com, or some other source isn't particularly meaningful. It is a real setup used outside of benchmarking, and he had access to the site.
I usually don't respond to trolls unless they are disgustingly stupid as well. I don't understand why you can't accept the fact that Nehalem isn't a panacea that fixes all the world's woes. That is backed by the world around us which continues to have all sorts of problems, and a "greener" CPU isn't going to save the environment any more than unplugging millions of cell phone charges that each consume 0.5W of power or less.
AMD is certainly worth a *look* today. Will you actually end up purchasing AMD? That depends largely on your intended use. I have old Athlon 64/X2 systems that do everything that they need to do. For a small investment, you can build a much better AMD HTPC than Intel - mostly because the cheap Intel platform boards are garbage. I'd take a lesser CPU with a better motherboard any day over a top-end CPU with a crappy motherboard. If you want a system for less than $300, the motherboards alone would make me tend towards AMD.
Of course, that completely misses the point that this isn't even remotely related to that market. Servers are in another realm, and features and support are critical. If you have a choice between AMD quad socket and Intel dual socket, and the price is the same, you might want the AMD solution. If you have existing hardware that can be upgraded to Shanghai without changing anything other than the CPU, you might want AMD. If you're buying new, you'd want to look at as much data as possible.
Xeon X5570 still surpasses AMD in the initial tests by over 30%, which is not insignificant. If that extends to 50% or more in 2-tile and 3-tile setups, it's even more in Intel's favor. However, a 30% advantage is hardly out of line with the rest of the computing world. SYSmark 2007 shows the i7 965 beating the Phenom II 955 by 26.6%. Photoshop CS4 shows a 48.7% difference. DivX is 35.3%, xVid is 15.9% pass1 and 65.4% pass2, and WME9 is 25%. 3dsmax is 55.8%, CINEBENCH is 42%, and POV-ray is 65.3%.
Which of those tests is a best indication of true potential for Core i7? Well, ALL OF THEM ARE! What's the best virtualization performance metric out there? Or the best server benchmark out there? They're ALL important and useful. vApus is just one more item to look at, and it still shows a good lead for Intel.
Where is the 100% perf/watt boost compared to last generation? Well, it's in an application where i7 can stretch its eight threaded muscles. Compared to AMD, the performance/watt benefit for an entire system is more like 40% on servers. For QX9770, i7 965 is 32% more perf/watt in Cinebench, or 37.6% in Xvid. I doubt you can find a 100% increase in performance/watt without cherry-picking the benchmark and CPUs in question, but that's what you're already determined to do. That, my friend, is true bias - when you can't even admit that anything from the competition might be noteworthy, you are obviously wearing blinders.
Zstream - Thursday, May 21, 2009 - link
Umm based on your two rants this means you have ZERO knowledge working with virtual desktops/terminal servers/virtual applications.I feel I need to make two corrections.
One: ATI's die size is roughly 75% of Nvidia's, how do you conclude that Nvidia is better? Well honestly you can not because if you scale the performance and had the same die size of Nvidia, then ATI would be killing them.
Second: Majority of enterprise's run AMD and Intel, in fact not till Neh. did Intel really come into the virtualization market.
tshen83 - Thursday, May 21, 2009 - link
"Umm based on your two rants this means you have ZERO knowledge working with virtual desktops/terminal servers/virtual applications. "Really? Just how did you come up with this revelation?
"One: ATI's die size is roughly 75% of Nvidia's, how do you conclude that Nvidia is better? Well honestly you can not because if you scale the performance and had the same die size of Nvidia, then ATI would be killing them. "
You don't know shit about GPUs.
"Second: Majority of enterprise's run AMD and Intel, in fact not till Neh. did Intel really come into the virtualization market. "
True. That's what I am saying too, if you listened. I said, "no one should be considering AMD today because Nehalem is here".
Zstream - Thursday, May 21, 2009 - link
I came to that conclusion based on your incoherent rants.Why would you say I do not know shit about GPU's? I provided you a fact, your illogical thinking does not change the matter. It comes down to die size and ATI wins performance/DIE. If you would like to argue that claim with then please do so.
Who would consider Neh in todays market? Very few, unless you are a self proclaimed millionaire who crazily spends or needing the extra performance boost in some applications like exchange.
Viditor - Thursday, May 21, 2009 - link
Guys, it's tshen...nobody over the age of 12 listens to his rants anyway, so don't feed the troll (or ban him if you can...).leexgx - Thursday, May 21, 2009 - link
LOL nice rant3dmark cant be used any more as its not an 3dmark any more its more like an 3d gpu/cpu mark the CPU can sway the total result
AMD cpus have been using dedicated bus that talks to each other cpu socket and has direct access to the ram, allso AMD does have V-amd as well on all amd64 am2 cpus as well as optrons an (baring sempron)
Makaveli - Thursday, May 21, 2009 - link
Ya what is the post all about.HardOCP killed the GPU market? I don't know about you but I never bought a videocard because of its 3dmark score. It's one benchmark that both companies cater to but is of little importance. Hardocp review method has much more valuable data for me than one benchmark.
Let me ask you this when you are buying a car or anything of siginicant value. Do you not do your homework is one review being either positive or negative enough to drop your hard earned cash?
If so Bestbuy is that way!
As for the rest of your post the personal attacks and childish language cleary show your not even worth taking seriously. Sounds more like the ramblings of a Highschool child who is trying to get attention.
Good day to you sir,
Godspeed
Zstream - Thursday, May 21, 2009 - link
You have no idea what you are talking about. The benchmark software can be downloaded. It is not our fault you are to poor to pay for a product.The rest I have to say "LOL".
DeepThought86 - Thursday, May 21, 2009 - link
Wow, just wow.