Home > Business Critical Applications, VMware > Jumbo Frames on vSphere 5

Jumbo Frames on vSphere 5

I read a great blog post a while ago from Jason Boche titled Jumbo Frames Comparison Testing with IP Storage and vMotion. The results of the tests showed at best marginal gains to be had from using Jumbo Frames with 1Gb/s NIC’s on ESXi 4.1. Based on reading this, and a lot of discussion that came out of PEX 2012 regarding Jumbo Frames I decided to conduct my own tests to see if the results were any different when using modern 10G switches and NIC’s. Some of the results were not what I expected.

Previous testing I had conducted in customer environments with 10G switches and NIC’s had shown anywhere from 10 – 30% improvement in raw throughput as well as lower latency and improved CPU efficiency. A lot of the performance characteristics are OS dependent, and in the case of Linux will depend on how you’ve tuned your kernel. Both switching equipment and NIC’s have improved a lot over the last couple of years, so it’s possible the differences I found in performance between MTU 9000 and MTU 1500 reflect that as well.

For the testing in my lab I wanted to do as close to valid testing as possible without changing more than necessary, as I have quite a lot of stuff that is deployed and relying on Jumbo Frames (like my storage). My entire underlying network infrastructure in my lab, including my routing switches are all configured for Jumbo Frames. The vSwitches used for the VM’s and VMK ports are also configured for Jumbo Frames, and were not modified during the tests. So my testing was limited to changing the MTU settings for the VMK Ports for vMotion and the NIC MTU settings for the Guest OS’s I tested.

Also check out my other article titled Jumbo Frames on vSphere 5 U1.

Lab Test Hardware:

Host Type: 2 x Dell T710, 72GB RAM, Dual Intel Xeon X5650, Intel X520-T2 Dual Port 10G NIC
vSphere Version: vCenter 5.0 GA, ESXi 5 – Build 515841
Switch: Dell PowerConnect 8024 – 24 Port, 10G-BaseT
Test VM’s:

  1. Windows 2008 R2 Enterprise 64bit – 32GB RAM, 3 vCPU, VMXNET3
  2. SLES Linux 11 SP1 64bit – 16GB RAM, 3 vCPU, VMXNET3
  3. Windows 2003 Standard 32bit – 4GB, 2 vCPU, VMXNET3

Additional details regarding My Lab Environment.

Lab Test Script:

For the vMotion Tests I used the Windows 2008 R2 systems while they were running a Prime95 x64 Torture Test. This ensures that as many memory pages as possible are changing as fast as possible. This places a lot of stress on vMotion, which will extend the migration times and should fully utilize the 10G NIC’s. The Hosts start with a single VMKernel NIC port configured for vMotion, and configured for Jumbo Frames. A second VMKernel port is configured on a separate port group ready when needed for the testing. I conducted multiple tests and used the average from the best test as the results.

vMotion Tests

  1. Start RESXTOP from vMA against both hosts in batch mode and record the output from both Test vSphere Hosts.
  2. Power on two Test VM’s on Server 1, start torture test on both VM’s
  3. Migrate by vMotion both VM’s to destination host Test Host 2 at the same time
  4. Migrate by vMotion both VM’s back to source host Test Host 1
  5. Repease step 3 and 4 again
  6. Enable the second VMKernel vMotion port on each of the test hosts
  7. Repeat steps 3 – 5
  8. Modify VMKernel Port MTU to 1500 on both VMKernel ports on both test hosts
  9. Repeat steps 3 – 5
  10. Disable the second VMKernel vMotion port on each of the test hosts
  11. Repeat steps 3 – 5
  12. Reset hosts to original configuration

For the Guest OS Network Performance Tests I used iPerf, which is an open source network performance test tool. Due to Windows 2003 not supporting receive side scaling I used 10 parallel streams to get the performance results, with both SLES and Windows 2008 R2 I used a single stream.

Guest OS Network Performance Tests

  1. Power on First Test VM on Test Host 1
  2. Power on Second Test VM on Test Host 2
  3. Configure each VM to MTU 1500
  4. Start iPerf in Server Mode on Test VM on Test Host 2
  5. Start iPerf on Test VM on Host 1 to commence the test
  6. Record the results
  7. Configure each VM to MTU 9000
  8. Repeat steps 4 – 6

For each of the Guest OS’s being tested execute the steps above. Below are the iPerf commands I executed during my tests.

Receiver Node: iperf -s -i 60 -w 1m -f m
Sender Node: iperf -i 5 -w 1m -f m -c <receiver_node_ip>

The Results:

Before starting this testing process I thought I was going to get a 15 – 20% difference between Jumbo and Non-Jumbo. I based this on previous experience and also that the offload capability of 10G NIC’s, Server CPU’s and 10G switches have all improved over the last couple of years. The difference was a bit less than I expected. But still a decent amount compared to what might be expected on a 1Gb/s network. I was not able to test the Jumbo Frames performance on Windows 2008 R2 due to a bug in ESXi 5 VMware Tools and VMXNET3 that prevents Jumbo Frames from functioning, see my previous post Windows VMXNET3 Performance Issues and Instability with vSphere 5.0.

Jumbo Frames vs No Jumbo on ESXi 5

The SLES 11 SP1 VM has had quite a lot of tuning from the out of the box configuration. The tuning probably resulted in the performance of that VM that is roughly the same as the vMotion throughput. If you have not tuned your Linux kernel I wouldn’t expect you’d get the same performance. The Windows 2003 and 2008 R2 were both out of the box configurations with only the VMXNET3 driver MTU modified on the 2003 system.

As you can see from the test results the Linux VM and VMKernel port used for vMotion can saturate a 10G link when using Jumbo Frames. The difference between Jumbo and Non Jumbo on Linux is probably higher than with vMotion due to vMotion VMKernel port being highly tuned for one purpose. The Non Jumbo performance of Windows 2008 R2 was quite close to the Linux Non-Jumbo performance, which shows the improvements that Microsoft has made to their IP stack since Windows 2003.

The Bottom Line:

Using Jumbo Frames requires that all devices from end to end in the network path between source and destination are configured correctly to support MTU 9000, i.e. switches, routers, vSwitches and Servers/VM’s. If Jumbo Frames is not enabled throughout the network path from source to destination you will get packet fragmentation, which will reduce performance back to that of Non-Jumbo. In an existing network if Jumbo Frames was not enabled when it was constructed it could involve considerable effort to change it. However you don’t necessarily have to change it everywhere or on everything, depending upon how your network is segmented. You might consider just enabling it on the segments of the network and servers/switches that could benefit the most from using Jumbo Frames.When implementing new 10G infrastructure it may be worthwhile configuring all new network infrastructure for Jumbo Frames, which is very simple during initial configuration. Even though modern NIC’s and switching equipment have reduced the difference between Jumbo and Non-Jumbo it can still be worthwhile in a number of cases.

My results suggest you could get anywhere from 10% to 13% for normal Guest OS traffic flows and between 8% and 19% for vMotion traffic flows. You will need to decide if the additional throughput, lower CPU usage on servers and network switches/routers and less latency is worth the effort. Two traffic flows that can benefit a lot from the implementation of Jumbo Frames are vMotion and also the Oracle RAC Private Interconnect Network. These types of traffic are normally isolated onto separate switches or non-routed VLANs and could be a prime candidate to implement Jumbo Frames in isolation from the rest of the network. In my vMotion 2 NIC tests the 19% improvement in throughput reduced the migration times by 10 seconds (from 50 seconds down to 40 seconds) for my 2 x 32GB RAM VM’s.

For Oracle RAC in particular Jumbo Frames is recommended even on 1Gb/s networks as a single DB block can then fit into a single IP packet, which reduces DB latencies across the private interconnect. With the latest version of Oracle RAC 11G R2 up to 4 private interconnect networks can be used to provide load balancing and high availability. For databases that make heavy use of the interconnect this can provide a big performance boost without having to completely re-architect the database.

A 10% performance degradation might not sound like much, but when you’re talking about a 10Gb/s network that’s like losing the performance of an entire 1Gb/s link. When you use multiple links it quickly adds up to be a substantial loss of performance. The benefit of Jumbo Frames is only going to grow with the new 40G and 100G Ethernet standards.  Let’s just hope that the OS IP stacks are improved enough to cope with the new standards when they start to become mainstream.

I would encourage you to test it out and implement it where appropriate. Not every application or use case needs Jumbo Frames, but there are a couple of good ones that do.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

  1. marco
    February 20, 2012 at 3:52 am

    Do you know what effect the jumbo frames have on cisco switching buffers?

    • February 20, 2012 at 6:13 am

      Hi Marco, I haven’t tested Jumbo Frames on Cisco switches for a while. Last time I tested Cisco equipment the performance was very good with Jumbo (but a bigger difference between Jumbo and Non-Jumbo). But this is very equipment dependent, there are many options with Cisco switches, and not all will perform the same way. Some line cards I know only have full buffers available if you distribute your connections to every 4th port, rendering the other ports unusable. But the assumption is that you won’t run every port on the line card at full performance all of the time. A customer recently ran perf tests using JPerf on Windows 2008 R2 64bit with ESXi 4.1 on Nexus 2K into 5K and was getting 14Gb/s using LACP when Jumbo was enabled. Not sure the impact on buffers though.

      • February 20, 2012 at 6:23 am

        I’m going to discuss this with my networking guy. We are currently using 2x 10Gb per ESX host, with one active other passive. Cisco 4900 switches. I can remember we turned off jumbo frames as per Duncan’s blog about “No Jumbo frames on your Management Network!” and some buffer thing with the cisco’s. I think It wass something when you enable jumboframes the cisco’s would use less buffers for your normal traffic and will use a lot for the big sized packets. I will ask him to clarify and will get back about this.

        And now that I browsed back to Duncan’s post, he corrected it into: Just received an email that all the cases where we thought vSphere HA issues were caused by Jumbo Frames being enabled were actually caused by the fact that it was not configured correctly end-to-end. Please validate Jumbo Frame configuration on all levels when configuring. (Physical Switches, vSwitch, Portgroup, VMkernel etc)

      • February 20, 2012 at 6:30 am

        I was quite surprised when I first saw Duncan’s post on that, as I’ve been running Jumbo for ages and had no problems with HA on vSphere 5. With your Cisco environment, do check things out carefully. If possible test the implementation in an isolated environment. You may find that you need to upgrade to the latest Cisco software release to get the best performance from Jumbo. I’ve not done any testing with Jumbo on 4900 series switches, so if you do test it I’d love to hear your results.

  2. February 20, 2012 at 4:03 am

    Very interesting results. I do recall the 1GbE test where the gains were minimal and in some cases negative. Thanks for performing these tests.

  3. Paul Kelly
    February 20, 2012 at 7:32 am

    I know you have some pretty extensive lab kit, would you believe I was going to pose this very question to you today? 😉

    Congratulations, you passed the mind reading test!

  4. mikidutzaaMihai
    February 21, 2012 at 3:29 am

    Do you have a graph with CPU usage also? I am curious what was the exact impact on CPU as well.


    • February 21, 2012 at 11:13 pm

      I didn’t keep the CPU graphs during the tests as my primary goal was throughput differences. I will repeat some tests and include VM CPU Usage. It was quite high, up to 70% if I recall correctly. Jumbo was similar CPU usage, but with higher throughput, yielding more CPU efficiency. Are you more interested in Host CPU or VM CPU?

      • mikidutzaaMihai
        February 22, 2012 at 3:51 am

        Thanks for the reply. I was curious whether there was a significant difference in host CPU usage (i.e. if it’s worth enabling for CPU efficiency reasons).

  5. April 13, 2012 at 5:51 am

    If you are testing on a pure 10Gb network, how well would a MTU of 15500 (not a typo) compare against 1500 and 9000?

    • April 13, 2012 at 1:25 pm

      Hi Tim, my switching equipment currently only goes up to 9216 or 10K, and vSphere only allows up to 9000. So it’s not possible to test an MTU of that size. It may have an incremental benefit if an when it’s ever supported, but it’s hard to say. Things may change with the adoption of 40G and 100G in the future. Most network equipment I regularly deal with uses 9216 as the maximum MTU, which is sufficient with the VM’s using 9000 and the necessary other overhead bytes that may be needed.

  6. Glenn
    May 18, 2012 at 10:35 am

    Interesting that there is no comment on flow control in this article. IMHO flow control & jumbo need to go hand in hand with 10G setups to prevent packet loss and big performance hits when a host or device is overloaded.

    • May 18, 2012 at 11:15 am

      Hi Glenn, you raise a good point. However flow control isn’t always a good thing. In some situations it can cause more problems than it solves (excessive pause frames). It will also depend on how much buffers are available per switch port and if all ports have full buffers, and if links are being oversubscribed at L2. I have flow control active on my switch however and it was active during the test. I have a split between customers where it is enabled and where it has been disabled. It’s definitely not a one size fits all decision. When I test vMotion again I will add a scenario with flow control turned off and see what difference if any are observed.

  7. -AM-
    July 23, 2012 at 9:12 pm

    @vcdxnz0 wrote:
    > Are you more interested in Host CPU or VM CPU?

    I really would be interested in both – but if had to decide: I would choose Host CPU.

    No real-life measurements about Host CPU load with iSCSI in 10Gb environments available on the web (not later than 2012).

    Any plans to re-test…?

    • July 23, 2012 at 9:42 pm

      Yes, I do plan to retest. I’m going to be retesting this when the next release of vSphere is GA’d later this year. When I retest I will include CPU usage into the calculations. With the modern 10G cards and modern CPU’s the performance of iSCSI and NFS is on par with 8G Fibre Channel when architected in a similar manner.

      • -AM-
        July 24, 2012 at 8:37 am

        Thanks Michael, looking forward to!

  1. March 29, 2012 at 1:36 am
  2. April 13, 2012 at 2:00 am
  3. July 15, 2012 at 10:05 pm

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: