Home > Business Critical Applications, VMware > The Good, The Great, and the Gotcha with Multi-NIC vMotion in vSphere 5

The Good, The Great, and the Gotcha with Multi-NIC vMotion in vSphere 5

One of the features many people may not be aware of that was released in vSphere 5 is Multiple-NIC vMotion. This is a feature that allows you to load balance a single or multiple vMotion transmissions over multiple physical NIC’s. This is of significant benefit when you’ve got VM’s and hosts with large amounts of memory, as vMotion migrations will complete significantly faster. So your Business Critical Applications with large amount of memory and CPU’s can now migrate without disruption even faster. Below I’ll briefly cover the good and great of this technology and also a gotcha that you need to be aware of.

The Good

I thought we’d start with the good news. With vSphere 5 you can now split single or multiple vMotion streams over multiple NIC’s. Up to 4 x 10Gb/s NIC’s or 16 x 1Gb/s NIC’s are supported. This magnifies even further the already impressive 30% improvement in vMotion performance vs vSphere 4.1.

The Great

It is super easy to set up multi-NIC vMotion. It’s all explained in KB 2007467. To briefly cover the set up.

  1. You set up multiple vmkernel port groups, each with a different NIC as primary, any other NIC’s as standby or unused, and a different IP address on the same subnet, .
  2. You then select the vMotion tick box on the vmkernel port.

That’s it.

Very simple. Now single vMotion’s and multiple concurrent vMotions will be load balanced over the NIC’s. There is absolutely not need to configure any complicated LACP or IP Hash load balancing to make this work, there is no need to use Load Based Teaming (Route based on physical NIC load). You can use this with standard switches, no need for distributed switch. It doesn’t even require Enterprise Plus licenses, but as the benefits are mostly with VM’s and hosts with lots of RAM you’re probably going to have Enterprise Plus anyway.

I tested performance of Multi-NIC vMotion with 2 x 10Gb/s NIC’s in my home lab and got almost 18Gb/s when using Jumbo Frames on vSphere 5. Hosts go into maintenance mode so fast you better not blink! I haven’t retested Multi-NIC vMotion again since upgrading to vSphere 5 U1 and the latest patches. I plan to test it when Update 2 or the next vSphere release comes out.

Here is the test results from my previous article. You can see the Multi-NIC vMotion Test at the bottom – vMotion 2 x 10G.

Jumbo Frames vs No Jumbo on ESXi 5

The Gotcha

There is a condition that may occur during long running vMotion operations that could cause all hosts ports configured for vMotion to be flooded with the vMotion traffic (on vSphere 5.0 prior to Update 2). The way I understand it this occurs when physical switches MAC tables start timing out the MAC’s (before the ARP timeout). The reason it occurs is because although the outbound traffic is split over multiple vmkernel ports and multiple NIC’s the ACK’s coming back from one MAC. So after a while the physical network may time out the other MAC’s as it’s not seeing any traffic from them. As the transmissions are still occurring the switches may start flooding every port that is configured for the vMotion VLAN. Because the problem is generated by MAC timeouts around the 5 minute mark you will be more likely to experience this problem with 1G vMotion NIC’s or with 10G vMotion NIC’s that have Network IO Control or QoS limits imposed, as your migrations will generally take longer.

To work around this problem you may be able to adjust the MAC timeout values on your switches, depending on the type of switches you’ve got. The default MAC timeout on Cisco switches is normally 5 minutes. On the Dell 8024 10G Base T switch I’ve got in my lab the Address Aging value defaults to 301 seconds and is adjustable. Be careful if you choose to adjust these values as there may be other consequences, any adjustments should be tested, and only applied to the switches connecting directly to your vSphere Hosts carrying the vMotion VLAN.

VMware is aware of this problem and is working on a fix has released a fix as part of vSphere 5.0 U2. The fix didn’t make it into ESXi 5 Patch 03 that was released on 13/07/2012 (07/12/2012 for those in the USA). I would hope that it makes it into the next vSphere 5 update release. I will have updated this article when now the problem is fixed and let you know what patch or updates you need to apply. Until then I hope you are able to make use of Multi-NIC vMotion by applying the above workaround. At least configure it in your test environments and see how it goes.

Update (20130103): This issue is fixed on vSphere 5.0 U2. All you need to do is update your hosts to 5.0 U2 and this problem will be resolved. The workaround is no longer necessary. 

I have just posted a workaround to this Gotcha in an article Workaround for Multi-NIC vMotion Unicast Flooding in vSphere 5. This workaround however is unsupported. So use at your own risk. It appears to work well on vSphere 5.0, 5.0 U1, and should work with 5.1 GA but hasn’t been tested.

Final Word

If you thought vMotion in vSphere 5 was already fast you ain’t seen nothing yet, till you’ve experienced Multi-NIC vMotion. Even with this slight gotcha it still has some benefits if you can apply the workaround in your environment. Especially with very large VM’s >96GB RAM, and large hosts >256GB RAM, it will significantly help your migration times.

Duncan Epping at Yellow Bricks has done a follow up article on this titled Clearing up a misunderstanding around CPU throttling with vMotion and Multi-NIC vMotion in vSphere 5. I would highly recommend that you read it. As noted in the comments on this article this should not be kicking in under normal circumstances and will only kick in if the vMotion would have otherwise failed. I’ll let you read Duncan’s article for the full story.

This article is also posted at the VMware Blog Site – Support Insider.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

  1. joshatwell
    July 15, 2012 at 10:28 pm

    Great Post! I was wondering if you had seen any issue or had any additional information about clock speed throttling when doing large number of vMotions as mentioned by Eric Siebert http://de.twitter.com/ericsiebert/status/210510083229093891

    I know in our environment with 10G interfaces when we put hosts in maintenance mode one of our monitoring applications shows huge spikes in response time to those VMs. Naturally it is safe to say that response time might be impacted. Who are we kidding, there are no pings dropped, but the impact on latency sensitive applications is certainly evident.

    I haven’t yet had an opportunity to dig into this deeper so thought I would see if you might already have additional information on the subject. Thanks again for the post!

    • July 16, 2012 at 12:22 am

      Hi Josh, The technology you’re refering to is SDPS (Stun During Page Send). It’s only used when the page dirty rate of the VM’s is faster then the ability to copy the pages to the destination host. SDPS only kicks in when a situation occurs that would mean a vMotion might otherwise fail. It’s a failsafe mechanism. So if you have sufficient bandwidth and you’re not migrating workloads with a page dirty rate that would exceed the 10G/s bandwidth then you shouldn’t be seeing any issues with SDPS. You may want to consider allowing more bandwidth for vMotion depending on the impact to particular applications you’re seeing, or reducing the concurrency of vMotion operations by modifying some advanced settings. Any changes should be tested.

      However if you are using QoS or vNIC’s in a UCS platform then you may have less vMotion bandwidth then necessary and then in some cases SDPS may kick in. But it is only slowing down the CPU of a VM for a few milliseconds at a time. This should not be sufficient to notice any significant response time impact. If you have very latency sensitive applications then they may require special attention and tuning.

      Generally things just work, but for business critical applications and latency sensitive applications they do require a different approach and more care and attention. Let me know what you find when you dig deeper and if necessary get VMware involved. I’d be keen to see what you come up with and exactly what the situation is.

  2. Paul Kelly
    July 16, 2012 at 12:12 pm

    Great article. Just wondering though; with this configuration I’m not sure how to configure this to work with a vDS using LBT, NIOC and SIOC.

    Do you need a VMkernel port for each chunk of 10GbE you want to allow for vMotion?

    If you wanted to guarantee 50% of the 4 uplinks to allow roughly 20Gbps vMotion traffic would you configure the environment somthing like this;

    vDS1 – LBT, SIOC, NIOC
    Management – 5 NIOC Shares, active uplinks 1,2,3,4
    VM Networking – 20 NIOC Shares, active uplinks 1,2,3,4
    iSCSI – 20 NIOC Shares, active uplinks 1,2,3,4
    FT – 15 NIOC Shares, active uplinks 1,2,3,4
    vMotion1 – 30 NIOC Shares, active uplinks 1,2,3,4
    vMotion2 – 30 NIOC Shares, active uplinks 1,2,3,4

    • July 16, 2012 at 12:18 pm

      Hi Paul,

      vMotion1 – Active Uplink 1, All others Standby or Unused
      vMotion2 – Active Uplink 2, All others standby or Unused

      All other port groups, except management, all uplinks active, using route based on physical NIC load. Best practice for management is active/passive with failback set to no, so that you’ve always got it on the same side of the switch fabric, and it doesn’t flip flop around. This reduces the chances of false isolation events a bit more.

      No need to configure LBT on the vMotion port groups, the other VM’s will move around them. NIOC can still be used to control the quality of service. I generally just use the normal, low, high shares without specifying specific values. It’s all relative. As long as the important traffic types get the bulk of the shares for when there is congestion. LBT and NIOC sort out most problems.

      Hope this helps.

      • Paul Kelly
        July 17, 2012 at 8:37 am

        Thanks. That all makes sense. I really do envy your ability to test this stuff in that epic lab of yours.

  3. Justin McDearis
    July 17, 2012 at 3:02 am

    Good question Paul, and great answer Michael! Thank you for the very helpful information. I have been looking for an answer to this exact question for our VMware environment.

  4. July 18, 2012 at 3:11 am

    Great article Mr. W. Keep up the good work & fantastic lab.

  5. September 9, 2012 at 11:31 am

    We have already gone to great lengths to setup our environment using the methods described in KB1004048 using NIC teaming. Aside from the additional configuration, is there any negatives to using this method versus the method you described above? Thanks!

    • September 9, 2012 at 8:38 pm

      Yes, you will in most cases not actually be able to use the aggregate bandwidth of multiple uplinks as any individual stream will be limited to the bandwidth of a single uplink. With multi-nic vMotion you can effectively use both NIC’s all the time. You are also now restricted to only using IP Hash as your load balancing algorithm, and all port groups on the vDS must be set to use it. For this reason multi-nic vMotion and IP Hash load balancing a mutually exclusive. This won’t be a problem if a single link is sufficient for vMotion traffic. You will need to ensure you use Network IO Control to prevent any one traffic type flooding out the others. You can read about the problems with EtherChannel and LACP vs load balancing based on physical NIC load here – http://longwhiteclouds.com/2012/04/10/etherchannel-and-ip-hash-or-load-based-teaming/

      • nikolab888
        December 2, 2012 at 9:34 pm

        guys, just be very carefull not to do Etherchanell on two physically separate chassis (switches), unless they for sure support and provide multichassis-etherchanell.

  6. VMguru
    September 13, 2012 at 4:49 am

    I’m wondering what the difference will be with a link failure between standby or Unused. If I have it set to standby and a link failure occurs during a vMotion, the vmkernel will get reassigned and the vMotion should complete successfully.
    But what about if it’s set to unused? What will happen to the vMotion that is using that path during a link failure, will it timeout and fail? Or will vMotion stop trying to use that VMK?

    • September 15, 2012 at 9:55 pm

      If on your vmk vMotion port the other uplinks are set to unused and the active uplink fails the vmk port will loose network access. But the remaining vmkernel ports configured for vMotion will continue the vMotion operations and provided there is still one surviving vMotion vmk port the vMotion operations will complete successfully. You can set the other uplinks to standby instead of unused. But given the redundancy at the vMotion vmk level there isn’t necessarily a requirement for that. I think Multi-NIC vMotion will be far more heavily used once the unicast port flooding situation is addressed.

  7. nikolab888
    December 2, 2012 at 9:31 pm

    just great man! you might brought “The Gotcha” part as it’s own article, it worth it. Great explanation of what is going on. thanx!

  1. July 16, 2012 at 11:53 pm
  2. September 8, 2012 at 2:49 pm
  3. September 12, 2012 at 2:44 am
  4. September 17, 2012 at 4:32 pm

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: