Home > VMware > HP Critical Advisory – NC522 and NC523 10Gb/s Server Adapters

HP Critical Advisory – NC522 and NC523 10Gb/s Server Adapters

HP has issued a critical customer advisory regarding some of their high performance server adapters. I’ve had a number of customers impacted by the NC522 and NC523 10Gb/s server adapters losing connectivity. It’s good to see there is a firmware update now that may solve this.

Customer Advisory c02964542, ProLiant and StorageWorks Systems: Certain NC-Series and CN1000Q Network Adapters – FIRMWARE UPGRADE REQUIRED to Avoid the Loss and Automatic Recovery of Ethernet Connectivity or Adapter Unresponsiveness Requiring a Server Reboot to Recover, has been released/revised.

Find the document at: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02964542

Affected Item(s):

HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter, HP ProLiant DL370 G6 Server series, HP NC522SFP Dual Port 10GbE Gigabit Server Adapter, HP NC522m Dual Port 10GbE Multifunction BL-c Adapter, HP NC375T PCI Express Quad Port Gigabit Server Adapter, HP ProLiant DL580 G7 Server series, HP NC523SFP 10Gb 2-port Server Adapter, HP ProLiant DL980 G7 Server series, HP ProLiant DL585 G7 Server series, HP CN1000Q Dual Port Converged Network Adapter, HP Business Data Warehouse Appliance, HP D2D4312 Backup System, HP D2D4324 Backup System


IMPORTANT: The network adapter firmware and driver upgrades provided in the Resolution are required to prevent the loss and recovery of Ethernet connectivity, or adapter unresponsiveness requiring a reboot to recover, from occurring. HP recommends performing these upgrades at the customer´s earliest possible convenience. Neglecting to perform the recommended action and not performing the recommended resolution could result in the potential for subsequent errors to occur.

The network adapters listed in the Scope section (below) may encounter either of the following:

* The adapter may temporarily lose Ethernet connectivity, and then automatically recover.


* The adapter may stop responding, requiring a server reboot to recover the operation of the adapter.

Note: There is a low probability of this occurring when operating under a normal network workload.

Driver and Firmware locations and instructions can be found at the following location:





This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +.

  1. August 31, 2011 at 8:54 am

    What is the recommended Firmware and Driver by HP for both the NICs? Thanks

  2. Matt
    December 29, 2011 at 7:44 am

    This “Fix” was a no go for us. We have 20 servers with 2 10Gb NC523SFPs and we continue to have an issue with random NIC flapping. One of the 2 nics will randomly lose connection to the switch for about 2 seconds, long enough to trigger that network redundancy was lost. We are running DL380 G7s and we have two separate datacenters with 10 Esx hosts in each one. We even tried putting two NC522SFPs in a server and when it “flapped” the only way I could get the network connection to stop bouncing was to reboot the server.

    • December 29, 2011 at 11:45 am

      Hi Matt,

      I can believe that. I know a number of customers still have ongoing problems with their NC522 and NC523 NIC’s and are still experiencing some disconnections, although much less frequently and more minor on the whole. The real solution would be to change to NIC’s that don’t have these problems. I use the Intel X520 dual port cards and have not had any issues. But in a lot of cases it may not be possible to change. I was made aware recently of another firmware revision for these cards, not sure about the driver. So it’s even more important with these cards to keep up to date with firmware and drivers, but really these problems should not happen with equipment that has been tested and certified to work.

  3. kaladark
    January 12, 2012 at 10:53 pm

    Hi, i follow the steps but only upgrades the driver version, the firmware is the same, how can i upgrade the firmware??

  4. Matt
    January 13, 2012 at 3:52 am

    I tried going to support.hp.com, but the drivers are not accessable that way. Instead, go to http://www.hp.com and then in the search box on the top right of the page, put the card in that you are looking for. A list where you can select your operating system should come up. Click “vmware esx/esxi 4.1”. Download the HP Qlogic P2P Flash Update Kit. You have to create a CD/DVD to boot off of, and it will patch the firmware from that.

    I don’t know if this works, but it is the direct link for the NC523SFP card.


  5. Simon
    January 30, 2012 at 10:57 pm

    We’re having similar problems with our storage NICs – HP NC522SFP. In one case is caused disk corruption on 30+ VM’s.

    It looks like HP don’t provide the firmware when using as ESX hosts and instead point you to QLogic -> http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=3913537&prodTypeId=329290&objectID=c02069256

    And the Qlogic firware update process just hangs for us.

    Do we take from this that HP don’t support ESX on their kit ??

    • January 30, 2012 at 11:42 pm

      Hi Simon, The customers I’ve helped with issues related to this advisory got the firmware and drivers directly from HP support (and had HP Techs update them). It is a bit of a process to go through when doing the update though as you have to boot from CD or USB stick. HP definitely do support the servers if it is one they have certified that is on the VMware HCL. Your colleagues in NZ IT experienced exactly the same issue and I helped them resolve it.

    • January 31, 2012 at 12:38 am

      Hi Simon, Just so you are aware there are a bunch of other BIOS settings that are recommended in addition to the drivers and firmware. It’s recommended that you configure static high performance for power management and enhanced cooling. One of the main reasons for this is that the cards are overheating under load, which is one of the problems. With these settings and the new drivers / firmware the symptoms should be greatly reduced, although I’m not 100% sure it’s a complete fix.

      • Matt
        January 31, 2012 at 2:50 am

        HP techs have confirmed to me that for some customers their firmware and drivers fixed the flapping, but for others it continues. Our solution was to replace our NC523SFP cards with Intel x520-DA2 cards. We have had no more issue and are now able to move on with our migration plans.

        HP does have some techs configuring firmware dumps for the network cards to try to capture what is causing the problem, but we had already replaced the nics when HP sent this to me. When calling them, make sure you ask about this.

    • January 31, 2012 at 10:02 pm

      Hi Simon, Thanks for your update today and for the link to the download for the service pack (http://h18004.www1.hp.com/products/servers/service_packs/en/index.html), which I have included in the post now. I hope this fixes the problems. Please post a comment back here if it is successful.

  6. Simon
    February 14, 2012 at 1:10 am

    Many thanks for all your help. We have run the SPP against all our hosts now (and updated the ‘VMWare’ NIC driver) and so far… have not experienced the same issue. We reviewed the BIOS settings previously when we had PSOD crashes when the hosts first went in, but that was a seperate issue that was resolved by a BIOS update.

    Anyway hopefully the latest issue is resolved (fingers crossed!)…

    I notice HP haven’t updated the advisory I linked to above which still suggests people go to Qlogic for the firmware, which is strange considering that the Qlogic firware hangs on these servers (our 6 core CPU builds anyway), and the HP SPP is bootable and does update the firmware succesfully.

    • February 16, 2012 at 8:37 am

      Great to hear Simon. Weird that the advisory hasn’t been updated. I do hope that your problems are gone.

  7. J
    February 29, 2012 at 4:32 pm

    We spent 2 long months trying to get these adapters to work on our DL580 G7 servers with VMware ESXi 4.1 u1 and had constant issues – many of the same issues Simon was reporting. We use NFS and iSCSI datastores on a NetApp using jumbo frames and Nexus 5ks and we had many SQL db and vmdk corruptions due to the NC522SFP’s experiencing “firmware hangs” or network entropy states…each week we had a new driver to update from HP and Qlogic but they never fixed the issues. We had to pull them and replace them with NC552SFP cards (Emulex OEM). Have not had one single issue since we installed the 552’s. We swapped them out in late December, so I don’t know if recent firmware has fixed the issues. Good riddance to QLogic – their high performance ethernet cards should have never passed QA. They put my organization through the ringer.

    BTW – we also had similar issues with their QLE8242 CNA’s and the NC522SFP was suppossed to bring stability to our environment…double whammy.

    • March 1, 2012 at 9:12 am

      Hi J, thanks for the feedback, it’s unfortunate how many customers have been impacted by these problems. The first release of vSphere 5 from HP had issues with the OneConnect Emulex NIC’s (mainly in blade infrastructures). This has been fixed in the latest patch releases. For a while the OneConnect wasn’t on the HCL and wasn’t supported by HP. So when the time is right to upgrade to ESXi 5 it would pay to run a test for a bit with the OneConnect cards and make sure you use the latest patch release.

  8. Jeff Smith
    April 24, 2012 at 2:50 pm

    We have 5 x HP DL585G7 servers with the onboard Quad Port HP NC375i, a separate Quad Port 1G NC375T and the Dual Port 10G NC522SFP (Yes that is 10 NIC’s per server).

    We have never had reliable networking with these servers (ESX 4.1, ESXi 4.1, ESXi5.0) and it is constantly a battle between ESXi versions and updates, driver updates, firmware updates and hope to keep things running. The only reliable way to ensure network connectivity is to reboot the servers every 4 weeks or so, before the network cards poop themselves.

    I have calls logged with HP, If I ever hear back with a resolution, I’ll let you know.
    Jeff Smith @jeffsmithtbp

  9. Jeff Smith
    May 10, 2012 at 5:52 pm

    Hi All

    As an update to this issue, I have been in contact with HP support for the last few weeks about this problem and it has done the rounds of their various support people. The short answer is I will be receiving new Main Boards and possibly NIC’s for my 5 x DL585G7 servers in late May. I can’t give any details unfortunately because the new boards aren’t available yet but they are an update, not a direct replacement, of the existing boards. If you are still having this issue with your HP servers and these (IMO) dodgy QLogic NIC’s hassle HP support, log a case, follow it up and don’t take “No” for an answer until they rectify the problem.

    Apparently the firmware and driver updates do fix the problems that some people are seeing so they are definitely worth a shot, but ours needed re-engineering to get past the issue.

    Jeff Smith @jeffsmithtbp

    • Rod Hope
      May 11, 2012 at 9:34 pm

      Hi Jeff,

      I just wanted to let you know that you aren’t alone on this one. After a good few calls with HP on this issue, they agreed to send us new system boards and NICs for our DL580 G7 servers which we started switching out. The first server went well but the second box now can’t load the onboard NIC drivers even though we have the latest firmware (4.0.585) and the latest drivers (4.0.614). I have an open ticket with VMware and with HP on this issue, so hopefully have resolution soon. I will post the resolution here (if we ever get one)….

      Rod Hope

  10. Jeff Smith
    June 12, 2012 at 1:09 pm

    Hi All

    HP came onsite almost two weeks ago to replace all 10 of the NIC’s on two of my servers. They had to replace the riser board that houses the 4 x LOM NC375i NIC’s, they replaced the 4 Port NC375T and the 2 Port NC522SFP.

    When I restarted the server after the replacement the only issue was the ILOM had been reset (it resides on the riser board), apart from that no issues.

    ESX started fine and all networking settings were the same, no issues with drivers or hardware. I migrated some test VM’s to these servers and have not had any outages yet.

    Everything seems good now and there have been no problems since. I can’t guarantee that this will fix the issue, but it’s a start and HP appear confident for the first time. I have HP coming onsite later this week to do the same thing on the last of my servers.

    I also have another client with same problem on a number of servers that are experiencing the same issue. They have vSphere 4.1, but are seeing the same problems I have with vSphere 5.0u1. Same escalation with HP and same fix.

    I’ll keep you updated on the progress

    Jeff Smith @jeffsmithtbp

  11. Denns Fisher
    September 10, 2012 at 8:08 am

    Did you see that a bunch of updates for some of these NIC cards just came out dated September 4, 2012. HP must still be working on this problem.
    I agree that the way to get at the correct link is to go to http://www.hp.com and go to drivers and search on your card number.

  1. September 11, 2011 at 2:03 pm

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: