How to do an Online Virtual Connect firmware upgrade

Okay, this is a follow-up to my previous post ... I was finally able to find out on my own how to do this. The answer is in HP's white paper "HP Virtual Connect Firmware Upgrade Steps and Procedures". This is a must read for anyone being concerned with the VC firmware upgrade process, I will try to summarize the most important points here.

You must use the Virtual Connect Support Utility (VCSU). The current version is 1.60 and is available for download here.

It helps to understand how the VCSU does the upgrade: First it uploads the new firmware to all VC modules simultaneously. This phase is absolutely uncritical, because the VC modules continue working normally during the upload. If you use the default parameters it will then activate the new firmware by rebooting the VC modules one after the other in a controlled manner - and this is the process that really impacts the network availability of your hosts and VMs!
Why? The controlled reboot takes 20 or more seconds, and - of course - the VC module will not properly forward and receive network traffic during that time. However, the blade servers, resp. their NICs that are connected to this module are not properly disconnected during that time, i.e. they do not get a link down notification! If you use the default failover detection method for your virtual switches (Link state only) the hosts will continue using the up-links to the module that is just rebooting, and this results in a loss of network connectivity.

So, how do you cope with that? One possible work around is to use Beacon probing as the failover detection method for the virtual switches. But in my opinion this is not the best and easiest choice. No, the real answer is on page 13 of the white paper:
"For the customer environments where changing Network Failover Detection options or HA settings is not possible, utilizing VCSU manual firmware activation order (-of manual) is recommended. In this case, modules will be updated but not activated and the user will need to perform manual activation by resetting (rebooting) modules via OA GUI or CLI interface. This option will eliminate potential of up to 20 sec network outage that may occur on a graceful shutdown of VC Ethernet and FlexFabric modules."
Using the manual activation order (parameters "-oe manual" and "-of manual") ensures that the VCSU will not gracefully reboot the VC modules at all. You then need to do that on your own (just manual), by resetting the VC modules through the Onboard Administrator (OA). When you do a hard reset of a VC module the connected hosts will immediately get link down notifications, just as if the module suddenly fails or loses all its own up-links because the external switch failed. You should just wait about 5 minutes for the resetted module to get fully online before you reset the second one.

If your ESX(i) hosts are properly and redundantly configured you will notice only a minimal network interruption during this process. In my test it was just a single ping drop.

Yes, that's the whole secret of doing an online VC firmware upgrade! For me only one questions remains: Why is HP making it so hard to find this information? If you search hp.com for instructions on how to do this you will find tons of useless and contradicting information on this topic, and even their own Support engineers are not able to give a quick and right answer to the question. At least, one of them sent me a copy of the white paper (he could not just provide a link to it, because he was not able to find it on the HP pages...).

6 comments:

  1. Hello and thank you for an nice guide in the HP firmware djungle.

    I can confirm that this guide works. However, worth mentioning is that in ny case the VM's running on the blades did experience a minor distruption on the networking causing applications to loose their connections. Several applications had to be restarted to re-establish their network connection. This happended when the active module was reseted.

    Also, there were some event in system log on VM's that they had timeout to disk. But no known impact there.

    We are running ESXi5, both NFS and network via Flex-10.

    So you might want to run this off-hours or inform users that there might be a short dip.

    Thanks.

    ReplyDelete
  2. Hi Folks,

    Does upgrading the Flex 10 Converged Network Adaptor (HP Emulex 10GbE Converged Network Adapters and 10GbE Network Adapters) requires me to upgrade the Virtual Connect module firmware as well ?

    ReplyDelete
    Replies
    1. It is definitely best practice to upgrade all at once. Please see HP's VMware Recipe for firmware interoperability/compatibility and update the different components using the SPP DVD. See my "HP&VMware Links" page for relevant links.

      Delete
  3. Hi

    Can anybody tell me if it is possible to update 1 module at a time?

    we have 2 Virtual connect modules, 1 with 3.70 Firmware and 1 with 3.60. im using HP smart update manager to update the modules but in order to update to V4.10 both the firmware of the modules must match? so I'm now stuck. i cannot update either to 4.10 because neither of the firmware matches.

    any help with this would be much appreciated

    ReplyDelete
  4. When doing a manual activation process, does anyone know if it matters what order you reboot them in?
    I have 3 enclosures, with uplinks from 6 Flex-10 modules.
    After the firmware update, I assume I have to reboot the primary and standby in enclosure 0 first, then the rest can be in any order?

    any help would be appreciated!

    ReplyDelete
    Replies
    1. Hi Jeff,
      See this white paper ("Virtual Connect Multi-Enclosure Stacking Reference Guide"), maybe you answer is on page 12:

      "When the modules on the left-hand side (the odd side) of enc1 and enc2 reboot to activate the new firmware, the servers in enc1 and enc2 would be isolated from enc0 and would lose network connectivity through the enc0 uplinks..."

      Regards
      Jorge Elizalde

      Delete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!