FAQ: CPU microcode updates and VMware ESXi

"CPU microcode update available. The guest OS tried to update the microcode ..., but VMware ESX does not allow microcode patches to be applied from within a virtual machine."
I recently installed a Windows Server 2012 on a VMware ESXi 5.0 host running on an older HP hardware (Proliant DL380 G5). Each time Windows reboots the message above is logged on the ESXi host. According to the message text it looks like I can safely ignore it, since I do not have any issues with this host or the VM. However, this made me curious - I started some research on the CPU microcode topic and decided to write up a little FAQ:

Disclaimer: I thoroughly researched the information provided below, but if you find something to be inaccurate or plain wrong, or if you can answer any of the questions/uncertainties that I raise then please comment - Thanks! 

1. What is the CPU microcode?

In modern CPUs the microcode is used to assist in translating "high level" machine code instructions into various configuration signals and more detailed circuit-level operations. It helps separate the machine instructions from the underlying electronics (arithmetic logical units, memory channels, IO buses etc.) so that instructions can be designed and altered more freely.

For a more detailed definition, examples and an explanation of why this is beneficial please refer to the Wikipedia article about Microprograms/Microcode.

The microcode of a CPU can be updated to fix known CPU bugs. Yes, CPUs have bugs (also known as errata) that can lead to system instability, strange behavior, false machine check exceptions (MCEs) and OS crashes! Only some of these bugs can be resolved through microcode updates, for others there are workarounds implemented in the mainstream Operating Systems (like the AMD erratum #383), and in very few cases a bug cannot be resolved in software, but needs a redesign of the hardwired CPU logic, so that the buggy revision of a CPU needs to be recalled (this happened with the famous Pentium FDIV bug).

2. How is CPU microcode updated?

A microcode update can be loaded into a processor by using special CPU instructions that are reserved for this purpose. Such updates are not permanent, but are stored in volatile memory - that means with each power cycle the CPU is reset to its built-in microcode, and the update needs to be applied again.

Most commonly CPU microcode updates are applied through a computer's BIOS when it is powered on or reset. Most mainstream Operating Systems (incl. Microsoft Windows, Linux and VMware ESXi) also try to update the CPU microcode at boot time. Microsoft e.g. sometimes publishes Windows updates that include such microcode fixes (see e.g. KB2493989).

3. How does VMware ESXi update CPU microcode?

VMware ESXi 5.x uses the vmkmicrocode utility to update CPU microcode. At boot time the command
   vmkmicrocode -q -d /etc/vmware/microcode
is executed. This command will check if any of the microcode data files in the directory /etc/vmware/microcode (there are three dat-files for Intel processors and one bin-file for AMD CPUs) includes an update that is applicable to the system's CPU(s), and if this is the case it will load the update into each core of each CPU.

I took a closer look at the microcode data files and found them to be to a bit old: the newest Intel data file is of 2010, and in fact these files were not updated by VMware since at least the GA release of ESXi 5.0 (I have not checked earlier ESXi versions). For me it's hard to believe that the current CPU models of Intel and AMD do not have any bugs that affect ESXi and need to be fixed through microcode updates ... so I guess that VMware relies on the hardware manufacturers to include the necessary microcode updates in the BIOS of their servers and doesn't really care that much about the update files that they supply with ESXi themselves.

4. Should I even care?

It depends. If you are running ESXi on a VMware supported modern server hardware with the latest vendor BIOS applied then you usually do not need to care about microcode updates. In such cases known problems with processors will be addressed by the hardware vendor that will include CPU microcode updates in BIOS updates.

If you are running older or whitebox/unsupported hardware, experience strange issues with it and have already ruled out other possible causes (e.g. bad RAM, unstable power supplies etc.) then you might want to look at the CPU microcode.

In the very most cases you really shouldn't bother ... But if you are paranoid or an "Update Junkie" ;-) then read on!

5. Where do you get the latest CPU microcodes?

Intel makes CPU microcode updates available on their Download Center pages. You can search these pages for "Linux Processor Microcode Data Files" to get to the relevant download links. The current version (as of today) is of Feb 22nd 2013 and consists of multiple dat-files packed into a tgz-file.

AMD used to provide the same directly on a web site names http://www.amd64.org, but this site is no longer available (for the story behind that read this post on the Gentoo Forums). They now seem to feed these files into a Linux software package called linux-firmware. You can download this package by searching for it in the Launchpad.net Ubuntu repository and will find the microcode bin-files in the sub directory amd-ucode of the downloaded tgz-archive.
The latest version is of Sep 10th 2010 though - I wonder if this is really still up-to-date or if AMD has completely dropped the ball on that now ...

6. How do I apply updated microcode in VMware ESXi?

Once you have unpacked the data files from the above mentioned sources you need to upload them to a datastore of your ESXi host and can then use the vmkmicrocode utility to check whether they are applicable to your CPU and also to apply them.
The command
   vmkmicrocode -c /vmfs/volumes/DS1/microcode.dat
will check whether the updates that are included in the referenced data file apply to your CPU. It will output a message like
Warning: CheckUpdate failed, num=140 version=0x306a9, platformID=4000000000000, status=0xbad0001
Warning: processor match failed
for each update that is not applicable to your CPU. If there is an update included for your CPU then it will log a message like this:
PCPU0 with revision (a) can use the update in file /vmfs/volumes/DS1/microcode.dat
update number 128       version(1), revision(23), date(0x1092013), size(2048)
for each core of the CPU. In this case you can finally apply the update by using the command
   vmkmicrocode -f /vmfs/volumes/DS1/microcode.dat

7. How can I install a microcode update file in ESXi so that it is automatically applied at each boot?

There are two methods to achieve that:

1. Copy the data file to a datastore that is accessible by the host and add the command to apply the file (see above) to the local boot script /etc/rc.local.

2. Install a VIB file that I created using the commands

   esxcli network firewall ruleset set -e true -r httpClient
   esxcli software vib install -v http://files.v-front.de/cpu-microcode-1.0.0.x86_64.vib --no-sig-check

in an ESXi shell.
This will install the current versions of the Intel and AMD microcode files (from the sources mentioned above) into the default directory /etc/vmware/microcode. To manually check and apply these updates before the next reboot run the command "vmkmicrocode -q -d /etc/vmware/microcode" then.

Update (2013-10-30):

In the meantime Intel made updated CPU microcodes available and also AMD has published new ones! Please find a new version of the VIB file at the URL
   http://files.v-front.de/cpu-microcode-1.1.0-1.x86_64.vib

Install it like mentioned above and you will get the Intel files of 2013-08-08 and the AMD files of 2013-07-10. By the way, the new AMD microcode includes a fix for an issue that affects ESXi resulting in a PSOD during vMotion with Opteron 62xx CPUs! For details see this forums thread: https://communities.vmware.com/thread/432033.

Update (2013-10-31):

I was just made aware that there is an even newer Intel microcode file available (of 2013-09-06), so I quickly compiled another version of the VIB. Here is the URL
   http://files.v-front.de/cpu-microcode-1.2.0-2.x86_64.vib

The new Intel file includes important fixes for Haswell CPUs. Thanks to Евгений Рощин for pointing this out (I accidentally deleted your comment to this post, sorry!).

Please note: I also had to duplicate the data files in the new VIB file, because the vmkmicrocode utility of ESXi 5.5 now looks into vendor specific sub directories (amd and intel) of /etc/vmware/microcode. So, this is the first version that will also work with ESXi 5.5!

Update (2014-02-10):

I updated my cpu-microcode package to include Intel's updates of 2014-01-22 - Thanks to S. Catnipp for pointing me to the new version.
The package is now available at the V-Front Online Depot, and can be installed or updated from there by using
esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -n cpu-microcode -d http://vibsdepot.v-front.de --no-sig-check
in an ESXi shell.


This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.



17 comments:

  1. I've always seeing these logs and they made me curious but never looked them up... Thanks for clarification. :)

    ReplyDelete
  2. Please create a VIB for the latest Intel microcode.

    I am about to install 5.5, and have not been able to get the CentOS 5.3 build environment working yet to create my own files.

    ReplyDelete
    Replies
    1. Hi Anonymous,

      I have made a new VIB file that includes the Intel microcode files of 2013-08-08 and the AMD microcode files of 2013-07-10. The URL is
      http://files.v-front.de/cpu-microcode-1.1.0-1.x86_64.vib

      - Andreas

      Delete
    2. BTW, you don't need a CentOS build environment to create a VIB file! Just use my Windows based ESXi Community Packaging Tools ...

      Delete
  3. Hi Andreas,

    great work so I can use my Homelab perfect.

    At the Moment I have get 3 cpu-microcode shown if I make a vib list from the depot. 2 are shown as status Downgrade and on the 1.4.0-1 is shown as installed.

    Is this OK or should i change something?

    Regards, Steffen

    ReplyDelete
    Replies
    1. Hi Steffen,

      that's perfect. It means you have installed the latest version, and installing any of the older versions would result in a downgrade.

      Andreas

      Delete
  4. Due to this microcode error i am unable to install Win 2012 R2 on VMWare Virtual Machine.is there any solution to install win 2012 R2 with such error?

    ReplyDelete
    Replies
    1. Well, that is interesting! I never heard before that a Guest OS installation would fail because of an outdated CPU microcode ... I suggest that you follow the instructions in my post to update the microcode of your host machine's CPUs. Try to update the BIOS to the latest available version first.

      Delete
  5. I've written a VMware blog post on how to use the microcode update feature in ESXi 6.0: http://blogs.vmware.com/vsphere/2015/05/using-esxi-6-0-cpu-microcode-loading-feature.html

    I linked back to this post for its info about 5.5 and earlier. Thanks!

    ReplyDelete
    Replies
    1. Hi Tim,

      Thanks for sharing and linking back to my post, much appreciated!
      In your blog post there are two links not working (starting with http://mts.VMware.com). I didn't find a way to comment there, so I hope you read my answer here and are able to fix that.

      Andreas

      Delete
  6. Hi everyone.. Question to the experts.. I have a 5.5 ESXi environment right now and am getting a microcode error when attempting to bring up a nested ESXi 6.0 host VM. It's a budget AMD based server and a BIOS update is likely not available. I am wondering if the latest microcode update is applied within the 5.5 physical host it would allow the 6.0 guest VM to boot without complaining?

    ReplyDelete
  7. Hi Anonymous,

    if the nested host is complaining about not being able to update the microcode then updating it on the physical host should help. Use the cpu5-microcode package for ESXi 5.5.

    Andreas

    ReplyDelete
    Replies
    1. Thanks for your assistance. Strange that these microcode packages don't get included as you apply standard patches to our hosts. I will read through the walkthrough above again, but is it as simple as installing a VIB package that is provided directly from VMware?

      Regards,
      Adam Tyler

      Delete
    2. Hmm.. Just went and tried this but got the following error. Any idea what I am doing wrong?
      ~~~~
      esxcli software vib install -d "/vmfs/volumes/datstore/Installs/cpu5-microcode-1.7.0-3.x86_64.vib" --no-sig-check

      Could not download from depot at /vmfs/volumes/datastore/Installs/cpu5-microcode-1.7.0-3.x86_64.vib/index.xml, skipping (('/vmfs/volumes/datastore/Installs/cpu5-microcode-1.7.0-3.x86_64.vib/index.xml', '', "[Errno 4]

      IOError: "))

      Regards,
      Adam Tyler

      Delete
    3. I apologize for all the posts. I managed to apply the offline bundle with the esxcli command and was able to start my nested ESXi 6.0 install successfully! Thanks for your help and the great blog post.!

      Regards,
      Adam Tyler

      Delete
    4. In my case, when I received this issue I was able to resolve the problem by changing the number of processors on the VM that I was getting this error on. I had an older server we were using for testing environments for various reasons. I found that on my older SuperMicro server would start having this issue if I would try to change the CPU settings to have anymore than virtual sockets(1core per socket). I've looked across the net to resolve my issue and sadly what's above didn't help, in my case, so I thought I'd spread the word about this possible fix.

      Delete
  8. Hi

    Amd64.org is still/again available: http://www.amd64.org/microcode.html

    Regards,
    Alex

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!