CPU Microcode Updates revisited: What's new in ESXi 6.0?


Almost two years ago I wrote an FAQ post on CPU Microcode updates with ESXi 5.x. This still gets a lot of hits, but - although the general information about Microcodes still applies - the ESXi related information needs an update, because things have changed with ESXi 6.0.

So what's new and different now?

The following information is largely based on a recent VMware blog post by Tim Mann, a VMkernel developer - I highly recommend reading it. Since I already wrote about this topic with regard to ESXi 5.0 I will only summarize what changed in 6.0:
  • CPU Microcode updates are now applied in a very early boot phase during VMkernel and CPU initialization. That is needed, because some updates cannot be safely applied when the CPU is already doing actual work.
  • The vmkmicrocode utility is no longer available - so there is no way to apply or check a Microcode update during runtime.
  • The Microcode patches are now contained in a separate VIB package (cpu-microcode) that can be updated independently from the base ESXi system.
  • The update files use a new binary blob format. In his blog post Tim provides a way to convert published update files into this format.
Using the information that Tim provided I built a new ESXi 6.0 compatible version of my cpu-microcode package. Please note that this version is not compatible with ESXi 5.x, but an ESXi 5.x compatible version is still available as cpu5-microcode (to avoid naming conflicts and confusion).

To update the CPU Microcode files on an ESXi 6.0 system run the following commands in an ESXi shell:
esxcli software acceptance set --level=CommunitySupported
esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -n cpu-microcode -d https://vibsdepot.v-front.de
A reboot is needed to apply the updated files. I tested the current version (7.0.0) of my package on both an Intel and AMD system, and it worked fine in both cases, but please remember: It includes all published CPU Microcode updates of Intel and AMD as of today, not only the ones that VMware has approved and included in ESXi. So use it at your own risk, it is definitely not supported by VMware! On the other hand you might need an updated CPU Microcode that neither VMware has included in ESXi nor your system vendor in an updated BIOS version - to fix a bug like in this example -, then this package is for you!

For ESXi 5.x use the package cpu5-microcode (instead of cpu-microcode). Apart from that the same commands and comments apply.

If you want to check what Microcode version your CPU has built in and if and to what version it was updated by ESXi then you can use the vsish shell tool. Look at the nodes (0, 1, ... representing each CPU core, resp. thread in case of Hyperthreading) in /hardware/cpu/cpuList. This works for both ESXi 5.x and 6.0. The relevant information is near the end and you can extract it like this:

  vsish -e cat /hardware/cpu/cpuList/0 | grep microcode -A 2

This will list the information for the first core of the first CPU. Use /1, /2 etc. (instead of /0) for the remaining cores, but they should really all list the same information. On one of my whiteboxes using an Intel Core i7-4770 processor the command produces e.g. the following output:

   Number of microcode updates:1
   Original Revision:0x00000012
   Current Revision:0x0000001c

On ESXi 6.0 you can also check what happened during the boot process by looking at the boot log file with a command like this:

   gunzip -cd /var/log/boot.gz | grep MicrocodeUpdate



This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.



3 comments:

  1. Sir, that was a lifesaver (money+time too)... I managed to update the AMD microcode in an old DELL (AMD 2419EE cpus) and the nested ESXi host managed to boot!! I didn't have issues with the (physical) ESXi on the nodes but the nested installations failed (6U2 or 5.5U3) with a PSOD due to a microcode update...

    ReplyDelete
  2. I was about to create an updated cpu-microcode vib for ESXi 6.5 with the latest bundle from Intel released on November 4th 2016.
    The blog post from Tim Mann uses the Vibauthor Fling which requires to run on Suse Linux and is deprecated in the meantime.
    I had hoped to be able to create the vib also with your ESXi Community Packaging Tools. However, it looks like this is not possible or I'm missing something.
    My vib did not just replace the uc_intel.b00 file in /bootbank. It prevented the linking of /bootbank to the actual bootbank partition and the system booted with just a few files in a /bootbank folder that resided on the same filesystem as /.
    The descriptor.xml generated for the cpu-microcode vib's which work, all seem to have some special attributes in the payload tag for which I couldn't really find any good documentation. In his blog post, Tim Mann uses the -p parameter to vibauthor and passes a number (200 or 201) which seem to relate to the boot attribute but the vibauthor documentation is not giving much detail. It just says:

    -p file,type[,name[,boot]]
    This option is deprecated and will be removed from the next release

    So what did you do to create your cpu-microcode vib's in the Online Depot ?
    Is it really required to use vibauthor and what about these nondocumented parameters ?

    geppi

    ReplyDelete
    Replies
    1. Hi geppi,

      yes, I used the vibauthor tool and Tim Mann's instructions to build the VIB file. My own tools are not able to handle this type of VIB payloads. Unfortunately I cannot find a backup of the parameter files that I used with vibauthor back then, but you can look at the descriptor.xml file inside my VIB file to get an idea (use 7zip to extract it from the VIB file).

      Andreas

      Delete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!