Are ESXi 5.x patches cumulative?

I find this question being asked again and again in the VMware forums by people who cannot just use Update Manager for patching, and most times the answers that are given there are a bit vague and unclear or even plain wrong. This is because ESXi patching is a really complex and not well documented matter. So I need to provide some background information before I'm going to answer this question:

The original ESXi system provided by VMware is made up of multiple software packages (currently 60 for ESXi 5.0 and 65 for ESXi 5.1) called VIBs (VMware Installation Bundles). There is e.g. one large VIB for the base system (esx-base), one for the VMware Tools (tools-light), and most of the remaining VIBs are hardware device drivers (e.g. net-e1000 or sata-ahci).
With an ESXi patch one or multiple of these VIBs are updated (or in very rare cases added). An example: The latest patch for ESXi 5.0 (ESXi-5.0.0-201209001 of Sep 2012) updates the VIBs esx-base, tools-light and misc-drivers.

ESXi patches are provided by VMware in the format of so-called Offline Bundles (in this case also called Patch Bundles) in ZIP format. Kyle Gleed has explained in this blog post how you can download them from the Patch download portal.
Now the most important point to understand is that these Patch Bundles do not only include the patched/updated VIBs, but all VIBs that make up an ESXi system, and in fact the latest version of them all.

So, strictly speaking the answer to the original question is No, because a single patch does only change one or multiple VIBs and thus just cannot be cumulative. However, patches are not provided separately, but only as parts of complete Patch Bundles, and the Patch Bundles are indeed cumulative! That means we need to install only the latest Patch Bundle to make ESXi fully patched.

To prove this claim I have complied a spreadsheet that shows all the Patch Bundles that have been released so far for ESXi 5.0 and 5.1, each with complete lists of the included VIB packages and their versions. VIBs that are updated in a Patch Bundle are marked in red color. I plan to keep this spread sheet updated when new patches are released.

Next question is: How do we install the latest Patch Bundle in the right way?

Before we can answer this question we need to understand the concept of Image Profiles. Each Patch Bundle includes exactly two or four Image Profiles, and these are just logical sets of VIB packages. As an example we look at the four Image Profiles that are included in the latest ESXi 5.0 patch mentioned above:
  1. ESXi-5.0.0-20120904001-standard contains all 60 (resp. 65) VIB packages. The three that are patched are included with their new versions, all others are included with the versions that were also included in the previous Patch Bundle (ESXi-5.0.0-201207001).
  2. ESXi-5.0.0-20120904001-no-tools is the same, but without the VMware Tools VIB (tools-light). This profile is typically used to build streamlined thin ESXi images, e.g. for AutoDeploy.
  3. ESXi-5.0.0-20120901001s-standard contains all 60 (resp. 65) VIB packages (like the first one), but only the patched VIBs that include security fixes (only esx-base in this case) are included with their new versions, and all others are included with the versions that were also included in the previous Patch Bundle.
  4. ESXi-5.0.0-20120901001s-no-tools is the same as the third one, but without the VMware Tools VIB (tools-light).
If a Patch Bundle does not include any security patches then it will include only the first two Image Profiles. Because VMware makes a clear distinction between security fixes and functional bug fixes it is also perfectly normal that a Patch Bundle includes two different versions of one VIB. In our example the ESXi-5.0.0-20120904001-standard profile includes esx-base version 5.0.0-1.22.821926, and the ESXi-5.0.0-20120901001s-standard profile includes esx-base version 5.0.0-1.21.822948! In the above mentioned spreadsheet both types of profiles are listed (if available) to visualize the differences.

For most machines you will want to update the system with the regular standard Image Profile. The following esxcli commands list the Image Profiles that are available in a Patch Bundle and install/update the system with one of them (run them in a local or remote ESXi shell):
# List image Profiles that are provided by the Patch Bundle
# (Replace /path/to with the datastore path of the Patch Bundle)
#
esxcli software sources profile list -d /path/to/ESXi500-201209001.zip
#
# The output will look like this:
# Name                              Vendor        Acceptance Level
# --------------------------------  ------------  ----------------
# ESXi-5.0.0-20120904001-no-tools   VMware, Inc.  PartnerSupported
# ESXi-5.0.0-20120904001-standard   VMware, Inc.  PartnerSupported
# ESXi-5.0.0-20120901001s-standard  VMware, Inc.  PartnerSupported
# ESXi-5.0.0-20120901001s-no-tools  VMware, Inc.  PartnerSupported
#
# Now update the system with the regular standard profile:
#
esxcli software profile update -d /path/to/ESXi500-201209001.zip -p ESXi-5.0.0-20120904001-standard
#
There are two different ways to apply the Image Profile: With esxcli software profile update (like in the above example) or with esxcli software profile install, and it is very important to understand the difference: The install command will remove all existing VIB packages from the installed system and replace them with all VIB packages that are part of the Image Profile. That means it would also remove any installed package that is not included in the Image Profile and downgrade any installed package that has a newer version than the one in the Image Profile! In most cases it is safer to use the update command instead: It will keep all installed packages that are not included in the Image Profile or have a higher version number than the one in the Image Profile.

If you have ever manually updated an ESXi device driver by installing one of the Offline Bundles that are available here for ESXi 5.0 and here for ESXi 5.1 then you should definitely use the update command to keep them. The same is true if you have installed your ESXi system with a customized ISO that was provided by a hardware vendor (like HP, Dell, etc.).

I hope that this post will answer the question about cumulative patches once and for all (although I will of course answer any related question in the comments). For those of you who want to learn even more about this topic here are some links:

17 comments:

  1. thanks for some really good information in this blog post!
    Do you know how PowerCLI installs patches when using the "Install-VMHostPatch" cmdlet? I mean is it equivalent to the "esxcli software profile update" or "esxcli software profile install" command? I'm asking this because I couldn't find an "Update-VMHostPatch" cmdlet.

    ReplyDelete
    Replies
    1. I don't know. Need to look into that.
      However, please note that you can also run esxcli commands remotely through PowerCLI. See e.g. Luc's post here:
      http://www.lucd.info/2012/10/15/update-a-remote-server-to-esxi-5-1/

      Delete
  2. Thanks for this article.
    The only one I found that clearly shows how the patches are cumulative, and the excel doc is a very good illustration.
    All other articles to be found are more like 'they are cumulative (or are not), because someone from VMware said so in a forum', and even VMware employees appear to have contradicting opinions on the issue.

    ReplyDelete
    Replies
    1. Hi Patrick,
      thanks for the feedback! You exactly named the reason for why and how I wrote this article. I'm glad you found it useful.
      Andreas

      Delete
  3. Thank you! Finally I know what I'm doing during patching ESXi servers.

    ReplyDelete
  4. Thank you Andreas

    Your article has been of great help. We had been struggling to patch the customised ISO by HP/DELL and it was resulting in PSOD. Neither vmware nor the hardware vendor was able to provide a soltuion even after weeks of troubleshooting.

    Referring to your article helped me understand the patch process and how to patch.

    Regards
    Satish

    ReplyDelete
    Replies
    1. Thanks Satish,

      I'm glad that you found my article helpful.

      Andreas

      Delete
  5. Thank you sir, this helped a casual ESXi user tremendously!

    ReplyDelete
  6. Awesome work. Has help a lot - and cleared up a few questions I had. Such a pain to sift through hundreds of posts and vmware support docs. This was all I needed.
    Thanks

    ReplyDelete
  7. Great post and still clearing up confusion to this day! Thank you for keeping your Patch Matrix up to date? What is your process for gathering the data populating the matrix?

    ReplyDelete
    Replies
    1. Hi Anonymous,

      thank your for your comment!

      The process of keeping the VIBMatrix up-to-date is semi-automated with a PowerCLI-script that queries the VMware Online Depot and some Excel formatting magic, but it also involves manual work today.

      If I find the time I will work on fully automating this process like I already did for my new service - the ESXi Patch Tracker.

      Andreas

      Delete
    2. thank you very much.i use "vihostupdate.pl --server 192.168.1.1 --username root --password vmware --bundle c:\update-from-esx4.1-4.1_update02.zip --install " and know nothing about it. I will use "esxcli software profile update".Could you write some about vihostupdate.pl update method? Thank you again.

      Delete
    3. Hi Anonymous,

      the vihostupdate.pl script was for ESXi 4.1 only (which is quite old now). If you are still on ESXi 4.1 then you should upgrade to 5.0 using vihostupdate.pl, and from there to 5.5 or 6.0 using esxcli (a direct upgrade from 4.1 to 6.0 is not supported).

      Andreas

      Delete
  8. I know the topic is old, however could you explain why some of the patch archives are almost twice the size than others?


    The size from one patch ZIP file to the next jumps from typically ~ 350 MB to something around 650 MB.

    However I did update from 6.0 Update 1 directly to the latest patch (as of writing: ESXi600-201602001 / build 3568940) and saw that all VIBs mentioned in your patch tracker that got updated between Update 1 and this patch were updated as well (for example 'xhci-xhci' which got updated just before the mentioned patch in build 3380124).

    ReplyDelete
    Replies
    1. Hi Anonymous,

      the reason for the differing patch bundle sizes is that some patch bundles include an additional security-only Imageprofile and possibly *two* versions of the large esx-base and tools-light VIBs.

      Example:
      The ESXi 6.0 Update 1b bundle includes esx-base and tools-light versions 6.0.0-1.26.3380124 (from Imageprofile ESXi-6.0.0-20160104001-standard), but also esx-base and tools-light versions 6.0.0-1.23.3341439 (from Imageprofile ESXi-6.0.0-20160101001s-standard).
      This nearly doubles its size (compared to a patch bundle that does not include an additional security-only Imageprofile).

      Andreas

      Delete
  9. Andreas, this is a life-saver post.

    While I am technically an I.T. professional I haven't done much with VMware for at least a year or two, and I spent the better part of a day today trying to figure out the disastrous VMware support/download portals and documentation which are, to put it nicely, very unhelpfully designed. It's amazing how many words they can put on a page without actually clarifying the matter at hand.

    To make matters worse there are TONS of misleading and misinforming blog and forum posts out there, including some right on VMware's own blog, that do little to clear up the confusion about exactly how the patching process works. When I ran across your post today I recalled your domain from when I was doing more with VMware, and I'm very glad I found it again.

    One of the problems I think is that it's much more complicated for those of us using free ESXi licenses. It would surely be much simpler to just use vCenter/Update Manager, but this is not an option for us. (It also leads me to wonder if VMware is not actually too sad that free users are left in perpetual confusion about how to get things done. ;) )

    Just wanted to let you know that 4.5 years after you posted this, it's still doing a lot of good. :)


    Phil in California

    ReplyDelete
  10. Info still useful today.
    Going from 6.0.0-2.34.3620759 to 6.0.0-3.110.10719132.

    VMware obviously want to popularise their software, hook you up and eventually make you pay for licenses.
    Citrix Xen implement the same strategy. Manual updating is non-trivial.

    Adam from London

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!