The great VMware Tools dilemma


Recently VMware made VMware Tools version 10.0.0 available as a standalone download. This version is now an official downloadable component of vSphere 6.0 with its own Release Notes document and a download page in MyVMware.

In the announcement blog post VMware's Brian Graf writes:
Good news. We have decided that there isn’t any specific reason that VMware Tools builds should be tied to vSphere releases/ESXi builds. Rather, when our engineering teams are ready with key features/updates, we should have the ability to get those benefits out to our customers as quickly as possible.
This release was announced and perceived as a great achievement, but - except for this good news that Brian shared with us - I cannot follow any of the other excited statements that were made about it. It's time to debunk some myths, and it's time to admit that the whole VMware Tools story is still a great mess!


Myth #1: Finally we have Release Notes for VMware Tools

People are complaining a lot about that they have no insight in what was changed or fixed with the various VMware Tools updates that happened in the past. Making version 10.0 available as a "stand-alone" download with its own Release Notes page is supposed to change this.

But, well, the original assumption is wrong: For every updated VIB package of ESXi VMware publishes a dedicated KB article that describes the updates. And that is also true for the tools-light VIBs that represent the embedded VMware Tools package of ESXi. You can consider these KB articles release notes, but VMware makes them really hard to find. However, you can get an easy overview by looking at my ESXiPatches site which summarizes all updated VIB packages of every ESXi (5.x and 6.0) patch and provides links to all those KB articles.

As examples you will find that the release notes of the VMware Tools embedded in ESXi 6.0 U1 are in KB2124717, and the release notes of those embedded in ESXi 5.5 U3 are in KB2110233.


Math #2: Finally we are able to update VMware Tools independently from ESXi patches

The fact is that we have always been able to do this! And this is for two reasons both of which still too many people are not aware of:

1. Every ESXi embedded VMware Tools release (starting with ESXi 3.5!) was and still is made available as a stand-alone download at http://packages.vmware.com/tools/esx/ (for Windows and various Linux Guest OSs). You can download any version there and install it in a VM, no matter what ESXi version the host has that runs the VM.
By the way the standalone version 10.0 is also available here in the directory /tools/releases.

2. VMware Tools releases are - to a great extent - compatible with earlier and newer versions of ESXi. E.g. the Product Interoperability Matrix of the VMware Tools from ESXi 6.0 U1 shows that these are interoperable with older ESXi releases down to 5.0 GA. And the VMware Tools from ESXi 5.5 U3 are even compatible with the full range from ESXi 4.0 to 6.0 U1.


Myth #3: Finally VMware will decouple VMware Tools from ESXi releases and concentrate on developing a single version

This was really my big hope when version 10.0 was published, until ... VMware released ESXi 6.0 Update 1 and 5.5 Update 3, and both updates came with yet another embedded VMware Tools package. And no, these are not identical with the standalone version 10.0! ESXi 6.0 U1 comes with VMware Tools version 9.10.5, and ESXi 5.5 U3 comes with version 9.4.15.


The 50+ Shades of ...

It is the sheer variety of VMware Tools versions that alone can drive you crazy: With the different ESXi patches and updates that were published (since ESXi 5.0 GA) VMware made available no less than 59 different VMware Tools versions, and the standalone version 10.0 is number 60! Lots of opportunities to update Tools, but also lots of opportunities to screw up!

(If you wonder how I came up with this number: I counted the different versions of the tools-light VIBs in my VIBMatrix.)

Why is that?


Why were VMware Tools embedded into ESXi in the first place? ...

So how do you install or upgrade VMware Tools on your VMs today? If you follow VMware's documentation for a manual installation or upgrade (e.g. on Windows VMs) then you use the VM's context menu dialog in the legacy vSphere Client (or the Web Client) to mount the VMware Tools ISO file to the VM and then launch the install from the virtual CDROM inside the VM (if that does not happen automatically). This can also be initiated from the VM console window of the legacy vSphere Client:


Obviously this is only possible, because the ESXi host that runs the VM has access to the appropriate VMware Tools ISO file, and that was installed on it via the tools-light VIB package.

For sure this is a very convenient way to install VMware Tools, and it also works for standalone hosts utilizing the free license. It also allows the ESXi host to upgrade Tools automatically on VM startup, and in combination with vCenter and Update Manager you can even orchestrate a VMware Tools update for multiple or all of your VMs.

So the VMware Tools were embedded into ESXi, because this enabled VMware to implement powerful functionality into ESXi and vCenter for deploying VMware Tools on your VMs and keeping them up-to-date.


... and what issues does this cause?

For a lot of people it is confusing that a VM's Tools Status (Current vs. Out-of-date) that is displayed in the vSphere Client and the Web Client depends on the patch level of the ESXi host that runs the VM. It might change when a VM is migrated between environments with different ESXi patch levels, or when you are in the progress of upgrading the hosts within a cluster.

After hosts are patched it very often happens that the VMware Tools of all VMs are shown as Out-of-date, because the hosts got a newer version of the Tools with the patch. You then feel urged to update the VMware Tools on all VMs (causing reboots and application downtimes) just to make this bad status disappear. Usually people have a hard time to figure out whether this is really necessary, because VMware does not offer a clear guidance. This is why there are different approaches today when it comes to updating VMware Tools, and all of them seem to be valid: Some only update the Tools when they know that the new version fixes an issue that they have. Some only update them with ESXi Update releases or new major versions. Few try to pro-actively update Tools on all VMs quickly after patching the ESXi hosts (which is a huge effort in large environments). Another common approach is to use regular maintenance windows for the VMs to update Tools to the latest release. Almost all these approaches have two things in common: You will never see that all your VMs have up-to-date Tools, and - after the next ESXi patch session - the whole story starts again ...

Few people know that you can break this vicious cycle by updating your VMs with Tools that are considerably newer than your ESXi version, but still backwards compatible. But this will only work if you are not already on the latest vSphere version and only temporarily - until your vSphere version and patch level gets ahead of the Tools version again.

In a blog post back in 2013 I explained how you can tackle the problem of VMware Tools version diversity in your environment, and it was Brian again who brought this idea up again in a recent VMware blogs article: You can configure an ESXi host to look for the VMware Tools ISO files in a central location (e.g. an NFS datastore) that it can share with other ESXi hosts. So you can have one central repository for all your hosts, and thus a single point to update the Tools only once for all hosts. This is great, and I'd recommend every serious VMware admin to look into this possibility! However, there are still two drawbacks:

1. It's a cumbersome manual process to copy the VMware Tools files from an ESXi host (that has them installed in a regular way) to the shared location.
2. It does not seem to be easily possible to have the new standalone version 10.0 of the Tools in such a shared location. Doh!


Ways out of the dilemma

I have three proposals for improving the current situation. I want VMware to ...

1. Stop embedding VMware Tools into ESXi!

Please deliver on the promise to decouple the Tools from vSphere/ESXi releases and patches: Make future version of VMware Tools available as standalone downloads only, like the current 10.0 release!

To be fair and clear: It is already possible to install ESXi without the tools-light VIB, so that it does not have the Tools embedded (by using the -no-tools Image profiles of the various patch releases), but I really want VMware to no longer provide the embedded Tools at all! I think this would free up a significant amount of developers' resources, so that they can fully concentrate on only one set of the Tools (starting with version 10.0).

2. Develop the idea of a central repository and make it a user-friendly feature

As I explained before the motivation for embedding the VMware Tools into ESXi is to make their life cycle (installation and upgrades) manageable within ESXi and vCenter without the need for any third party software. By further developing the idea of using a shared central repository VMware could keep this functionality even if the Tools are no longer embedded in ESXi. How about using an NFS share on the vCenter server as the repository (or on a dedicated appliance to not exclude non-vCenter customers) and providing an easy to use management interface for updating the Tools files in the central location? This would be a great way moving forward ...

On a more radical approach one could even question if we need VMware to manage the Tools' life cycle at all! Actually the VMware Tools are just a software package that you install in your VM guest, and in Enterprise environments IT people use modern software distribution systems (like e.g. Microsoft SCCM for Windows) to deploy software. If you have such a system available you would also use it to deploy and manage VMware Tools!
For Linux VMs you typically manage software by using built-in tools that access the vendors' (or your own) central software repositories, and consequently VMware and most Linux vendors have leveraged the Open Source version of the VMware Tools (the open-vm-tools package) and include that in their repositories.

3. Stop telling us that Tools are "Out-of-date". It's useless!

Why? The Tools status "Out-of-date" just tells us that the ESXi host has a newer version of the Tools embedded than what is currently installed on the VM. So what? Do I need to update the Tools now to fix a potential issue that I'm maybe not yet aware of? But is there not also a risk that only the update will introduce a new issue that did not exist before?

There is no such thing as bug-free software, so there have been quite a few known issues with VMware Tools in the past. Instead of "Out-of-date" I'd really love to see a new status named like "Update needed" that is only displayed if there is a known issue with the Tools version that is currently installed which would be fixed by a new version. This status display would be ideally implemented as a link that opens the relevant KB article in a browser window when you click on it.

I could also think of a status "Update recommended" that tells the user that a newer version of the Tools is available which has new useful features.


What do you think?

Am I exaggerating, or are my ideas just stupid? I'd really love hearing other peoples' opinions - well, always -, but particularly on this topic! It's also a great opportunity to chime in and let VMware know that they need to change something with regards to VMware Tools. So please comment, thanks!



This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.





16 comments:

  1. Hi Andreas,

    I agree, VMware Tools are a mess.
    Especially the VM owners with vCenter server access, that are not that deep into VMware vSphere had a really hard time with all the tools status information.

    That's why I configured all the windows VMs to automaticly upgrade the tools on boot and switched to OSPs for the linux VMs. Now windows shows "current" and linux shows the strange version number, followed by "managed by guest".

    Unfortunately this is not ideal for linux either, as you have to configure the VMware repo inside the VMs to install the packages and the VMware documentation tells you not to use the tools/esx/latest directory, so that you need to specifiy the esxi-version in the repository path.
    So, you are basicly out of date again with each new major host update.

    And since VMware recommends to use the open-vmtools for RHEL 7, etc. I've switched up to the open version wherever I could, to make it a little bit less messy.


    Tim

    ReplyDelete
  2. Another great post and a fine read Andreas! I assume that these new VMware Tools can be installed on VMs created/running on Workstation 12 & Fusion 8?

    ReplyDelete
    Replies
    1. Thanks Alex. Yes, the version 10.0 can also be installed on the desktop hypervisor products! I haven't touched those in my post to not open another can of worms :-)

      Delete
    2. Aha! Please disregard previous question...it's in the release notes!

      Compatibility

      VMware Tools 10.0.0 is compatible with supported versions of VMware vSphere ESXi 5.0 and later, VMware Workstation 12.0 and VMware Fusion 8.0. See VMware Compatibility Guide for more information.

      Delete
  3. I'm with @Tim here - I have removed the VMWare tools ffrom all my RHEL VMs and installed open-vm-tools instead.
    Makes updates easier as they are managed with all the other RHEL updates via yum package manager.

    What advantages do VMware tools have over open-vm-tools?

    ReplyDelete
    Replies
    1. Well, I have no idea. Anyway, in one of their blog posts that I referenced VMware says that "open-vm-tools are the future", so you are doing the right thing!

      Delete
    2. Regarding the advantages or differences for the Linux VMware Tools (host locals, open-vm-tools, OSPs) you might want to check out Brian's blog post and the mentioned KB article: https://blogs.vmware.com/vsphere/2015/09/open-vm-tools-ovt-the-future-of-vmware-tools-for-linux.html
      It's still work in progress here too and mostly a pain in the ass. Time to streamline, VMware!
      Steffen

      Delete
  4. Regarding open-vm-tools - If you have an environment with multiple versions/releases of Linux (ie RHEL 6, RHEL 7), and you want to eventually move to open-vm-tools, then you're stuck managing VMware tools in at least two different ways in the meantime. Open-vm-tools for the newer Linux VMs, and OSP or ISO version on the older Linux VMs. Since it appears that open-vm-tools is only supported (and comes by default) on newer versions (ie 7 and higher).

    ReplyDelete
  5. I spent hours yesterday trying to understand why my new VMs, created using a vCenter VM template with VMware tools v10.0.0 installed would keep power cycling and starting the customization screen again and again.

    As soon as I created a template using a previous version which is embedded on the host, new VMs using that Template worked fine.

    Now it seems obvious that the problem was the host didn't have the up to date VMware tools on it but at the time it was frustrating.


    Naysan

    ReplyDelete
  6. I've found customisation specifications didn't get applied with v10 installed on the template. No attempt at running (no c:\windows\temp\vmware-imc folder). I've had to roll my templates back to tools v9.

    ReplyDelete
  7. Regarding RHEL6:
    I've been syncing lastest tools to my local yum repository and a simple "yum update" is all it takes. Where's the problem?
    RHEL7:
    comes with open-vm-tools. So again where's the problem?

    ReplyDelete
    Replies
    1. No problem if you run Linux with open-vm-tools from a vendor repository (for Linux this is even recommended by VMware now).

      But in all other cases, especially Windows VMs all points above apply.

      Delete
    2. Ok you have a point.
      And there is little breakage with RHEL6, Tools v10 under ESXi 5.5u3a, too. :(
      The tools are reported as "3rd party / independent".
      Despite this:
      # cat /etc/vmware-tools/tools.conf
      [vmtools]
      disable-tools-version=false

      To make things worse RHEL7.2 GA will most probably ship with broken open-vm-tools:
      https://bugzilla.redhat.com/show_bug.cgi?id=1155913

      The guy from vmware promised to look after the issue dec 2014 but failed to show up again.

      Delete
  8. I totally agree with you. I don't understand how is possible that Vmware didn't think the same things that you said. Vmware should have some of the best engineers but, in my opinion, lately they are doing many crap
    Would we speak about web client? Yes now they are moving in the right direction (html5) but I'm amazed that Vmware didn't do it since the beginning.
    But the vmware tools question is really incredible

    I think you should be hired by Vmware :)

    ReplyDelete
  9. Andreas, you're a genius!! I totally agree with you. This makes so much sense. Why install a new version of VM Ware tools unless the features help you out and normally when we update the VM ware tools, it introduces other problems. I wish VM Ware would get this right!! I agree with Marco, you should be working for VM Ware!! So many times, we do it because that's the way it's been and etc., and I have thought about this myself to but you hit the jackpot!! We need to tell VM Ware, to create a central repository just like Linux boxes access a central repository and to only update VM Ware tools if the features will help!

    ReplyDelete
    Replies
    1. Thank you for your comment! As of today VMware does not seem to have made progress on this topic. This is somewhat disappointing.

      If you are a VMware customer I can only recommend to reach out to them through every possible channel and ask them to improve the situation.

      Delete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!