pfSense 2.2 was released - How to install VMware Tools (sort of)


Update 2016-05-04: If you are using pfSense 2.3 (or newer) then please do not follow this guide, or it will break your system! Use the pfSense Package Manager to install the open-vm-tools package instead!

I'm a big fan of pfSense, an Open Source firewall and router appliance, that I use in my hosted lab. Recently version 2.2 of pfSense was released with a lot of bugfixes and new features. Please review the announcement blog post to find out what's new.

In my lab I upgraded one of the "not so important" pfSense VMs to version 2.2 and then tried to get VMware Tools installed and running again following my own guide that I wrote a while back for pfSense 2.0 and 2.1. But, well, things have changed a lot with pfSense 2.2, because it is now based on the latest FreeBSD version 10.1. Here is what I found out and what I ended up with.

First an important heads-up: FreeBSD 10.1 as a Guest OS is supported since VMware ESXi 5.5 Update 2, but not earlier! In any case you need the VMware Tools of 5.5 Update 2, earlier versions will fail to install on FreeBSD 10.1 / pfSense 2.2. If you run an earlier version of ESXi and do not want to upgrade then you can download the Tools package of 5.5 U2 from the VMware Online Depot (right-click, save as...), extract the freebsd.iso file from it using 7zip like described in this post, and connect that ISO to the VM in step 4 below.

The good news is that FreeBSD 10.1 has a built-in driver for vmxnet3 network adapters, so you do not really need to install VMware Tools just to get this driver. Nevertheless I wanted to try out if the guide that I wrote for pfSense 2.0 and 2.1 still works. Well, the process is still the same, but commands have slightly changed in FreeBSD 10.1:

Warning: Before trying the following steps please create a snapshot of your pfSense VM so that you can quickly roll back in case things go wrong - for me it went wrong :-(, and I love snapshots :-)
  1. To install the required packages for VMware Tools run
      pkg bootstrap
      pkg install perl5
      pkg install compat6x-`uname -m`

    ("uname -m" will output either "i386" or "amd64" depending on whether you installed the 32- or 64-bit version of pfSense. Answer all pkg prompts with "yes".)
  2. To update shared libraries just run
      /etc/rc.d/ldconfig start
    (No need to edit /etc/rc anymore)
  3. Create a symlink for the perl executable, because the VMware Tools install script expects it in /usr/bin:
      ln -s /usr/local/bin/perl /usr/bin/perl
  4. Attach the VMware Tools ISO to the VM's CD drive by choosing "Install/Upgrade VMware Tools" in the VM/guest menu of the vSphere Client. In the VM run these commands then:
      mkdir -p /mnt/cdrom
      mount_cd9660 /dev/cd0 /mnt/cdrom
      cd /tmp
      tar xvzf /mnt/cdrom/vmware-freebsd-tools.tar.gz
      cd vmware-tools-distrib
      ./vmware-install.pl -d --clobber-kernel-modules=vmxnet3,pvscsi,vmmemctl

    (just like in the original guide, only the CD device name has changed from /dev/acd0 to /dev/cd0)
  5. Reboot the VM.
Please note that with the FreeBSD builtin vmxnet3 driver the network interface names are vmx0, vmx1 etc. With the VMware supplied driver these names will change to vmx3f0, vmx3f1 etc., so after the reboot you need to reassign the interfaces at the console of the pfSense VM.

Okay, I did all this and ran into trouble: With the VMware supplied vmxnet3 driver I had very strange networking issues. I was able to ping the network interfaces of the pfSense VM from the outside, I was also able to ping out from inside the VM at its console, but I was not able to log in via ssh and the pfSense web interface was also not reachable.

Maybe that's only me, and it works fine for other people ... I decided to not tinker with this for too long, because there is an alternative to the official VMware Tools! So I reverted to the snapshot that I created at the beginning (you have been warned!) and installed the Open-VM-Tools package using the pfSense package manager in its web interface.

The Open-VM-Tools would suit my basic needs. Now that the vmxnet3 driver is already builtin I only wanted the ability to shutdown or reboot the pfSense VM from within the vSphere Client, but ... guess what was not working after installing open-vm-tools? :


With the out-of-the-box install of open-vm-tools you are not able to shutdown or reboot a pfSense 2.2 VM using the vSphere Client ... not on the first try, but interestingly the second try would work (as the error message above even suggests). Obviously there is something wrong with the VMware Tools scripts inside the VM. These scripts are launched by the VMware Tools inside the VM prior to a shutdown, boot-up, suspend or resume event, and they can be customized. I decided to just disable them and found instructions for this in the official VMware docs:

In the VM run the following commands;

  vmware-toolbox-cmd script power disable
  vmware-toolbox-cmd script resume disable
  vmware-toolbox-cmd script suspend disable
  vmware-toolbox-cmd script shutdown disable


This will create a file tools.conf in the directory /usr/pbi/open-vm-tools-amd64/local/share/vmware-tools and add the following lines to it:

  [powerops]
  poweron-script=
  resume-script=
  suspend-script=
  poweroff-script=


I guess you can also just create and edit the file manually, but I haven't tried that. I restarted the VMware Tools daemon with

  /usr/local/etc/rc.d/vmware-guestd.sh restart

et voilĂ : a shutdown or reboot from the vSphere Client worked fine - on every try!

While I was looking at the files of the open-vm-tools package I noticed a second glitch: The other boot script /usr/local/etc/rc.d/vmware-kmod.sh is supposed to load the VMware Tools kernel modules (like vmmemctl for the memory ballooning feature), but it looks for these in a directory that just doesn't exist. To fix that I created a symbolic link to the correct location:

  cd /usr/local/lib
  ln -s /usr/pbi/open-vm-tools-amd64/local/lib/vmware-tools vmware-tools

and after a restart of the boot script the modules were correctly loaded:

  /usr/local/etc/rc.d/vmware-kmod.sh restart

You can check what kernel modules are loaded with the command kldstat.

As always I hope that you find these instructions useful. If you have any questions or remarks, or if you know what's wrong with the VMware supplied vmxnet3 driver then please let me know in the comments. Thanks!


This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.



16 comments:

  1. Very useful information Andreas. Thanks for the advices.

    ReplyDelete
  2. FYI, if you are running 2.2+, you can open /boot/loader.conf with you favorite text editor, and delete the lines that the VMware installer created. This will default the pfSense install to the pf provided driver.

    ReplyDelete
    Replies
    1. This is a golden tip. I used 2.2.3 and installed the official VMWare tools as stated above. (Perl5 installation gave the same POST-INSTALL error).
      Had the same issues as stated, however after commenting out the two rules in /boot/loader.conf, the network was functional. The shutdown problem was still there.

      Delete
  3. I had the same weird network problem - could ping fine, but no other packets make it in or out.
    VMWare list Freebsd 10.1 as compatible with vmxnet3, so could be a pfsense specific problem...

    ReplyDelete
  4. I can confirm that a standard installation of FreeBSD 10.1 (upgraded from 9.x) is compatible with vmxnet3. However, as did Andreas, I had strange networking issues with pfsense and vmxnet3 that forced me to revert to the out-of-box vmx driver.

    On top of this, I couldn't get perl5 or perl5.20 installed without errors either (it says the POST-INSTALL script fails). I've run into this on both an upgraded pfsense 2.1 and a fresh VM with 2.2, so not sure why it's just me unless it's the amd64 arch. Unfortunately the first machine I tried to upgrade (before finding this page) I stupidly forgot to take a snapshot. Please heed the above advice and take the snapshot!

    ReplyDelete
    Replies
    1. Loots like the cause of the strange networking issue is tx/rx checksumming:
      https://forum.pfsense.org/index.php?topic=87675.msg506040#msg506040

      Delete
  5. So looks like with ESXi 6 vmx3 and pfsense 2.2 is a complete nogo. I tried to use the integrated driver that came with FreeBSD, nogo ... traffic wont go thru. Tried installing the driver from the VMware tools cd ... same thing, traffic wont go thru.

    ReplyDelete
    Replies
    1. Interesting ... I have not yet tried that.

      BTW I use e1000e now with my pfSense VMs and that gives me very good performance (similar to vmxnet3).

      Delete
  6. Hi Andreas,

    I used the vmware-toolbox-cmd script xx disable commands as you mentioned and it worked well. However we are on pfSense 2.2.2 now and I had to rerun these commands on every update. I came to realize that there's an easier solution.

    With the guest OS shut down, in vSphere Client go to Edit Settings -> Options -> VMware Tools and uncheck the options to run scripts on the various power states.
    Now I can just shut down the VM without having to disable the scripts from within the guest.
    Not sure why you would want to run additional scripts in a firewall appliance anyways.

    cheers!

    ReplyDelete
    Replies
    1. Thanks Wil,
      this sounds like another good solution!

      Delete
  7. HI,

    Has anyone figured out the wired network issues with vmxnet3, I can ping but nothing else. and the card now shows 10Gbase-T which is what I want to see.

    ReplyDelete
    Replies
    1. Hi Anonymous,

      no, I haven't pursued this. All I know is that this problem does not happen with a pure FreeBSD 10.1 VM, so it is pfSense specific.

      Andreas

      Delete
  8. Thanks for this great little how-to guide Andreas and it works a treat for me on ESXi 6.0, pfsense 2.2.6, and using e1000 cards !

    I can now use the VMWare Startup/Shutdown configuration to start up and shut down all my servers, including my firewalls on my dev servers which is brilliant !

    Thanks,


    Nick

    ReplyDelete
  9. Note you shouldn't do this on pfSense 2.3. Use the built-in open-vm-tools package. Following these instructions reportedly will break the system completely.

    Andreas: could you please add a note for 2.3 users that they should not follow these instructions?

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!