VMware silently adds native USB 3.0 support to ESXi 5.5


The October 2014 patch of ESXi 5.5 already got a lot of attention, because it introduced additional Transparent Page Sharing (TPS) management capabilities to prepare for TPS being disabled by default in upcoming ESXi releases. And William Lam pointed out that this patch enables support for the Apple Mac Pro 6,1 ... but there is even more goodness in this patch!

When adding the associated Image Profiles to the VibMatrix I noticed that this bundle includes a new package named xhci-xhci. The related KB2087362 article only includes the standard disclaimer, but no information about what this really means: xHCI stands for Extensible Host Controller Interface, a USB standard that supports USB 3.0 (or SuperSpeed) controllers and devices.

That means with the latest ESXi 5.5 patch you are - for the first time - able to utilize USB Passthrough with USB 3.0 devices!

Here are the steps to get this working:

1. Install the ESXi 5.5 October 2014 patch

With an Internet connected standalone ESXi host you can use the well known esxcli method to patch the host directly from the VMware Online Depot:
Enable SSH access on your host, log in to it (e.g. using putty) and run the following commands:
# open firewall for outgoing http requests:
esxcli network firewall ruleset set -e true -r httpClient
# Update using the ESXi 5.5 p03 Imageprofile from the VMware Online depot
esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.5.0-20141004001-standard
# Reboot your host
reboot
If you do not have Internet access with your ESXi host then you need to download the Patch Offline Bundle from MyVMware for Offline patching.

Please note: This patch bundle also includes the ESXi 5.5 Update 2 that was released earlier, because ESXi patches are cumulative.

2. Load the xhci VMkernel module

After you have booted your newly patched ESXi host you can check with the shell command lsusb what USB devices were detected. You will probably not see any USB 3.0 devices then, because the necessary xhci driver module is not loaded automatically (at least not on my whitebox that uses an add-on Renesas chip based controller).

Go and load the module with the command
vmkload_mod xhci
in an ESXi shell. After that lsusb should also list your USB 3.0 device(s).

If you want to make use of your USB 3.0 devices on a regular basis then I recommend adding this command to the file /etc/rc.local.d/local.sh. Then it will always be automatically executed at boot time.

3. Configure USB passthrough

The method to use a host-connected USB device with a VM is called USB passthrough, and the steps to configure it are described in the vSphere docs. Here is a short summary:

In the hardware properties of the VM add a USB controller. If you want to use a USB 3.0 device at full speed then you need to add a controller of type xHCI. Otherwise the device will only work with USB 2.0 speed:


Please note: This will only work with Guest Operating Systems that include a generic USB 3.0/xHCI driver, and that is any Linux (with kernel version 2.6.38 and later), Microsoft Windows 8/8.1 and Server 2012 (R2), but not Windows 7 (resp. Server 2008 R2) or earlier versions of Windows!

The next step is to add the host-connected USB device to the VM. Add a USB device in its hardware properties and map it to the host device by picking the right one from the presented list:


You can now power on the VM and use the USB device with it!

Test results

I tested the performance of USB 3.0 passthrough on my whitebox ESXi system with a Windows Server 2012 R2 VM using an external 3TB hard disk and a 64GB thumb drive. I used the free ATTO Disk Benchmark tool to measure read and write throughput with various block sizes and got the following results:
  • CN Memory 3TB External hard disk (using an ASMedia USB to SATA converter): max. 90 MB/s write, max. 140 MB/s read
  • ADATA 64GB flash drive: max. 45 MB/s write, max. 90 MB/s read
I am aware that these values are far from the theoretically possible USB 3.0 transfer speeds, but I ran the same benchmarks on Windows 8 running natively on the same hardware and the results were only slightly better (+5%).

Summary and conclusion

The latest ESXi 5.5 patch enables you to pass host-connected USB 3.0 devices through to VMs (Linux and Windows 8+, 2012+) where you can achieve great near-native performance.

Besides from USB passthrough there is another way to make use of host-connected USB 3.0 devices. Unfortunately you cannot format a USB disk with VMFS to use it as a datastore, but William Lam has described how to use large USB keys to automate VM deployments using ESXi kickstart. This makes it possible to fully automate the provisioning of ESXi and a set of virtual machines as part of the initial deployment. Using a USB 3.0 device for this process will speed it up significantly!

Update (2014-11-03):
William Lam pointed out that there is a better way to enable the autoload on boot for the xhci module. Use this esxcli command once instead of editing the local boot script:
You never stop learning ;-) Thanks William! But ... there is a typo in William's tweet (it's xhci, not xchi), and after Paul Braren and I tested this it turned out that it doesn't really work as expected: You can load the xhci module using
esxcli system module load -m xhci
instead of vmkload_mod xhci, but the command
esxcli system module set -e true -m xhci
will not lead to the module being automatically loaded at boot time. So you still need my workaround with editing /etc/rc.local.d/local.sh.



This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.



37 comments:

  1. Andreas, this is awesome, and yes, it works!

    FYI, I did spot a small (tranposed character) typo in what William typed:
    esxcli system module set -e true -m xchi
    It should read:
    esxcli system module set -e true -m xhci
    but nevermind that, it doesn't seem to stick after reboot, so I wound up doing your item #2 above step instead, adding vmkload_mod xhci to /etc/rc.local.d/local.sh (I cheated a bit, using WinSCP)

    It's all now in a home lab video I took of me getting it working, over here:
    http://youtu.be/N6eA_cOAcvw

    Thank you for the fun project tonight and new possibilities!

    ReplyDelete
  2. Thanks for this great article! Something odd I ran into, however: the offline installer doesn't seem to have the updated xhci module. My build number was the same (2143827), but I got a "module not found" error when running vmkload_mod xhci. Now, I was still passing through my USB3 controller when I initially ran the update, but I ran again after disabling passthrough and rebooting, and still no go. I then tried the commands above to install, and the upgrade flew by, installing only the xhci module. Just throwing this out there in case anyone else runs into this weird issue!

    ReplyDelete
  3. Thank you very much for the informations! They are very useful and works at my HP Microserver Gen8 fabulous with the HP Image of Esxi 5.5 Update 2. I don't know if the mentioned update from October 2014 from the article was necessary but I have installed it without any harm on the Gen 8.

    ReplyDelete
  4. Doesn't work for me

    ~ # vmkload_mod xhci
    VMKMod_ComputeModPath(xhci) failed: Not found
    vmkload_mod: Can not load module xhci: not found


    I've tried 5.5 updated to 5.5U2 and then fresh install of 5.5U2

    there is no XHCI module

    ReplyDelete
    Replies
    1. Hi Alex,

      Update 2 does not include the xhci module. You need the October patch. Install it like described in my post.

      Andreas

      Delete
    2. I have same problems.

      I tried
      [esxcli software vib update -d /vmfs/volumes/datastore1/iso/ESXi550-201410001.zip]
      =>NG

      [esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.5.0-20141004001-standard
      ]
      => OK

      Thank you very much.

      Niriakot

      Delete
    3. Hi niriakot,

      if you add "-p ESXi-5.5.0-20141004001-standard" to the first command then this should also work.

      Andreas

      Delete
  5. What a shame, passthrough switches off snapshot ability...

    ReplyDelete
    Replies
    1. This is true for PCI passthrough (aka VMDirectPath), but not for USB passthrough!

      Delete
  6. Andreas,

    I am curious about your post and want to clarify a couple of things.
    Are you saying that the october patch enables USB passthrough with the onboard controllers on the motherboard or with a pci usb card ?

    I am also a bit curious about the screenshot above showing the logitech video device being detected. Is the webcam connected to the host ?

    Please let me know

    ReplyDelete
    Replies
    1. Hi Romaric,

      this should work with both onboard controllers and add-in cards.

      And yes, the Logitech USB device is a webcam. But it is a USB 2.0 device and with USB 2.0 devices passthrough has always been possible in ESXi 5.x (before the ESXi 5.5 October patch already).

      Andreas

      Delete
  7. I'm having issues getting this to work. I followed all of the directions, but when I add either a Seagate USB 3.0 external HDD, or a Newegg USB 3.0 hdd adapter, I get an error on the guest operating system (Win2k12 R2) for both devices, the USB Attached SCSI (UAS) Mass Storage Device. The error on both is "This device cannot start. (Code 10)". Has anyone else had this problem? ESXi detects both devices properly.

    ReplyDelete
    Replies
    1. I have the same issue as yours.
      The OS is Windows 8.1 Pro.
      The USB 3.0 pcie card is the Startech PEXUSB3S24 with NEC chip.
      I tried to install the driver that I downloaded from Startech.com and it's the same error.
      I followed the instruction to the teeth.

      Which card do you have?
      Anyone can help us with this issue?
      Much appreciated guys. Keep up the good work.

      *I'd open the question on the VMware communities site but unfortunately I just changed job so my current userid is still attached to the old email. The comm manager is working on getting me transferred to my current email.

      Delete
    2. Just want to add to my previous post.

      When I followed this TinkerTry's youtube video, it worked.
      https://www.youtube.com/watch?v=N6eA_cOAcvw

      Delete
  8. I have gone through this and many other posts in an attempt to get the xhci module installed. Whern attempting to follow this post I get output of InstallationError The transaction is not supported, pretty much all of the vib's in this update can not be live installed. I was able to get all updateable vib's to install using the vCLI command line tool, however the xhci-xhci vib along with a few others show "New" in the status, instead of installed. I have scoured the internet in search of how to get this module installed here but have hit a road block it seems. Suggestions ?

    ReplyDelete
    Replies
    1. Hi Anonymous,

      have you powered off all running VMs on the host that you want to patch?
      Try to put it into maintenance mode.

      Andreas

      Delete
  9. Yes its a single vm currently. Powerd off, device is in maint mode. All attemptes to enable xhci mod, retuen not found. when viewing installed vib's , xhci is not listed.

    ReplyDelete
    Replies
    1. Try to install only the xhci-package by running

      esxcli software vib install -n xhci-xhci -d -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

      (all on one line)

      Andreas

      Delete
  10. I followed the guide and everything seems to have installed properly. Loaded the xhci module and lsusb does confirm a 3.0 Hub but when logging into my Win 8.1 VM I'm still getting 2.0 data rates via CrystalDiskMark. I don't see what I'm doing wrong. esxcfg-module -q shows xhci loaded, esxcli system module list shows xhci loaded. Any suggestions?

    ReplyDelete
    Replies
    1. Hi Anonymous,

      is the device that you are measuring USB 3.0 capable, and is it really connected to the USB 3.0 controller?
      Have you added a USB 3.0 (xhci) controller to the VM? In the VM is it detected and installed as a USB 3.0 controller?

      Andreas

      Delete
    2. Yes the device is USB 3.0 capable. I used both a dock with a HDD and a usb stick both of which are USB 3.0. I made sure it's a SS port as well. The xhci module is installed and the controller has been added to the VM. I even went ahead and disabled all USB 2.0 ports to see if that forces USB 3.0 but that didn't help.

      One thing I did notice was that when watching one of the commenter's (Paul Braren) youtube videos his device manager shows a Generic Superspeed hub while I only have generic hubs listed. I forced drivers for Generic superspeed but that didn't work either.

      Delete
    3. Check the virtual USB controller of the VM. Is it of type "xHCI" like shown in the first picture in my post?

      Delete
    4. Yup, It's an xHCI controller. This does not seem to make sense.

      Delete
  11. Hello everyone,
    I'm seeing the same "issue" here. Updated to the latest ESXi 5.5-201412001 release. Enabled kernel module like mentioned above. Was then able to insert a xhci USB 3.0 controller and an USB3.0 device (Sharkoon SATA Quickport XT) to a windows server 2012 r2 guest machine
    The performance tested with ATTO Disk Benchmark was still on par with USB 2.0 (about 35 MB/s read and 32 MB/s write speed.)
    The only thing I can think of, but wasn't able to verify yet, I have connected a USB 2.0 USB thumb drive to an internal USB 3.0 header as the ESXi boot drive, maybe this USB 2.0 device forces the whole bus to be on USB 2.0 level !?
    I will try to figure this out in the next days.

    ReplyDelete
    Replies
    1. I have the same setup. ESXi launches off a USB 2.0 stick on a USB 3.0 Port. Maybe there is something to this? I'll have to try this as well.

      Delete
    2. That's actually what I had hoped for ;)
      Let's keep each other updated, thanks...

      Chris

      Delete
    3. Hi Chris,

      blog post comments are not the ideal place to have conversations like this ... although I would love having the outcome of your discussion linked here.

      How about opening a thread in the VMware Community Forums and link it here?

      If you stay here with the discussion then please keep the volume low, because I need to approve every comment.

      Thanks
      Andreas

      Delete
    4. Just a quick follow up:
      I managed to find the time to do the testing regarding the issues I described above.

      This is what I found out:
      The assumption that a USB 2.0 thumb drive connected to the USB 3.0 ports might be the culprit was wrong.

      With my Supermicro X10SLH-F, the only working configuration to get full USB 3.0 speed out of the chipset integrated Lynx Point xHCI Controller was to force it to be "enabled" in the BIOS/UEFI settings (neither "Auto" nor "Smart Auto" did work!), whatever "xHCI hand-off" settings was chosen.
      The problem with this is, that during bootup the boot stick gets disconnected and you're running without /bootbank and will run into troubles while trying to upgrade your esxi host etc.
      The same unfortunately also happens if the stick has been connected to one of the USB 2.0 ports, as it seems the eHCI controllers aren't available at all anymore with this setting (at least they don’t show up anymore with lsusb).

      As another approach, I even tried to put the Lynx Point xHCI controller into /etc/vmware/passthru.map with its vendor-id:device-id "8086:8c31" to finally use VT-d to pass through the whole controller. It was visible for passthrough configuration under “advanced settings”, but it didn't show any "properly" connected devices inside a Windows Server 2012 R2 VM afterwards.

      So yes, it basically works, but seems to only be reasonable with on- or off-board 3rd party controller chips (as long as the intel chipset integrated one isn’t properly controlled by the mainboard manufacturer BIOS/UEFI code).

      Chris

      Delete
    5. Good to know, many thanks for sharing, Chris!

      Delete
  12. Darn! It just won't work! I got ESXi6 .0, where xhci should work out of the box.
    But my installed Win8.1 client will never ever take the device for USB3.0!
    It's always the same like "You connected an USB3.0 device to a USB2.0 port blah blah" :(
    I always installesd xhci controller and tried several USB3.0 device on every super speed port available on my Poweredge T20, but this silly thing just won't do 3.0!

    Seems like superspeed support ist just a PR stunt..

    ReplyDelete
  13. Having the same issue as some posters here, installed the latest patches (now at build 2456374) on the free licence version of ESXi 5.5 U2 via the offline "esxcli software vib update -d /path-to-zip-file" but the xhci-xhci vib is always skipped for some reason?

    Thus I get the below result:

    ~ # vmkload_mod xhci
    VMKMod_ComputeModPath(xhci) failed: Not found
    vmkload_mod: Can not load module xhci: not found

    Tried with the "-p ESXi-5.5.0-20141004001-standard" option as offered above and that didn't install anything, perhaps my syntax was incorrect? Anyone have the complete command?

    Any assistance would be much appreciated. Purchased a PCI-e USB 3 card today (NEC uPD720200 chipset) and passed it through to a Windows 8.1 VM, works beautifully, now if I could only get the ports on my Supermicro X10SLL-F to do the same :/

    Thanks!
    Craig.

    ReplyDelete
    Replies
    1. Hi Craigm

      please try to explicitly install this package like I showed in this comment.

      Andreas

      Delete
  14. Has anyone successfully found an addon usb 3.0 card that works directly in vmware?
    They don't even have a section on the hcl for something like that.
    I want to add usb 3.0 to my r710, didn't know if anyone reading this has had any luck

    ReplyDelete
    Replies
    1. The one that I used (and wokred well) was a cheap Renesas chip based one.

      Delete
  15. i have a weird issue ... with my R730xd when i enable usb 3 in the bios and reboot , even i add the line to /etc/rc.local.d/local.sh i have an error no core dump target as been configure and have to manualy enable usb3 with the command : vmkload_mod xhci this enable usb 3 but not fix core dump issue ... anyone have seen this issue before ?
    tks!

    ReplyDelete
    Replies
    1. i know its late about this issue but i was having the exact same issue ! with this server
      so the problem is the usb3 controler running on the same bus of your sd card where esxi is install and this create a conflic :( si you have 2 choices:
      1- you disable usb 3 and install a usb 3 card
      2- you add hd in your server and install esxi on it and disable sd card controler

      i do the first option and its work perfectly

      Dell take 3 weeks to understand what append ! and on my side it corrupt 6 of my 10 vm's
      stop buying dell products it's my advice

      Delete
  16. Thanks a lot, it is helpful!!

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!