How to build device drivers for ESXi 5.x

Since I published my ESXi-Customizer and ESXi-Customizer-PS tools to slipstream driver packages into an ESXi installation ISO I was asked dozens of times if I can provide a driver for the unsupported device xyz or whether I can give instructions on building such a driver. So far my answer was always No, because - although I'm quite familiar with using Linux - I do not have any of the Linux kernel hacking skills that I thought would be required to do this.

However, I was always aware that
  • ESXi device drivers are derived from Linux device drivers,
  • VMware has published the necessary source code to build the drivers that they ship with ESXi (because they are obliged to do this by the GPL license terms)
  • over at the vm-help.com forums there are some people posting who have already successfully built ESXi drivers for unsupported white box devices
Last weekend I stumbled over two posts in the vm-help.com forum by a user with the nickname trickstarter:
This is the first time that I found fairly detailed instructions on how to build ESXi 5.x device drivers, and it encouraged me to look at this myself.

Here are the required steps to set up a build environment for ESXi 5.x drivers and compile the already included drivers:

1. Download the ESXi 5.0 U2 Open Source code archive from MyVMware (select the file VMware-esx-open-source-5.0.0u2.oss.tgz). It looks like the OSS code of ESXi 5.1 was not yet published by VMware and is only available on request.

2. The archive includes a file BUILD.txt at the top level, and this file includes a description of the recommended build environment: It's a CentOS 5.3 x64 machine. This version is a bit aged now, but you can still download the installation ISOs from http://vault.centos.org/5.3/isos/x86_64/. There are seven CD ISOs, and I cannot tell which of them are really required, because I decided to download the complete DVD ISO that I found on another mirror server.

3. Install CentOS 5.3 in a VM using the downloaded installation ISO(s). The file BUILD.txt also includes a list of the software package sets that you need to select for installation.

4. Download the OSS archive into the VM and unpack it. Change to the sub directory vmkdrivers-gpl. There you will find another BUILD.txt file, and we now need to follow the instructions included in there. There is also the recommendation to run the following steps as the root user. This is against best practice - you should normally use root only if it is really necessary, but not for compiling code -, but in this case it makes life easier, because using root you can just follow the instructions and use the build scripts as-is without the need for any modifications.

5. The drivers' source code comes with its own gcc toolchain that you need to compile first. Enter the sub directories "glibc-2.3.2-95.44", "binutils-2.17.50.0.15-modcall" and "gcc-4.1.2-9" one after the other and execute the included BUILD.txt files with
   bash ./BUILD.txt
This will build the toolchain and install it in the directory /build.

6. Now unpack the drivers' source code:
   tar xzf vmkdrivers-gpl.tgz
and run the included build script:
   ./build-vmkdrivers.sh
It should build all existing drivers. You will see some gcc warnings, but no fatal errors.

At this stage you can now try to build an own driver by adding its Linux source code to this environment and creating a build script using the existing one as a template. This still requires some Linux hacking skills, and if you have never compiled software in Linux before then you will probably get lost quickly.

For more details and instructions I strongly recommend looking at trickstarter's posts that I mentioned above. He plans to make this a series of posts with even more parts, so join and stay tuned into the vm-help.com forums if you are seriously interested in this topic! This is also the right place to look for help if you get stuck.


13 comments:

  1. In answer to point 2:

    If you install CentOS 5.3 'Gnome Workstation' and customise it to add exactly what is in BUILD.txt you will need CDs 1-5 and 7, so unless your build machine lacks a DVD drive, downloading and installing from the DVD is far simpler.

    Using the 'Workstation' install gives you an X desktop which allows terminal windows with more than 80x25 characters as well as other freatures that make life much easier...

    ReplyDelete
  2. ESXi 5.5 Open Source code released



    These files are available on the VMware download portal.




    Open Source Licenses for VMware vSphere ESXi 5.5

    File size: 1.2M
    File type: txt
    Name: open_source_license_server5.5.txt
    Release Date: 2013-09-22

    MD5SUM: 320d88ccb5fb1083058353771eed0697
    SHA1SUM: e81a565a34b3464dc108a239450675a3637cd8f3




    Open Source Code for VMware vSphere ESXi 5.5

    File size: 710M
    File type: iso
    Name: VMware-ESX-5.5.0-ODP.iso
    Release Date: 2013-10-31

    The vSphere ESX Open-source Disclosure Package is available as the ISO9660 image file VMware-ESX-5.5.0-ODP.iso. If you would like to rebuild using the instructions included in this image, you will also need the associated Toolchain Open-source Disclosure Package VMware-TOOLCHAIN-5.5.0-ODP.iso.

    MD5SUM: 5adccda95ef82ff98e5b3bb3f23ff66f
    SHA1SUM: a56424ecb1da1595669a6d57391822bb30e437d9




    Supporting Toolchain packages for VMware vSphere ESXi 5.5

    File size: 3.1G
    File type: iso
    Name: VMware-TOOLCHAIN-5.5.0-ODP.iso
    Release Date: 2013-10-31

    MD5SUM: b73944bfc5ebe4489ac90ca448b9a3ec
    SHA1SUM: 9fc2f4ee484e5091f90c39fa2911e5acc60a6487

    ReplyDelete
    Replies
    1. Great! Thanks for sharing, that looks promising, esp. the supporting toolchain stuff ...

      Delete
    2. I got this download, but no vmkdrivers inside ^(
      Something wrong with it.

      Delete
    3. Yes, I can confirm that. The vmkdrivers code is missing in the ESXi 5.5 Open Source download.

      Use the 5.0 code instead. That will still work with 5.1 and 5.5.

      Delete
  3. I actually created a script to setup your build environment. You can get it from here: https://www.dropbox.com/s/vl4pcpmf1gg5t ... v_setup.sh . In order to make it work for apt-based linux distros, just update the script to use YUM instead of apt-get.

    Hope this helps.

    ReplyDelete
    Replies
    1. Thanks, Robert! But this dropbox URL doesn't work for me?!

      Delete
  4. Here's a better link to my ESXi 5.0+ build environment setup script -> http://pastebin.com/V46PcHdZ

    ReplyDelete
  5. In regards to a VIB for a driver, do you know how it should be packaged for non-device drivers? Character Drivers that are NOT attached to a PCI device? I package one and placed it in the /usr/lib/vmware/vmkmod directory, but its not automatically getting loaded. The VIB installs fine, but the driver just doesn't get loaded automatically.

    Thanks.

    ReplyDelete
    Replies
    1. Hi Robert,

      for PCI devices the .map file is responsible for mapping PCI device IDs to a driver name and triggering auto-load. I'm not sure if this mechanism will work for non-PCI devices and don't know how you would need to change the map file contents to achieve this.

      A safe way to automatically load the driver at boot time would be to create a service/init script in /etc/init.d. For details see the section "Automatic startup" in this post: http://www.v-front.de/2012/11/a-daemons-vib-part-2-building-software.html.

      - Andreas

      Delete
    2. That's exactly what I needed (and that also solved another future product we're working on that needs to start a service). Thanks man!

      Delete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!