Announcing the V-Front Online Depot for ESXi software


A while ago Google announced that their Google Code service will no longer allow downloads to be hosted. Existing projects will no longer be able to create new downloads starting at January 15th 2014. So far I have made all my tools and ESXi software packages available on my Google Code page, so it's time to think about other ways to distribute my stuff.

What is the preferred way to distribute software for VMware ESXi? How does VMware do this? They use a so-called Online Depot! VSphere Update Manager makes use of this, and you can also access the VMware Online Depot using PowerCLI ImageBuilder, e.g. to build a customized ESXi installation ISO. Basically an Online Depot is just a web site (accessible via the http[s] protocol) that has a well defined directory structure and some XML configuration files to glue it all together.

And you can build your own Online Depot! HP (and other hardware vendors) did this to distribute their CIM providers and tools for ESXi. So did I, and this post is about how to use the brand new V-Front Online Depot for ESXi software.

There are (at least) three different ways to use an Online depot: Via esxcli, PowerCLI ImageBuilder and Update Manager. Let's look at how to use the V-Front Depot with each of these methods:


Installing and updating software using esxcli

With the following esxcli commands you can query the depot and install/update software from it. By default the ESXi firewall will block outgoing http[s] traffic, so first you will need to open it for this kind of traffic, either through the vSphere Client (see Host Configuration / Security Profile / Firewall properties... / check the rule [x] httpClient) or via the first esxcli command listed here:
# Open firewall for outgoing http/s requests:
esxcli network firewall ruleset set -e true -r httpClient
#
# List packages available in the V-Front depot:
esxcli software sources vib list -d http://vibsdepot.v-front.de
The Status column of each package will show if it is already installed (= Installed) on the local system or not (= New), or if an older version is already installed and can be updated (= Update).
# Get information about a package in the depot
esxcli software sources vib get -d http://vibsdepot.v-front.de -n esxcli-shell
# Install a community supported package from the depot:
# - Lower the system's acceptance level to match the package
esxcli software acceptance set --level=CommunitySupported
# - Install the package
esxcli software vib install -d http://vibsdepot.v-front.de -n fw-ntpd
If you omit the -n package-name then all packages that are available in the depot will be installed. Most often this is not what you want.

In a depot you can have multiple versions of the same package. By specifying only the name you will automatically install the most recent version. If you want to install a specific (older) version of the package then you must use the name:version notation instead (e.g. -n sata-xahci:1.8-1).
# Install an unsigned package from the depot:
esxcli software vib install -d http://vibsdepot.v-front.de -n cpu-microcode --no-sig-check
Since I do not have an official VMware partner status I am not able to properly sign my ESXi software packages, so all my packages are unsigned. This is no problem with packages that use the CommunitySupported acceptance level, but for all higher acceptance level a valid electronic signature is required. Luckily there are ways to work around this, for esxcli you need to use the parameter --no-sig-check.
# Update installed packages from the depot:
esxcli software vib update -d http://vibsdepot.v-front.de
This command will update all packages that you have formerly installed from the depot to their most recent versions, but will not install any new packages. Like with install you can also limit the command to a specific package using -n package-name, and it might also be necessary to change the acceptance level before or use the --no-sig-check parameter to skip the signature verification like shown before.


Using ImageBuilder to include software into customized installation ISOs

I wrote about PowerCLI ImageBuilder in detail before and explained how to use it with the VMware Online Depot to create an ESXi installation ISO (see my Deep Dive part 1, part 2 and part 3). So here is just an example script that will build an ESXi installation ISO that includes my sata-xahci package to enable the ESXi installation with unsupported SATA controllers:
# Connect the VMware Online depot
Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

# Connect the V-Front Online depot
Add-EsxSoftwareDepot http://vibsdepot.v-front.de

# Clone the ESXi 5.5 GA profile into a custom profile
$CloneIP = Get-EsxImageProfile ESXi-5.5.0-1331820-standard
$MyProfile = New-EsxImageProfile -CloneProfile $CloneIP -Vendor $CloneIP.Vendor -Name (($CloneIP.Name) + "-xahci") -Description $CloneIP.Description -AcceptanceLevel "CommunitySupported"

# Add the latest version of the sata-xahci package from the V-Front depot
Add-EsxSoftwarePackage -SoftwarePackage sata-xahci -ImageProfile $MyProfile

# Export the custom profile into an ISO file
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToISO -FilePath c:\temp\ESXi-5.5.0-1331820-standard-xahci.iso

My ESXi-Customizer-PS script is a wrapper script for ImageBuilder that simplifies and automates its usage. A future version of it will include new options that will allow it to make use of the V-Front Online depot.


Add the V-Front Depot to Update Manager

Here is how to add the V-Front Online depot to your Update Manager configuration. In the legacy vSphere Client navigate to Update Manager / Configuration / Download Settings:


1. Click on the Add Download Source... link in the top right corner
2. Enter the Source URL http://vibsdepot.v-front.de/index.xml (you can also use https://... if this is required in your environment for whatever reason).
3. Click on Validate URL to check if Update Manager can reach this URL (this is optional).
4. Click on OK to add the depot.

After that click on Apply and Download Now to immediately synchronize the packages from the V-Front depot. They will then appear in the Patch Repository and can be included in baselines for installing them on the connected ESXi hosts:


Unfortunately Update Manager won't show you the acceptance level of a package, so you need to find this out by other means. It is important to find this out though, because - like with the other deployment methods - you need to change the acceptance level of a host to Community Supported before you are able to install a Community Supported package on it. You can do this in the vSphere Client (see Host Configuration / Security Profile / Host Image Profile Acceptance Level / Edit...):

Please note: You will not be able to install unsigned packages of another acceptance level through Update Manager, because there is no way to suppress the signature check here (at least I do not know one)!


What is available in the V-Front Online depot?

Currently all the ESXi software packages that I have created so far are available in the depot:
  • fw-ntpd - A firewall rule to enable incoming NTP client traffic
  • esxcli-shell - An esxcli plugin that allows to run any shell command via esxcli
  • ProFTPD - The famous FTP server for ESXi 5.x
  • cpu-microcode - CPU microcode updates from Intel and AMD
  • sata-xahci - To make unsupported SATA AHCI controllers to work with ESXi 5.x
Of course I will also add new packages that I create and newer versions of them here.

If you open the URL http://vibsdepot.v-front.de in your favorite browser you will find a Wiki that I will soon populate with information regarding the V-Front Online depot, with direct download links to the VIB files and Offline Bundles (for offline usage). Here I will also add information about and download links for the other (Windows based) tools that I have created (ESXi-Customizer, the ESXi-Customizer-PS script and the ESXi5 Community Packaging Tools).
Right now there is not much to see there, just the default installation of the MediaWiki software.


Come contribute!

If you have created an ESXi software package (e.g. a hardware driver for some unsupported NIC) yourself then you are welcome to distribute it through the V-Front Online Depot. This way you will get a central standardized distribution point for your package - for free! Just get in contact with me if you are interested!


This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.




16 comments:

  1. Thanks for the many great articles. Very very helpful.

    I am trying to create my own "ISO" base on your scripts and it is working well except for the microcode update.

    I am trying this line:

    $MyProfile.AcceptanceLevel = "CommunitySupported"
    Add-EsxSoftwarePackage -SoftwarePackage cpu-microcode -ImageProfile $MyProfile

    I get this error:
    Export-EsxImageProfile : The VIB VFrontDe_bootbank_cpu-microcode_1.4.0-1 does not contain a signature.

    What is strange is that your sata-xahci "package" works fine, just the CPU microcode one does not.

    ReplyDelete
    Replies
    1. The cpu-microcode is not of acceptance level "CommunitySupported". You need to use "-force" with Add-EsxSoftware-Package and "-NoSignatureCheck" with Export-EsxImageProfile.

      Details are in this post: http://www.v-front.de/2012/11/update-esxi5-community-packaging-tools.html

      BTW, Instead of writing your own scripts you can also use my ESXi-Customizer-PS script. With that it's only one line:

      ESXi-Customizer-PS-v2.2.ps1 -vft -load sata-xahci,cpu-microcode

      - Andreas

      Delete
    2. Correction: The usage with ESXi-Customizer-PS needs to be:

      ESXi-Customizer-PS-v2.2.ps1 -vft -load sata-xahci,cpu-microcode -nsc

      Delete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. Hi Afro,

      I have added comments to the document that you shared. I will now remove your comment here, because I'm not sure how your doc is protected.

      Andreas

      Delete
  3. Thanks for amazing article, wonderful blog with a lot of information

    ReplyDelete
  4. I'm a bit lost on 3 ESXi hosts i have. On the 4th one, it all works fine. But on the 3 others, when i do the command to list the available packages, the command stucks and nothing is done.

    I guessed it was a firewall thing, but the firewall rule is well set :
    [root@xserve1-esxi:~] esxcli network firewall ruleset rule list |grep http
    httpClient Outbound TCP Dst 80 80
    httpClient Outbound TCP Dst 443 443
    [root@xserve1-esxi:~] esxcli network firewall ruleset list |grep http
    httpClient true

    Any idea where i could look at ?

    ReplyDelete
    Replies
    1. Yes !!! It totally worked ! Thanks a lot.
      And yes the 4th host is not on same network and is on an IPv6 compatible network when the 3 who didn't work were on a non IPv6 compatible one...

      Thanks A LOT !

      Delete
  5. Andreas,
    Much to my dismay I have not reviewed your blog before and I really wish I had. You have a lot of great articles that will help with the work I'm doing. Thank you.

    I've been looking for documentation on how to create my own online VIB depot. I keep turning up empty. Do you have a posting on how to do it or point me to a link on creating them?

    Matthew

    ReplyDelete
    Replies
    1. Hi Matthew,

      thank you for your message!

      Well I *planned* to write such a post, but then backed away from it, because it's quite a big effort, but would probably not help a lot of people (you are the first to show interest). I also do not know of another public source explaining that.

      The way I learnt it is just by reverse-engineering what I found at known public Online Depots like VMware's one (starting at https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml) and HP's one (starting at http://vibsdepot.hp.com/index.xml).
      Follow the references in there, download metadata zip files and look into them ... Try to understand how this is all linked and mimic that with your own web server.

      Sorry, I cannot help anymore right now.

      Andreas

      Delete
    2. Andreas,

      Thank you. I was afraid that was the answer. I've started digging into HP's depot and keep hoping I'll find the magic document. I'll look at yours, VMware's, and HP's.

      I look forward to returning to your site and learning more.

      Matthew

      Delete
    3. I've seen that document before. It does provide an outline but doesn't really get into the depth of building the environment that I'm hoping for. Leaves a lot to be figured out in my opinion. And I haven't found the articles mentioned in it that really seem to have the answers.

      Delete
  6. Hello,
    I'm not sure if you were aware but your online depot is not functional for downloads. When I add
    http://vibsdepot.v-front.de/index.xml
    I can't get any patches to download. If I look at VUM log I see errors like:

    [httpDownload, 420] Download http://vibsdepot.v-front.de/depot/vft/cpu-microcode-7.0.0-1.x86_64.vib failed: Error retrieving document 'http://vibsdepot.v-front.de/depot/vft/cpu-microcode-7.0.0-1.x86_64.vib'. Status code: 404

    and when I try manually finding that file, it doesn't exist there.

    But the WIKI has a correct link populated I can download; http://vibsdepot.v-front.de/depot/vft/cpu-microcode-7.0.0/cpu-microcode-7.0.0-1.x86_64.vib

    it looks like the repository isn't correctly pointing to vib locations.

    ReplyDelete
    Replies
    1. Hi Anonymous,

      thanks for the heads-up. It looks like this also happens with other packages in the depot. Looks like there have been changes in recent versions of VUM. I will look into this.

      Andreas

      Delete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!