ImageBuilder Deep Dive, Part 3: The Power of the Shell

Let's start this third and last part of my ImageBuilder series with a short retrospective summary: In part one we introduced an easily adaptable script for building a customized ESXi 5.0 installation ISO. In the second part we looked closer at the basic cmdlets and learnt some new advanced ones.

In this final post we will walk through an improved and more universal version of the first script. It is more about Powershell in general than about ImageBuilder, so it will definitely help if you are already somewhat familiar with Powershell. But even if this is not the case you can still benefit by just using the improved script as it is. It took me some time to write it, because - according to Powershell - I was an absolute beginner when starting. I learnt a lot while developing it, but it is probably still not perfect ...

You can just download a copy of the script and use it. For the curious here is the listing:
#
# ESXi-Customizer-PS.ps1 - a script to build a customized ESXi installation ISO using ImageBuilder
# Version: 1.0
# Author: Andreas Peetz (ESXi-Customizer@v-front.de)
#

param(
    [string]$obDir = $(Split-Path $MyInvocation.MyCommand.Path),
    [string]$isoDir = $(Split-Path $MyInvocation.MyCommand.Path),
    [switch]$hp = $false,
    [switch]$help = $false
)

$AccLevel = @{"VMwareCertified" = 1; "VMwareAccepted" = 2; "PartnerSupported" = 3; "CommunitySupported" = 4}

function AddVIB2Profile($vib) {
    if ($AccLevel[$vib.AcceptanceLevel.ToString()] -gt $AccLevel[$MyProfile.AcceptanceLevel.ToString()]) {
        write-host -nonewline (" [New AcceptanceLevel: " + $vib.AcceptanceLevel + "]")
        $MyProfile.AcceptanceLevel = $vib.AcceptanceLevel
    }
    Add-EsxSoftwarePackage -SoftwarePackage $vib -ImageProfile $MyProfile | Out-Null 
    if ($?) { " [OK]" } else { " [FAILED]" }
}

# Write info and help if requested
write-host "`nScript to build a customized ESXi installation ISO using ImageBuilder"
if ($help) {
    write-host "Optional parameters:"
    write-host "   -help         : display this help"
    write-host "   -obDir <dir>  : directory of Offline bundles to add (default = script directory)"
    write-host "   -isoDir <dir> : directory to store the customized ISO (default = script directory)"
    write-host "   -hp           : add packages from the HP VIBs Depot (default = no)`n"
    exit
} else {
    write-host "(Call with -help for instructions)"
}

# Load the ImageBuilder Snapin (if not already loaded)
if (!(Get-PSSnapin -name VMware.ImageBuilder -ErrorAction:SilentlyContinue)) {
    if (!(Add-PSSnapin -PassThru VMware.ImageBuilder)) {
        # Error out if loading fails
        write-host "FATAL ERROR: Cannot load the ImageBuilder Snapin. Is the PowerCLI installed?"
        exit
    }
}

# Add the VMware ESXi base depot
write-host -nonewline "`nConnecting the VMware ESXi base depot ..."
if ($baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml) {
    write-host " [OK]"
} else {
    write-host "`n   FATAL ERROR: Cannot add VMware base Online depot. Please check your internet connectivity and/or proxy settings!"
    exit
} 

# Get the newest ImageProfile from the base depot
$LatestIP = (Get-EsxImageProfile "ESXi-5.0.0-20*-standard" | Sort-Object -Descending Name)[0]
write-host ("Using latest ImageProfile " + $LatestIP.Name)
write-host ("(dated " + $LatestIP.CreationTime + ", AcceptanceLevel: " + $LatestIP.AcceptanceLevel + ",")
write-host ($LatestIP.Description + ")")

# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile $LatestIP -Name ($LatestIP.Name + "-customized") -Description ($LatestIP.Description + " + customizations")

if ($hp) {
    # Add the HP VIBs depot
    write-host -nonewline "`nConnecting the HP Online Depot ..."
    if ($hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com) {
        write-host " [OK]"
    } else {
        write-host "`n   FATAL ERROR: Cannot add HP Online depot. Please check your internet connectivity and/or proxy settings!"
        exit
    }
    $hpDepot.Channels[0] | Get-EsxSoftwarePackage | foreach {
        write-host -nonewline ("   Add VIB " + $_.Name + $_.Version )
        AddVIB2Profile $_
    }
}

# Loop over Offline bundles
write-host "`nAdding Offline bundles from" $obDir ...
foreach ($oBundle in Get-Item $obDir\*.zip) {
    write-host -nonewline "   Adding" $oBundle ...
    if ($ob = Add-EsxSoftwareDepot $oBundle) {
        write-host " [OK]"
        $ob.Channels[0] | Get-EsxSoftwarePackage | foreach {
            write-host -nonewline "      Add VIB" $_.Name $_.Version
            AddVIB2Profile $_
        }
    } else {
        write-host " [FAILED]`n      Probably not a valid Offline bundle, ignoring."
    }
}

# Export the Imageprofile into an installation ISO file
$isoFile = $isoDir + "\" + $MyProfile.Name + ".iso"
write-host -nonewline ("`nExporting to ISO file " + $isoFile + ". Please be patient ...")
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath $isoFile -Force
if ($?) { " [OK]" } else { " [FAILED]" }

write-host "`nAll done."
Let's look at the script line by line:

Line 7 - 12: The script accepts command line parameters that are defined inside a param() statement. One of the possible parameters is the switch -help. If specified the script will print a help text (explaining the rest of the parameters) and exit (see line 25ff.). Another optional switch is -hp which will add the HP Online VIBs depot (s. line 65ff.). Using -obDir you can specify a directory where the script will look for Offline bundle zip files to be added. And with -isoDir you can specify the directory to store the created ISO file. The default for both directories is the path where the script itself is stored (which is determined by $(Split-Path $MyInvocation.MyCommand.Path)).

Line 14: One of the advanced Powershell features is using hash tables like the one that is created here. It contains the string representations of the four possible acceptance levels that a package or an image profile can have. They are assigned a ranking value (1 to 4) that is used to determine what the most restrictive acceptance level is ("VMwareCertified" = 1) as opposed to the least restrictive ("CommunitySupported" = 4).

Line 16 - 23: In these lines we define a Powershell function named AddVIB2Profile that adds a VIB package ($vib) that is passed as an argument to the custom image profile. As a first step it compares the acceptance level of the software package with that of the image profile (line 17) to determine if the latter needs to be lowered in order to accept the package (line 18 - 19). This is where the above mentioned hash table is used. Then it adds the package using the Add-EsxSoftwarePackage cmdlet. As I want the output of the script to be pretty and not garbled by useless information I pipe the output of the cmdlet into out-null. That means that it is just discarded. In line 22 I use the special Powershell variable $? to test whether the previous command was successful or not. This is a basic and the easiest way to add error handling to a Powershell script, and I will use that a lot here.

Line 25 - 36: This is actually the start of the main script. It prints a title line (line 26), then it will either display a help text and exit the script if -help was specified (line 27 - 33), or it will display only a short line explaining that -help can be used for instructions (line 35) and continue. BTW the special characters `n represent a new line when used in Powershell strings.

Line 39 to 45: Here we will check whether the required snapin VMware.ImageBuilder was already loaded (e.g. because you used the PowerCLI desktop shortcut to start your Powershell session). If not then we try to load it with Add-PSSnapin (line 40). If this fails (e.g. because PowerCLI is not installed on the computer) then the script errors out, because it depends on the ImageBuilder snapin being available and loaded.

Line 47 to 54: Now we add the VMware ESXi Online depot to the ImageBuilder session (also with some error handling). Nothing spectacular, we've seen this before.

Line 57 to 60: In line 57 we grab all standard image profiles from the VMware depot, create a sorted array of them and just pick the first one (with index [0]). This is the newest one, because we sorted the array in descending order. In the following lines we output some information about this image profile that is stored in its attributes.

Line 63: Here we create our custom image profile by cloning the latest one from the VMware depot. Its name and description are derived from the original VMware profile's properties.

Line 65 to 78: If the command line switch -hp was specified then we connect the HP Online VIBs depot to the ImageBuilder session (line 68 to 73) and add all included software packages using the AddVIB2Profile function that we defined at the beginning of the script.

Line 81 to 93: The foreach loop starting at line 82 is another essential part of the script code: Here we loop over all zip files that are located in the obDir directory and try to add them as Offline bundles to the ImageBuilder session (line 84). If it succeeds because a zip file is a valid Offline bundle (and not some other sort of file archive) then all software packages that are included in the bundle will be added to the custom image profile, again using our self-defined AddVIB2Profile function (line 88).

Line 96 to 99: Finally we export the custom image profile into an installation ISO. Its file name is derived from the image profile's name.

How do you run this Powershell script (and others)?

Open a plain Powershell window from the Start menu, or use the PowerCLI desktop shortcut to start an already customized Powershell session.

By default Powershell will only execute scripts that were electronically signed for security reasons. This script (and most other useful scripts that you will find in other places) is not signed, so you will need to relax the security check by using the Set-ExecutionPolicy cmdlet like shown in the following screenshot. You can do this for the current user only by specifying -Scope CurrentUser. This will also work if you do not have administrative permissions on the computer. If you have administrative permissions then you can leave the -Scope parameter and make the setting system global. Setting the execution policy needs to be done only once - it will be saved to the registry and reused for future Powershell sessions.

You can just call the script by its name (adding parameters as required). Since the script is from an "untrusted" source you still need to confirm its execution every time you start it:

ESXi-Customizer-PS help screen
Finally this is an example output from a complete script run:

ESXi-Customizer-PS example run
The script that I provide here is fully functional and flexible enough to meet most requirements. However, it is certainly not perfect. E.g. it could have better error handling. If you have any comments or ideas on how to improve it (or if you run into errors when trying it) then please add a comment to this post!

Update (2012-06-07): In the meantime I updated this script to fix errors. Please watch out for the latest version using this download link!


How to disable Storage I/O Control for an unavailable datastore

If you run your VMs on FC based storage then you occasionally have the necessity to unmap a storage LUN from one or more ESX(i) hosts (e.g. when retiring a storage array or re-organizing your storage layout). It is important to do this in the right way using the procedures that are documented in
The procedure is very complex for vSphere 4.1, but fortunately is has become much easier with vSphere 5.0.

What happens if you fail to follow these procedures and just unpresent the LUN on the storage array, so that the hosts can not access it anymore? The ESX(i) hosts will detect an APD (all paths down) condition in this case, and - particularly ESX(i) 4.1 hosts - can become very unhappy about this (see KB1016626 and KB1030980). Again, ESXi 5.0 hosts are much more resilient to APD conditions: they will eventually turn them into PDL (= permanent device loss, s. KB2004684) conditions and will completely recover from the LUN loss after rescanning the HBAs ... unless you have Storage I/O control (SIOC) enabled on the lost datastores ... and this is what happened to us today :-( The vmkernel.log files were flooded with the following messages, because SIOC was trying to access the lost datastore:

Permanent Device Loss (PDL) with SIOC enabled
Now the problem is: You cannot just disable SIOC on a lost datastore using the vSphere client - you should have done this before unmapping the storage LUN!

One way to recover from this situation is to reboot any of the affected hosts. However, I really wanted to save the time to put all the hosts in maintenance mode and reboot them one after the other. So I looked for a way to forcibly unmount the datastore directly on the host via esxcli or similar. All the ways that are documented in the VMware KB articles and docs did not work in my case, but I finally stumbled over this wonderful blog post by William Lam:
Does SIOC actually require Enterprise Plus and vCenter?
It is a bit old and refers to ESX(i) 4.1, but it is still valid for ESXi 5.0! Here William describes a way to enable (and disable!) SIOC for a datastore directly on a host without using (even without having available!) a vCenter server.
And it is really easy: All you need to know is the device ID of the datastore/LUN. In ESXi 5.0 you can find it out by using the command
   # esxcli storage vmfs extent list
It will list all datastores with their labels and device IDs (starting with naa.). And then you can use the following vsish command to disable SIOC for the device
   # vsish -e set /storage/scsifw/devices/<naa-id>/iormState 1496
That's it! After waiting a few seconds the affected datastore suddenly disappeared from the list of mounted datastores in the vSphere client, and the VMkernel.log error messages also stopped.

Please note: vsish is a powerful but largely undocumented utility to query and set VMkernel parameters. It is only available in an ESXi local or remote shell. William has quite a few posts about it on his virtuallyGhetto blog. This time it saved us the trouble and time of rebooting a whole cluster of hosts ...

What's new with HP ProLiant Gen8 servers

I recently visited an HP roadshow presentation of their new ProLiant server generation Gen8. This has been around for a while, and you might already have seen some announcements, slides etc. Sometimes it can be hard to really learn about what is actually innovative, new and better (compared to older generations) with Gen8, because of all the buzz words and marketing going on about initiatives like "Voyager", "ProActive Insight Architecture" and so on. So I took some notes about "the hard facts" and decided to turn them into this blog post.

According to HP Gen8 is basically about reducing the following three cost factors:

  • Manual actions
  • Energy costs
  • Unplanned downtimes

Before I focus on these three items, let's quickly summarize what hardware changes come with Gen8:

  • Support for Intel Sandy Bridge EP (Xeon E5-2600) processors. These new CPUs have
    • the I/O hub integrated on the CPU die
    • four (formerly three) memory channels per CPU socket
    • two (formerly one) Quick Path Interlinks (QPIs) per socket
    • up to 8 CPU cores
    • Support for PCIe 3.0 (formerly 2.0)
  • The new iLO generation 4
    • incorporating a new 4GB flash RAM (for storing drivers, firmwares etc.)
    • removing the necessity to deploy hardware management agents into the OS (everything is monitored by and through iLO)
    • introducing a mobile app (for Android and iOS) for remote management
  • New SmartArray controllers (22x and 42x models) with FBWC (= Flash Backed Write Cache) only (no more BBWC = Battery Backed Write Cache)

Reducing manual actions

Together with Gen8 HP introduced a new version (5.0) of their Smart Update Manager (SUM). It was completely rewritten to be more intelligent and shall be able to completely automate the update process of complex hardware setups with multiple dependencies. If you have ever planned and executed firmware updates for HP Blade enclosures then you know that this was a very time consuming and error prone process so far with lots of manual tasks. HP SUM 5.0 now promises to make it a no-brainer and a streamlined, completely automated experience without the requirement of planned downtimes. I am very curious to try this out ...

At the same time HP announced a new support policy for their Service Packs for ProLiant. These are bootable ISOs containing certified sets of drivers and firmwares for ProLiant hardware and the new HP SUM tool. They are the successors of the Firmware maintenance CDs/DVDs that you may already know. HP plans to release a new Service Pack every four months and will support at least three consecutive releases - that means you may update all your firmwares only every 12 months while keeping your environment fully supported by HP. Of course there will be hotfixes between releases to address urgent issues. You can find the relevant links on my HP & VMware links page.

Reducing Energy costs and unplanned downtimes

HP has greatly increased the number of hardware sensors in the new ProLiants, they are distributed all over the chassis on multiple levels forming a "3D sea of sensors". With "HP ActiveHealth" (another great marketing term) there is also a new system for detecting and pro-actively mitigating hardware failures. This intelligence is part of the new iLO4 board that will also store a complete history of more than 1.600 system parameters and measured values. This data can be exported and uploaded to HP SnS for hardware analysis and troubleshooting.

They also claim to have made their power supplies even more efficient: The best efficiency value is still 94%, but it is now kept over a broader range of output powers.

Availability and next steps

At the time there are Gen8 version of the two socket models ML350p, DL160, DL360p and DL380p, BL460c, SL230s and SL250s. Although one of the main drivers for the new platform is the new Intel CPU architecture there will also be Gen8 models of the AMD based ProLiants available later this year.

One last note: If you already have gotten your hands on a new Gen8 ProLiant and want to install VMware vSphere on it then be sure that you use the HP Customized ESXi ISOs. The VMware vanilla ISOs do not yet include the required hardware drivers.


Updated: What's the deal with EMC CX arrays not supporting VAAI with vSphere 5.0?

(If you have read this post before then skip to the bottom for recent updates)

We are in the process of upgrading our production environment to vSphere 5.0 U1. A new vCenter 5.0 server is already in place, and we attached the production ESXi hosts (that are still on 4.1 U2) to it.
The next step would be to upgrade the hosts. In the meantime, since this is a qualified production environment running more than 1.500 virtual servers I became a bit paranoid about hardware and software/firmware compatibility, and decided to double check if the environment is fully supported (or if we should upgrade any firmware first). I was pretty confident that we are on the safe side because we never had any issues with vSphere 4.1 and always kept the environment up to date.

But then I stumbled over this VMware KB article: EMC CX and VNX Firmware and ESX requirements for vStorage APIs for Array Integration (VAAI) support. It states that ESXi 5.0 does not support VAAI on our Clariion CX4 arrays even although they have the latest FLARE code:


vSphere 4.1vSphere 5.0
CX4 Series Flare 30/29/28+VAAI SupportedNot Supported
VNX Series OE 31 or Later *VAAI SupportedVAAI Supported

I checked the test hosts that we already updated and they showed VAAI being supported on the CX4 LUNs in the vSphere client. However, the KB article recommends to disable VAAI on these hosts ...

I quickly searched the Internet for relevant posts and statements and browsed through the VMware Community forums. There I found this post where people complain about the VAAI status shown as unsupported. It looks like there are dependencies to the LUN size (smaller or larger than 2TB) and to the array's failover mode (sounds like you need to use ALUA / failover mode = 4 which we are already using). But from this post I get the impression that everything is fixed with the latest 30.x FLARE code releases).

We contacted both VMware and EMC to get some clarification now, and I will keep this post updated with any new information. In the meantime I ask everyone using ESXi 5.0 with EMC CX storage to comment on this post whether you are using VAAI, if you have any issues with it, and what FLARE code you have on the arrays. BTW, you can check what VAAI primitives are in use with your storage LUNs by running the following esxcli command:
   esxcli storage core device vaai status get

Thanks in advance for any helpful comments!


Update (2012-05-20): I got some feedback from other users, VMware and EMC on this issue, and used my best Google-Fu to get related information about vSphere 5, VAAI, VMFS-5 and/or EMC CX support. Here are the results (somewhat unsorted):
Regarding my specific issue (lack of official support for VAAI with EMC CX arrays and ESXi 5.0) I finally got one really detailed and helpful comment from an EMC representative: EMC tested the VAAI functionality of ESXi 5.0 with their EMC CX4 arrays (I specifically asked for the CX4). It has been "tested as functional", but it only passed the certification tests of the ATS and Zero primitives, but failed the certification test of the XCopy primitive.
I wonder why this happened, because VAAI was fully supported with ESXi 4.1 (i.e. it must have also passed the XCopy test), and with ESXi 5.0 there were no changes to the XCopy primitive (at least I could not find any information about such a change).

Anyway, this is probably the reason why EMC and VMware do not officially support VAAI with ESXi 5.0 on CX arrays. From a customer's point of view this is disappointing, but from EMC's point of view this decision is understandable, because they want to focus on their current products and not waste support resources on somewhat outdated arrays like the CX ones. However, according to this EMC representative it may be possible to come to an individiual support agreement with EMC and VMware (EMC calls this RPQ = Request for Product Qualification) if you nevertheless must or want to have an official support statement for whatever reason.

I also had the idea of disabling only the XCopy primitive to be on the safe side, but I don't want to do this globally, because we also use a fully VAAI supported VNX array with our production hosts. So I was glad to find a link to the promising KB article KB2012967 (see list above), but this link currently doesn't work. I will ask VMware about this ...



Update (2012-06-20): In the meantime an EMC representative confirmed to me that all three VAAI primitives work fine with vSphere 5.0 and CX arrays, and that they even support it, but you need to file an RPQ with them in order to get formal support.

We have this configuration (VAAI turned on for both our CX4 and VNX arrays) running with vSphere 5.0 in production for about 6 weeks now without any problems.

The link to VMware KB article KB2012967 still doesn't work, but it is now listed to as being "[Archived]" in the reference section of KB1033665 ...





Update (2012-07-11): It turned out that you no longer need an RPQ with EMC. They now officially support the vSphere 5's VAAI implementation on the CX4. The combination is listed in their latest Simple Support Matrix for VMware vSphere 5 (Powerlink account needed for download):
Snippet from the EMC Simple Support Matrix for vSphere 5 (of July 2012)
VMware though does still not officially support it (KB2008822 is unchanged as of today). In another blog post I already explained the reason for this.







ImageBuilder Deep Dive, Part 2: A closer look and advanced functions

In the first part of this Image Builder Deep Dive I introduced a script that you can use to build your own customized installation ISO and can easily be adapted to your own needs. In this second part we will take a closer look at the basic cmdlets we used and get to know some more cmdlets with advanced functionality.

To make best use of the following explanations I suggest that you load the complete script that we go through in this part into the PowerShell ISE and run it line by line while reading the post. You can also experiment with the various code samples by pasting them into the command prompt of the ISE.

Using Online depots with proxy servers

At the beginning of the first part's script we added two Online depots to the ImageBuilder session using the Add-EsxSoftwareDepot cmdlet. I stated that this is not possible when using a proxy server, and  - after some testing - I need to correct this statement: I got it working with a proxy server that does not require authentication. With a proxy server requiring authentication it worked on the first try and failed when adding the second online depot (?!). Originally I had a proxy auto-configuration URL entered in the Internet Explorer settings, and this made adding an Online depot consequently fail. Your mileage may vary ...

By default PowerCLI will use the Windows proxy settings (that means what you have configured in Internet Explorer). You can change that by using the Set-PowerCLIConfiguration cmdlet. However, the possibilities are somewhat limited. You can either use the system proxy settings or no proxy at all:
# Load the VimAutomation Snapin
Add-PSSnapin VMware.VimAutomation.Core
#
# Set and get the proxy configuration
Set-PowerCLIConfiguration -ProxyPolicy NoProxy
Set-PowerCLIConfiguration -ProxyPolicy UseSystemProxy
Get-PowerCLIConfiguration
#
Please note: You need to add the VMware.VimAutomation.Core snapin first (if not already added) to use these cmdlets. To display the current setting use the Get-PowerCLIConfiguration cmdlet.

How to "download" the VMware Online base depot

Now you may wonder: What if your computer has no Internet connection (or only via a proxy configuration that causes the above mentioned problems)? Can I somewhat "download" an Online depot and re-use it later on the same or another computer? Yes, you can!

To be exact: You cannot create an offline copy of a complete Online depot, but you can export a specific ImageProfile not only to an installation ISO, but also into a re-usable Offline bundle:
# Load the ImageBuilder Snapin
Add-PSSnapin VMware.ImageBuilder
#
# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
#
# List available ImageProfiles sorted by name (filtered on updated standard profiles)
Get-EsxImageProfile "ESXi-5.0.0-2012*-standard" | Sort-Object -Descending Name
#
# The previous command outputs "ESXi-5.0.0-20120504001-standard" as the newest profile (as of May 5th 2012)
# Export this Imageprofile into an Offline bundle zip file
Export-EsxImageProfile -ImageProfile ESXi-5.0.0-20120504001-standard -ExportToBundle -FilePath 'U:\ESXi-5.0.0-20120504001-standard.zip'
#
Now you can re-use this exported zip file any time and on any other computer by adding it with the Add-EsxSoftwareDepot cmdlet. Obviously this Offline bundle zip file only contains the ImageProfile that you exported and only the packages that are part of this ImageProfile.

Please note: The HP Online depot cannot be "downloaded" in this way, because it does not expose any ImageProfiles that you can export. However, in this special case you can also open the http://vibsdepot.hp.com URL in a browser and download the Offline bundles that it includes from the linked web pages.

Looking closer at the ImageProfile object

Apropos ImageProfiles: So far we only looked at the names of these objects. Obviously VMware codes the release date of an ImageProfile (= patch level) into its name using the format YYYYMMDD. Hopefully they will stick with this naming standard - it makes it easy to identify the newest ImageProfile at first sight and programmatically.

The ImageProfile object has more interesting attributes though:
# Examining the ImageProfile's attributes
$ip = Get-EsxImageProfile ESXi-5.0.0-20120504001-standard
$ip | format-list
$ip.VibList
#
By piping the object through the format-list cmdlet you can display all its attributes in a nicely formatted list view:
Name            : ESXi-5.0.0-20120504001-standard
Vendor          : VMware, Inc.
Author          : 
Description     : For more information, see http://kb.vmware.com/kb/2019863.
CreationTime    : 30.04.2012 21:47:06
ModifiedTime    : 30.04.2012 21:47:06
ReadOnly        : False
VibList         : {ata-pata-atiixp 0.4.6-3vmw.500.0.0.469512, net-nx-nic 4.0.557-3vmw.500.1.11.623860, scsi-rste 2.0.2.0088-1vmw.500.1.11.623860, net-e1000 8.0.3.1-2vmw.500.0.7.515841...}
AcceptanceLevel : PartnerSupported
Guid            : 076da48a930e11e1b58d0017a477682c
Rules           : 
StatelessReady  : True
The Description attribute is interesting, because its value includes a URL to a VMware KB article with further information about this ESXi 5.0 patch release.

The VibList attribute is an array of all included software packages. The list is shortened in the formatted object view to retain readability of the output. The third command above ($ip.VibList) will output the complete list of packages with names and versions.

The AcceptanceLevel attribute is something that you normally don't need to take care of, but we will come back to it at the end of this post.

How to compare two ImageProfiles

If you want to quickly check the differences of two ImageProfiles you could list their attributes one after the other like shown before and compare the outputs, especially the VibLists. However, there is a more elegant way to do this - you can use the Compare-EsxImageProfile cmdlet:
# Comparing two ImageProfiles
Compare-EsxImageProfile ESXi-5.0.0-20120504001-standard -ReferenceProfile ESXi-5.0.0-20120404001-standard
#
This command will create and output an ImageProfile comparison object. In this example it looks like this:
Equal               : False
PackagesEqual       : False
RefAcceptanceLevel  : PartnerSupported
CompAcceptanceLevel : PartnerSupported
OnlyInRef           : {}
OnlyInComp          : {}
UpgradeFromRef      : {VMware_bootbank_esx-base_5.0.0-1.13.702118}
DowngradeFromRef    : {}
What do we see here? The two ImageProfiles are not equal (Equal = False), they do not contain the exact same list of packages (PackagesEqual = False), but they have the same AcceptanceLevel (PartnerSupported). The remaining four attributes summarize the differences in the software package list of the two ImageProfiles. Their names are pretty self-explanatory, and in this example the only difference is that the comparison profile (ESXi-5.0.0-20120504001-standard) contains a newer version of the esx-base package than the reference profile (ESXi-5.0.0-20120404001-standard).

Examining, adding and removing single software packages from a depot

In the first part of this series of posts we used a pipeline of commands to add all packages of a depot to our custom profile. Now we are going to take a closer look at a single package using the Get-EsxSoftwarePackage cmdlet:
# Reference downloaded HP offline bundle for be2net driver
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
#
# Look at all versions of a single software package (net-be2net)
Get-EsxSoftwarePackage net-be2net
# Look at all properties of a package's specific version
Get-EsxSoftwarePackage net-be2net -Version 4.0.88.0-1vmw.500.0.7.515841 | fl
#
The output of the command in line 34 looks like this:
Name                     Version                        Vendor     Release Date    
----                     -------                        ------     ------------    
net-be2net               4.0.88.0-1vmw.500.0.7.515841   VMware     15.12.2011 00...
net-be2net               4.0.88.0-1vmw.500.0.0.469512   VMware     19.08.2011 01...
net-be2net               4.0.355.1-1OEM.500.0.0.406165  Emulex     16.09.2011 00...
That means that there are three different versions of the net-ne2net package, the first two are in the VMware base depot, and the third one is in the Offline bundle we downloaded from HP.

In line 36 we examine the properties of a specific version of a package. We will see that the SoftwarePackage object has some interesting attributes:
Name            : net-be2net
Version         : 4.0.88.0-1vmw.500.0.7.515841
Vendor          : VMware
Summary         : Updates the ESX 5.0.0 net-be2net
Description     : For build informatin, see KB  http://kb.vmware.com/kb/2007675
ReleaseDate     : 15.12.2011 00:00:00
Depends         : {vmkapi_2_0_0_0, com.vmware.driverAPI-9.2.0.0}
Conflicts       : {}
Replaces        : {}
AcceptanceLevel : VMwareCertified
MaintenanceMode : False
LiveInstallOk   : False
LiveRemoveOk    : False
SourceUrls      : {https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/net-be2net/VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841.vib}
Guid            : VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841
StatelessReady  : True
Tags            : {driver, module, category:bugfix, severity:general}
Like with the ImageProfile object the Description property contains a link to a VMware KB article with further information on this package. The SourceUrl indicates from where the package will be downloaded. And the package's Tags provides additional classification categories.

The three properties Depends, Conflicts and Replaces describe relationships to other packages (or features that are provided by other packages). Their names are self-explanatory, and ImageBuilder will take care of them for us. You will e.g. not be allowed to add a package that conflicts with another package that is already part of the profile.

Now let's add this single package to a custom profile using the Add-EsxSoftwarePackage cmdlet:
# Create a new ImageProfile
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120504001-standard -Name MyProfile
#
# Add an older version of the net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile "net-be2net 4.0.88.0-1vmw.500.0.7.515841"
# Add the neweset net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile net-be2net
#
If you just specify the name of the software package (net-be2net in this example), and there are multiple versions of this package available then Add-EsxSoftwarePackage will automatically add the newest version. If you want to add an older version then you can specify it like shown in line 42. Whenever the ImageProfile already includes a package of the same name, but a different version, it will be replaced by the added package, even if it is older! If the ImageProfile already includes the exact same version then the command will fail, or in other words: you cannot add the same version twice.

Finally, you can also remove a package from an ImageProfile using the Remove-EsxSoftwarePackage cmdlet:
# Remove a package from the custom profile
Remove-EsxSoftwarePackage -ImageProfile $MyProfile net-bnx2
#
You won't be able to remove packages that are required by other packages of the ImageProfile. In some case you can override this dependency check by using the parameter -Force, but you really shouldn't do that, because it will most likely result in an invalid ImageProfile.

Adding community developed packages

If you have ever used my ESXi-Customizer script to add a Community developed software package to an ESXi 5.0 installation ISO then you may wonder: Can you also do that with ImageBuilder? Yes, you can, and here is how:
# Add an Offline Bundle with a community developed software package
Add-EsxSoftwareDepot 'U:\$Download\fwenable-ntpd-1.2.0-0-offline_bundle.zip'
# Change the AcceptanceLevel of the custom profile to "CommunitySupported"
$MyProfile.AcceptanceLevel = "CommunitySupported"
# Add the package to the profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile fwenable-ntpd
In this example we add a software package that includes a custom firewall rule to allow incoming NTP queries (this way you can use the ESXi host as NTP server. Read this post for background information). The Offline bundle was created using my ESXi5 Community Packaging Tools by converting a standard tar.gz file to a VIB file first, and then packaging the VIB file into an Offline bundle zip. The project page of the tools contains detailed instructions on how to do this.

There is only one particularity that you need to be aware off when adding Community developed packages this way: Software packages that were created (and certified) by VMware or trusted partners contain an electronic signature that is checked when adding (or installing) the package. This qualifies them for one of the following three so-called AcceptanceLevels: VMwareCertified, VMwareAccepted or PartnerSupported. Of course, Community developed packages do not have trusted electronic signatures, which means that you can only install them if they have the least restrictive (and least trustworthy) AcceptanceLevel called CommunitySupported. The ImageProfile object also has an AcceptanceLevel, and that must be lowered to the same level before you are able to add a Community developed package. This happens in line 52.

This ends the second part of the ImageBuilder Deep Dive series. The next and last part will not deal with the ImageBuilder snapin specifically, but with Powershell coding in general. We will then develop the script from the first part to a more general and versatile solution.