[Updated] esxcli plugin to run shell commands (new version)

[Please note: I have updated this blog post to reflect a new version of my esxcli-shell plugin and the changes in there. What's new in version 1.1 (of 2012-01-12):
- supports for advanced shell function like pipelines, I/O redirection
- execute multiple shell commands at once (or a single command spanning multiple lines)
- logging of shell commands to /var/log/shell.log ]

The esxcli command is a very powerful tool to query and configure various aspects of an ESXi host's configuration, and you can not only use it in an ESXi shell, but also remotely through the (perl based) vSphere CLI and the (PowerShell based) vSphere PowerCLI.

The esxcli commands are organized in so-called namespaces (e.g. hardware, software, network etc.) for managing the sub-components of the server, and - out-of-the-box - you are limited to the commands that are exposed through these namespaces. But you cannot run arbitrary ESXi shell commands through it ... until now!

I have released an esxcli plugin (for ESXi 5.x) that allows to run any ESXi shell command through esxcli. It implements the esxcli namespace shell:

esxcli shell help
What is this good for? It allows you to run any ESXi shell command remotely without enabling and using ssh! This means e.g. that you can start e.g. shell scripts (like William Lam's ghettoVCB.sh script) through a remote esxcli or a PowerCLI script.

Installation

The esxcli-shell package is of the AcceptanceLevel CommunitySupported, so you need to change your host's configuration to this level before you can install the package. You can do this via the vSphere Client (see Host Configuration / Security Profile / Host Image Profile Acceptance Level: Edit...) or by running the esxcli command
   esxcli software acceptance set --level CommunitySupported

Download the latest version of the esxcli-shell Offline Bundle from the V-Front Online Depot and upload it to a datastore of your ESXi host. Then install it with the following esxcli command:
   esxcli software vib install -d /vmfs/volumes/datastore1/esxcli-shell-X.X.X-Y-offline_bundle.zip

(X.X.X-Y is the current version number of the plugin. Replace datastore1 with the name of the datastore that you have uploaded the bundle to.)

Alternatively you can install the package directly from the Online Depot (provided that your host has direct Internet access) with the commands
   esxcli network firewall ruleset set -e true -r httpClient
   esxcli software vib install -d http://vibsdepot.v-front.de -n esxcli-shell


In order to make the installation effective you need to either reboot the host or restart the hostd daemon by running
   /etc/init.d/hostd restart

in an ESXi shell. Please note: This will not interrupt any VMs, but will drop any management connections like vSphere client and remote esxcli sessions!

You can also deploy esxcli-shell through Update Manager by importing the Offline Bundle and adding it to a baseline. The hosts' acceptance level still needs to be changed (like described above) though, and you need to manually reboot the hosts or restart hostd on them, because the package does not force a reboot.

Usage with vSphere CLI

Obviously it doesn't make any sense to use esxcli shell commands in a local or remote (ssh) ESXi shell, because there you can run any shell commands directly. However, you can also run esxcli commands remotely from a Windows or Linux machine by installing the vSphere CLI on it. To run an ESXi shell command you then need to use the following syntax
   esxcli -s hostname -u username -p password shell cmd -c "command1" [-c "command2" ...]

hostname is the name of the ESXi host that you want to run the command on. username is the name of a privileged user on this host (normally root), and password is the password of this user. command1, command2 etc. are the shell commands that you want to execute (at least one is required). If a command contains spaces then enclose it in quotes (like "ls -la").

Since version 1.1 you can use any character in a command line that is also available at a regular shell prompt. This includes the pipeline (|) and input (<) and output (>) redirection symbols. For easy output redirection of all specified commands you can also use the options -l (or --logfile=) and -e (or --errlog=) to specify log files for redirecting the standard and error output channels to. If you do not want to overwrite existing log files, but append to them, you need to use the flag -a (or --lappend). Finally you can make esxcli change the working directory before executing the command by specifying it with  -d (or --workdir=). The default working directory is /var/log/vmware. So the complete syntax like shown in the screenshot above is:

   esxcli -s hostname -u username -p password shell cmd -c "command1" [-c "command2" ...] [-l logfile] [-e errorlogfile] [-a] [-d workdir]

Here are some more examples using the vSphere CLI in Windows, from simple to complex:
REM #--- Display the plugin's about info and disclaimer:
esxcli -s vESXi01 -u root -p *** shell about

REM #--- Run a command pipeline 
esxcli -s vESXi01 -u root -p *** shell cmd -c "vmkload_mod -l | grep e1000"

REM #--- Run "lsof" and append output to a log file:
esxcli -s vESXi01 -u root -p *** shell cmd -c "lsof" -l "/vmfs/volumes/LStore1/lsof.log" -a
REM #- or -
esxcli -s vESXi01 -u root -p *** shell cmd -c "lsof >>/vmfs/volumes/LStore1/lsof.log"

REM #--- Run "vmkfstools -y" for space reclamation via UNMAP on datastore DS01:
esxcli -s vESXi01 -u root -p *** shell cmd -c "vmkfstools -y 90" -d "/vmfs/volumes/DS01"
REM #- or -
esxcli -s vESXi01 -u root -p *** shell cmd -c "cd /vmfs/volumes/DS01" -c "vmkfstools -y 90"

REM #--- Run a vendor program that prompts for input twice, answer with "answer1" and "answer2":
esxcli -s vESXi01 -u root -p *** shell cmd -c "echo -e \"answer1\\nanswer2\" | /opt/vendor/bin/program"
REM #- or -
esxcli -s vESXi01 -u root -p *** shell cmd -c "(cat <<END" -c "answer1" -c "answer2" -c "END" -c ") | /opt/vendor/bin/program"

Usage with PowerCLI

Here are the same examples in PowerCLI syntax:
# Connect to ESXi host (will prompt for credentials)
Connect-VIServer vESXi01

# Create the Esxcli object
$esxcli = Get-EsxCli -VMHost vESXi01

#--- Display the plugin's about info and disclaimer:
$esxcli.shell.about()

#--- Run a command pipeline 
$escli.shell.cmd("vmkload_mod -l | grep e1000",$null,$null,$null,$null)

#--- Run "lsof" and append output to a log file:
$esxcli.shell.cmd("lsof",$null,$true,"/vmfs/volumes/LStore1/lsof.log",$null)
#- or -
$esxcli.shell.cmd("lsof >>/vmfs/volumes/LStore1/lsof.log",$null,$null,$null,$null)

#--- Run "vmkfstools -y" for space reclamation via UNMAP on datastore DS01:
$esxcli.shell.cmd("vmkfstools -y 90",$null,$null,$null,"/vmfs/volumes/DS01")
#- or -
$esxcli.shell.cmd(("cd /vmfs/volumes/DS01","vmkfstools -y 90"),$null,$null,$null,$null)

#--- Run a vendor program that prompts for input twice, answer with "answer1" and "answer2":
$esxcli.shell.cmd("echo -e `"answer1\\nanswer2`" | /opt/vendor/bin/program",$null,$null,$null,$null)
#- or -
$esxcli.shell.cmd(("(cat <<END","answer1","answer2","END",") | /opt/vendor/bin/program"),$null,$null,$null,$null)
Passing options to an esxcli command is a bit tricky in PowerCLI: You always need to list all the possible arguments in a specific order - and this is the alphabetical order of the long option names. The --help screen (as shown in the screenshot above) will show the options in exactly the right order:
  1. command(s) to run (--command), specify as array if more than one: ("cmd1","cmd2", ...)
  2. name of error log file (--errlog)
  3. flag for append mode (--lappend): $true$false or $null for the default (=$false)
  4. name of log file (--logfile)
  5. name of working directory (--workdir)
If you do not want to use an optional parameter then just specify the value $null for it.

Syslogging / Auditing

For auditing reasons any command that you type in a regular shell is logged to /var/log/shell.log. In version 1.1 of the esxcli shell plugin I added code that will also log all executed commands to this file. Unfortunately all the commands that are run through esxcli will appear as if they have been run by the user root - even if you used another account for the remote esxcli connection -, because they are run through the hostd daemon that runs in the security context of root. I haven't found a way to safely determine the remote user's account.

Defeating command timeouts

If the shell command(s) that you execute through the esxcli shell plugin in PowerCLI take a very long time to complete then the command might fail with an error message "The operation timed out". You then need to raise the PowerCLI timeout for web operations with the command

  Set-PowerCLIConfiguration -WebOperationTimeoutSeconds n

with n being the new timeout value (in seconds). The default is 300 sec. (= 5 min.). You can also disable the timeout by using a negative value for n, but I would not recommend this.

Please note that this feature requires PowerCLI 5.1.

The Making of ...

I created this esxcli plugin after reverse-engineering already existing plugins. It turned out to be relatively easy and only requires some understanding of XML and Linux shell code. I will soon post more details about this ...

18 comments:

  1. Great work Andreas..will be a great help.

    ReplyDelete
  2. I like this reverse-engineering stuff...

    ReplyDelete
  3. Can we deploy the plugin using update manager ?

    thanks for the awesome work

    ReplyDelete
    Replies
    1. Yes, you can deploy through Update Manager. You still need to change the hosts' acceptance level though (like described in the post), and you need to manually reboot the hosts or restart hostd on them after the remediation, because the package does not force a reboot.

      Delete
    2. Have you actually tested deployment of custom VIBs via VUM? From my testing, even after setting the acceptance level to CommunitySupported, it seems that VUM still prevents custom VIBs to be staged & installed.

      Delete
    3. It works, I just tested the deployment of my esxcli-shell bundle with VUM in vCenter Server 5.1.0b and an ESXi 5.1 host.

      It will fail if a CommunitySupported VIB contains files that are not allowed in it (like your virtuallyghetto.vib), but this is not the case with the esxcli-shell VIB.

      Delete
  4. Haven't seen much activity here, but does this plugin work with ESXi 6?

    ReplyDelete
    Replies
    1. Hi Ben,

      yes, the plugin is still compatible with ESXi 6.0 (incl. U2). See the Wiki page.

      Andreas

      Delete
  5. Thank you so much sir, you save my weekend by providing this plugin, GOD bless you

    ReplyDelete
  6. Hi Andreas, is this plugin compatible with ESXI 6.5?

    ReplyDelete
    Replies
    1. I would expect that it is still compatible with ESXi 6.5 (and 6.7), because there were no known changes introduced to the esxcli subsystem. However, I have not tried it myself, because I do not really see a purpose for this nowadays.

      What do you want to use the plugin for?

      Delete
    2. Hi Andreas,
      This was an awesome find! Thank you very much for creating and posting this. I installed in on ESXi 6.7u3 and it works great.

      Delete
  7. Works with 6.7 , thanks man ....

    ReplyDelete
  8. Can't change root password, is that due to the security context the VIB is running under?

    $PwdStr = "NewPassword"
    $EsxCli.shell.cmd("echo -e `"$($PwdStr)\\n$($PwdStr)`" | passwd root")

    RESULTS:
    Enter new password: Re-type new password: passwd: Authentication token manipulation error
    Changing password for root

    You can now choose the new password.

    A valid password should be a mix of upper and lower case letters,
    digits, and other characters. You can use a 7 character long
    password with characters from at least 3 of these 4 classes.
    An upper case letter that begins the password and a digit that
    ends it do not count towards the number of character classes used.

    Alternatively, if no one else can see your terminal now, you can
    pick this as your password: "Avid8cloud8Delay".

    passwd:

    Password is not changed, however using the string generated by:
    "echo -e `"$($PwdStr)\\n$($PwdStr)`" | passwd root"

    And pasting into a putty SSH session, works fine.

    ReplyDelete
    Replies
    1. Greetings,

      you don't need my esxcli shell plugin to change the root password.
      Use this esxcli command instead:

      esxcli system account set -i root -p newpassword -c newpassword

      - Andreas

      Delete
    2. Great catch. Thanks. Follow-up question, does EsxCli inherit permissions from vCenter? Being that EsxCli-Shell allows console level commands to be run, could it allow a user to gain elevated access beyond what is assigned in vCenter permssions?

      Delete
  9. I have figured out how to install and uninstall this VIB using only native PS/PowerCli commands, HOWEVER, I'd love to see if this can be done without SSH, a Reboot, or DCUI, in order to get hostd to load the newly installed VIB.

    It seems a bit ironic that in order to save on using SSH, and expedite making calls to the host, I have to use SSH, the Console, or a full reboot to use it. Would be great to be able to install without disruption. (When you manage 15,000 hosts time is in short supply)

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!