vCSA 6.0 tricks: shell access, password expiration and certificate warnings


It is no secret anymore that VMware is putting more and more energy in getting rid of the Windows based vCenter product and pushes us towards the Linux based vCenter Server Appliance (vCSA). Up to now the appliance was already a good choice for labs and other small deployments, but it lacked the scalability of the Windows version, and was not a good fit for large production deployments.

With the release of vSphere 6 this has changed due to the evolvements of the vCenter architecture and significant updates that were made to vCSA. The biggest remaining caveat now is that you still need a Windows machine for VMware Update Manager (VUM). I do not really use VUM in my lab, so I will definitely switch to the vCSA - whenever the time will come for me to do the 6.0 upgrade.

So I took a closer look at vCSA 6.0 ... and stumbled over some issues that are definitely worth sharing.

I will not explain you how you install the vCSA ... because with the new guided installer this is as easy as pie (Good job, VMware!). However, in the installation wizard there were two things that got me thinking immediately: You need to define sufficiently complex passwords for the local root user of the appliance and the SSO admin, and there is an option to Enable ssh in the Networking section. Both are security related for sure, but there is more to it.


An appliance is an appliance is an appliance ... (*)

VMware has worked very hard on making the vCSA an appliance that can be fully configured and managed through a web interface and CLI tools using well defined APIs. That reminds us of the ESXi hypervisor into which they put the same efforts, and if you look at the vCSA console (after having looked at an ESXi console) you will have a strong déjà vu:



... but this one is also a Linux box ;-)

Nevertheless (just like with ESXi) you can still enable and use shell access to the vCSA, either by pressing Alt-F1 at the console, or - way more comfortable - by enabling ssh for remote shell access. If you have not already done this in the deployment wizard then you can also enable ssh in the customization menu (F2) at the console (see Troubleshooting Mode Options) or in the vSphere Web Client (buried in System Configuration / Nodes / Manage / Settings / Access).

When you log on to the appliance via ssh using the local root credentials you will not come to a standard Linux shell (bash), but you will be greeted with the command prompt of the appliance shell:


The appliance shell exposes command line access to API functions that you can use for system configuration (network settings etc.) tasks - for a complete list use "help api list". The interface is extensible through plugins which include standard Linux commands like ps and top, but also real gems like vimtop and the Ruby VSAN Console rvc (use "help pi list" for a complete list).

And there is also a plugin named shell that starts a standard bash shell. But it needs to be enabled first using the Troubleshooting Mode Options menu at the console (see above) or directly here with the CLI command shell.set like shown in the welcome message. What the welcome message does not show is the additional parameter --timeout of this command. By default the timeout is set to 3600s (= 1 hour), and that means that the shell command will be automatically disabled again after one hour.

There is no way to disable this timeout completely, but the maximum value for it is 231-1 = 2.147.483.647. Using this value will get you covered for the next 68 years :-)

   shell.set --enabled true --timeout 2147483647

By the way you can watch the timeout counting down with the command shell.get. This will show you whether the shell is enabled and - if yes - how many seconds are left until automatic disablement. Please note that this counter will not be reset when the appliance is rebooted, so you need to use the above command really only once to get (somewhat) permanent bash access to the vCSA!

When you have started the shell this warning message will be displayed:


Please take this advice seriously and try to let VMware know of scenarios where bash access is still required! In the following I will describe such scenarios and I will let them know ... However, when I tried to open
   http://communities.vmware.com/community/vmtn/vcenter/cloudvm
in my browser I only got a "Not found" message - well, I will also let them know about that :-)
For now you will have more luck with
   http://communities.vmware.com/community/vmtn/vcenter


Using WinSCP to exchange files with the appliance

There are few occasions where you need to upload files to the vCSA, e.g. for installing the Microsoft sysprep files for the vCenter Guest customization feature (as per KB1005593). You typically do this by enabling ssh like described above and then using a Windows tool like WinSCP (or scp for Linux users).

When I tried to connect to my vCSA 6.0 with WinSCP it throwed this error message at me:


Yes, WinSCP and also other scp based utilities expect a standard shell at the other end and cannot handle the appliance shell of the vCSA!

There is a workaround for this that does not require any changes to the appliance: In WinSCP configure the vCSA connection to use the SFTP protocol (instead of SCP) and in the Advanced Settings of the connection enter "shell /usr/lib64/ssh/sftp-server" (without the quotes) to be used as the SFTP server.

Another workaround is to (temporarily) change the default shell of the appliance's root user to /bin/bash. Use the command

   chsh -s /bin/bash

to achieve this (after starting the shell plugin from the CLI prompt). You can revert to the standard appliance shell with the command

   chsh -s /bin/appliancesh


Password expiration and complexity

Are you aware of that the passwords of both the local root user and the SSO admin account have expiration dates set by default?! This is forgotten very often when new appliances are deployed ... Of course, from a security standpoint this is best practice, and there are environments where it is even required to have expiration dates on all passwords. But in lab environments this annoys lots of people and leads to confusion and unexpected issues whenever a password expires unexpectedly.

For the SSO admin you can configure the password policy in the Web Client (Administration / Single Sign-On / Configuration / Policies / Password Policy):


By default the password maximum lifetime is 90 days. You can disable expiration completely by setting the value to 0.

In this dialog you can also edit the password complexity requirements. I like to disable the "At least 1 special characters" requirement here, because - historically - there have always been issues with special characters in passwords like the one that is documented in KB2060746.

What about the local root user? You can check its password expiration settings with the command

   chage -l root

in a bash shell. By default it has a maximum age of 365 days (= 1 year). You can change it with the same command and the switch -M. Setting a value of -1 will disable expiration completely:

   chage -M -1 root

To change the password complexity settings of the local root user you need to edit the file

   /etc/pam.d/common-password

with your favorite editor - this is vi, right? :-)

Please refer to the man pages of pam_cracklib and pam_pwhistory to understand the meaning of the parameters in this file!

Another note: There is a daily cronjob (/etc/cron.daily/pass-expiration) in the appliance that checks for an expiring password and is supposed to send out an e-mail with a warning a week before. But this will only work if an e-mail address was defined for the root account! So - if you leave password expiration enabled - then you better do this by using the command

  user.set --username root --email yourmail@somewhere.org

in the appliance shell.


These darned certificate warnings ...

A while ago I wrote an article about How to avoid browser warnings when using the vCenter Web Client with a self-signed certificate. With vSphere 6 VMware has vastly improved certificate management - in fact vCenter now includes a Certificate management service that - by default - creates an own Certificate Authority (CA) root certificate and signs all other used certificates with it. So you "only" need to trust this CA certificate. In the vCSA 6.0 this is in the file

   /var/lib/vmware/vmca/root.cer

I have updated the above mentioned post with this information. Please refer to it to learn how to trust this certificate and eliminate the SSL warnings in your web browser.



This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.



17 comments:

  1. Thanks for the tip on importing the CA root certificate, couldn't find this detail anywhere in VMware docs.

    ReplyDelete
  2. Useful tips, especially the SSH config required. Thanks.

    ReplyDelete
  3. Some very handy tips - VCSA 6 is the way to go for a lot of use cases now.

    ReplyDelete
  4. Hi... Do you have a way to regenerate a certificate through CLI on VCSA 6.0?

    ReplyDelete
  5. Hey, do you know where to find that link in Web Client to enable SSH. Cannot find it anywhere, and F2 doesn't work. Latest vCSA download from a week ago.

    ReplyDelete
    Replies
    1. Good point, Anonymous,

      since Update 1 the VAMI interface is back (on https://...:5480), and there you can enable SSH and configure shell access now.

      Andreas

      Delete
  6. Last Night An Admin Saved My LifeNovember 25, 2015 at 10:31 AM

    Nice article. Coming from a 5.1 (Windows) Vcenter I find it a little bit hard to adopt... :P

    Do you know how can someone grant shell access to the VCenter Appliance 6 for a specific user / group? Only the root user can login via SSH and I haven't found any option that could enable other users too..

    ReplyDelete
    Replies
    1. Good question. What is your use case? Do you want to log in with a local user or an AD user?

      Delete
    2. Last Night An Admin Saved My LifeNovember 25, 2015 at 4:32 PM

      I have my VCenter configured with LDAP. It would be nice if I could login with both, a local user and with an LDAP user.

      *btw "adapt" not "adopt". Just noticed it :P

      Delete
  7. useful post especially for Wincsp workaround
    thank you

    ReplyDelete
  8. Great post. I have just found another way to stop the vCenter Appliance root password expiring (and to enable/disable SSH and Bash). In vCenter 6, you can use the VMware Appliance Manager here: https://ApplianceHostname:5480

    Naysan

    ReplyDelete
  9. The shell trick does not work anymore :(
    The maximum timeout is 86400 seconds(1 day). (code 2005)

    ReplyDelete
  10. Eliminating the SSL waring should be done the right way after selecting the right option for your environment. Even in a lab it makes good practice: http://www.derekseaman.com/2015/02/vsphere-6-0-install-pt-1-introduction.html

    ReplyDelete
  11. Thank you! This saved me from jumping through hoops to get a file onto the appliance.

    ReplyDelete
  12. VMware doesn't recommend directly modifying pam.
    Instead got to the esxi host/advanced settings/Security.PasswordQualityControl
    Then enter the pam setting there. Be careful as you can render the host unusable.
    To set esxi 6 password controls back to esxi 5 use:
    "retry=3 min=8,8,8,7,6"

    ReplyDelete
  13. Please note as of 10/8/2016 vCenter 6.0 will no longer accept a shell timeout greater than one day (86400). When entering your command above (shell.set --enabled true --timeout 2147483647)the following error is thrown:

    Error in method: The maximum timeout is 86400 seconds(1 day). (code 2005)

    Bummer!

    Thanks,
    Steve

    ReplyDelete

***** All comments will be moderated! *****
- Please post only comments or questions that are related to this post's contents!
- Advertising and link spamming will not be tolerated!