Home > CA SSL Certificates, Security, VMware > vCenter Server Virtual Appliance – Changing SSL Certs Made Easy

vCenter Server Virtual Appliance – Changing SSL Certs Made Easy

I’ve been updating my vCenter and ESXi certificates recently and I ran into one particular system so far that had absolutely no documentation or KB articles to help with changing default SSL certificates for CA signed ones. The system was my vCenter Server Virtual Appliance. You might remember that I wrote about this as a means of using it as the vSphere Web Client without needing an additional Microsoft Windows License and then I used it with a load balancer to Increase vSphere Web Client Availability and Scalability. But a lack of documentation wasn’t going to stop me. Being a SLES based virtual appliance though meant things were quite different when changing the certs. If you want to save yourself a lot of time changing the SSL Certificates for the vCenter Server Virtual Appliance then read on.

If you want a way to fully manage the certificate lifecycle and replace certs automatically then you’ll want to check out vCert Manager – Changing VMware SSL Certs Made Easy. This will completely automate the SSL certificate process in vSphere environments. 

I tried to find information on changing the SSL certificates for the vCenter Server Virtual Appliance (VCVA) via Google and also on the VMware web site. The only information I was able to find that was in the correct context was the following:

vCenter Server Appliance: Where Do I Upload SSL Certificate on the VMware Communities Site

vSphere 5 vCenter Server Virtual Appliance Quick-Start Guide

Automating vCenter Server Virtual Appliance Configurations by William Lam

vCenter Management Appliance Certificates by Doug Baer

Now before we begin, because the vCenter Server Virtual Appliance is Suse Linux Enterprise Server based you will have to be used to a Linux command line, using scp, and generally navigating around in order to successfully change your certificates. All operations will be done as root. The default password is vmware. Like in my previous articles regarding changing SSL Certificates I have included an example OpenSSL configuration file that you can use to generate your certificates.

The following directories on the VCVA contain SSL certificates in one form or another:

/opt/vmware/etc/lighttpd/

/etc/vmware-vpx/ssl

/usr/lib/vmware-vpx/inventoryservice/ssl

/usr/lib/vmware-vsphere-client/server/config

I will go through what needs to go where after I’ve given you what you need to create the certificates.

Step-by-Step Process for Changing SSL Certificates on VCVA

You could execute a similar process to the one I’m about to describe using an OpenSSL or Public CA and using the Unix/Linux version of OpenSSL, however this is how I did it successfully in my lab and with my customer. As mentioned in the vSphere 5 Security Guide VMware uses X.509 v3 SSL certificates (base-64 encoded) for encrypting traffic between various components. If you CA has been set to support only SHA512 hash that is fine, it will work. The three key files for an VCVA are rui.crt, rui.key and rui.pfx.

In order to generate the certificates you’ll need to get a copy of OpenSSL x86 v0.98r or higher, and have access to a Microsoft CA (2003 or higher). The certificates will use a clone of a standard web server request template with Subject Alternative Name added, for my lab I modified the default Web Server Certificate Template to accept up to 15 years for certificates. On the system where you will generate the certificate signing request rui.csr) you will need to ensure you have Microsoft Visual C++ 2008 Redistributable Package (x86) before installing OpenSSL. For the purposes of this process you will use the Microsoft CA Web Pages to submit the certificate request and download the resulting base-64 encoded certificate. You can use the certreq command if you wish also (not covered here). Before applying the certificates to your environment you should ensure that your clients and vCenter server trust your CA, if it’s an AD integrated CA this should be automated (if using Internet Explorer), else you may have to pre-trust the Root or Intermediary CA  by loading the CA public cert into your clients and vCenter server (not covered in this process).

Note: If you are using a keypair provided by a commercial CA or already have a key and certificate generated outside of the VCVA you will need to import the keypair into the keystore to cover steps 21 – 23. Example instructions of how to do this you can follow Importing private keys into a Java keystore using keytool by Graham Leggett.

Prerequsites:

Microsoft CA (2003 or above, with Web Server Template with Subject Alternative Name included and configured to your liking).
Microsoft Visual C++ 2008 Redistributable Package (x86) on the system where you will generate the certificate signing request (CSR).
OpenSSL 0.98r or above on the system you will use to generate the CSR vCenter 5.0.
A VCVA Deployed and configured the way you like it.
A backup or snapshot of your VCVA prior to beginning this process would be recommended.

Process Step by Step:

  1. Before you start this process you should log into vCenter Server and check that all the services linked with Web Services are working, such as Hardware Status Tab, vCenter Service Status, and also Profile Driven Storage. These are the areas that are very likely to get broken if the process is not followed correctly.
  2. After having installed Microsoft Visual C++ 2008 Redistributable Package (x86) and Open SSL 0.98r or later on a management system (vCenter or other system, not the CA) open a command prompt (As Administrator if on Windows 2008) and change to the OpenSSL\bin folder. Use the same command prompt opened As Administrator for all the OpenSSL actions in this list.
  3. Edit the openssl.cfg file and ensure it looks similar to the one included at the bottom of this article but with your organization specific information, save the configuration.
  4. Execute the following command – openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg. Note: there will be no prompts as all the information is contained within the configuration file. This is a bit different than when generating the CSR for the ESXi hosts, but that is because there are more of them, and you may not want to have to generate entire config files for each host. For ESXi hosts it is much easier to just hit enter a few times and then specify a common name (fqdn) and then more on. However for vCenter and Update Manager it is better to have everything in the config file, especially as you will likely be specifying multiple Subject Alternative Names (SAN’s – not to be confused with storage area networks).
  5. Copy or submit rui.csr to your CA, submit an advanced certificate reqeust, using the Web Server template that you modified, and download the base-64 encoded certificate to the system with OpenSSL that was used to generate the CSR (Screenshots of this available here: How to use CA certificate to replace VMware certificate on ESX(i) 4 and vCenter or here: vSphere 5 Certificates – Replacing the Default vCenter 5 Server Certificate.)
  6. Download your root CA certificate and save it as root.cer, copy it to the same location where you generated the CSR.
  7. Execute the following command – openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx. This will create a rui.pfx file which we will now verify.
  8. Execute the following command – openssl pkcs12 -in rui.pfx -info. When prompted enter the password testpassword. You should see an base64 encoded string or characters displayed on the screen and information about the file. Note you will be asked to enter the password twice when it is displaying the private key.
  9. Log into the VCVA through the console or via SSH as root and create a directory called /root/certs where you will copy the new certificate files and root CA files.
  10. Using WinSCP or another SCP/SFTP client tool copy rui.crt, rui.key, rui.pfx and your root CA cert (root.cer) to /root/certs on the VCVA.
  11. On the VCVA console (or SSH login) as root execute cd /root/certs.
  12. Execute mv /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.bak, this will backup the lighthttpd cert.
  13. Execute mkdir /etc/vmware-vpx/ssl/backup.
  14. Execute mv /etc/vmware-vpx/ssl/rui.* /etc/vmware-vpx/ssl/backup, this will backup the vCenter Server SSL certs.
  15. Execute cat rui.key rui.crt >/opt/vmware/etc/lighttpd/server.pem, this will create the new lighthttpd cert.
  16. Execute cp rui.* /etc/vmware-vpx/ssl, this will update the vCenter Server SSL certs.
  17. Execute mkdir /usr/lib/vmware-vpx/inventoryservice/ssl/backup.
  18. Execute mv /usr/lib/vmware-vsphere-client/server/config/keystore /usr/lib/vmware-vsphere-client/server/config/keystore.bak, this will backup the keystore used by the vSphere Web Client.
  19. Execute mv /usr/lib/vmware-vpx/inventoryservice/ssl/rui.* /usr/lib/vmware-vpx/inventoryservice/ssl/backup, this will backukp the Inventory Service SSL certs.
  20. Execute cp rui.* /usr/lib/vmware-vpx/inventoryservice/ssl, this will update the Inventory Service SSL certs.
  21. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -genkey -keyalg rsa -alias s2dmk.
  22. When prompted enter the information corresponding to your environment, where the first and last name is the fqdn of the VCVA. An example is as follows:
    What is your first and last name? [Unknown]:vcva.example.com
    What is the name of your organizational unit? [Unknown]:Engineering
    What is the name of your organization? [Unknown]:Example Corporation
    What is the name of your City or Locality? [Unknown]:Palo Alto
    What is the name of your State or Province? [Unknown]:California
    What is the two-letter country code for this unit? [Unknown]:US
    Is CN=vcva.example.com, OU=Engineering, O=”Example Corporation”, L=”Palo Alto”, ST=California, C=US correct?[no]:yes
    Enter key password for <http> (RETURN if same as keystore password):
  23. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -certreq -alias s2dmk – file s2dmk.csr.
  24. Execute /usr/lib/vmware-vpx/jre/bin/keytool -storetype JCEKS -storepass testpassword -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -import -alias root -file root.cer.
  25. Using WinSCP or SCP or another SFTP client copy /root/certs/s2dmk.csr to your CA and submit an appropriate certificate request and download a base64 encoded certificate, save it as s2dmk.crt.
  26.  Using WinSCP copy s2dmk.crt to /root/certs on your VCVA system.  WARNING: Please make sure you use the Text transfer method else your certificates will be corrupted with additional ASCII characters (^M).
  27. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -import -alias s2dmk -file s2dmk.crt.
  28. Execute /usr/lib/vmware-vpx/jre/bin/keytool -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword -list to verify that the s2dmk and root certificate have been successfully loaded.
  29. Reboot the VCVA by executing the command reboot, or using the vSphere Client to restart the VCVA guest OS.
  30. Log back into the VCVA as root using the console or thruogh SSH, as we now need to re-register the vCenter Server with the vSphere Web Client to update the SSL certificate thumbprint.
  31. Execute the following command to unregister the local vCenter system from the vSphere Web Client: /usr/lib/vmware-vsphere-client/scripts/admin-cmd.sh unregister https://localhost:9443/vsphere-client localhost root <rootpw>.
  32. Now we will execute the command to re-register the local vCenter server again with this vSphere Web Client: /usr/lib/vmware-vsphere-client/scripts/admin-cmd.sh register https://localhost:9443/vsphere-client localhost root <root pw>.
  33. Type CRTL-D to log out of the VCVA.
  34. When the VCVA has finished restarting you can log into it using the vSphere Web Client, or vSphere Client and check that the certificates are correct. Provided your client trusts your root CA you should not be given a warning message.

Based on William Lam’s comments below it is possible to simplify the above process as follows:

Step 15 and 16 can be replaced with:
vpxd_servicecfg service stop
vpxd_servicecfg certificate change rui.crt rui.key

I would recommend that you check out his post on Automating vCenter Server Virtual Appliance Configurations.

Please let me know if you have any trouble with the above process, and also if it works for you, your comments and feedback are appreciated.

Example OpenSSL Configuration file (openssl.cfg) without most of the normal comments and white space that is included:

# Example vSphere Web Client OpenSSL Configuration File HOME   = . RANDFILE  = $ENV::HOME/.rnd oid_section  = new_oids

[ new_oids ]

[ ca ] default_ca = CA_default  # The default ca section

[ CA_default ]

dir  = ./demoCA  # Where everything is kept certs  = $dir/certs  # Where the issued certs are kept crl_dir  = $dir/crl  # Where the issued crl are kept database = $dir/index.txt # database index file. new_certs_dir = $dir/newcerts  # default place for new certs. certificate = $dir/cacert.pem  # The CA certificate serial  = $dir/serial   # The current serial number crlnumber = $dir/crlnumber # the current crl number      # must be commented out to leave a V1 CRL crl  = $dir/crl.pem   # The current CRL private_key = $dir/private/cakey.pem# The private key RANDFILE = $dir/private/.rand # private random number file x509_extensions = usr_cert  # The extentions to add to the cert name_opt  = ca_default  # Subject Name options cert_opt  = ca_default  # Certificate field options default_days = 5475   # how long to certify for default_crl_days= 30   # how long before next CRL default_md = sha512  # which md to use. preserve = no   # keep passed DN ordering policy  = policy_match

[ policy_match ] countryName  = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName  = supplied emailAddress  = optional

[ policy_anything ] countryName  = optional stateOrProvinceName = optional localityName  = optional organizationName = optional organizationalUnitName = optional commonName  = supplied emailAddress  = optional

[ req ] default_bits  = 2048 default_keyfile  = privkey.pem distinguished_name = req_distinguished_name attributes  = req_attributes x509_extensions = v3_ca # The extentions to add to the self signed cert

input_password = testpassword output_password = testpassword encrypt_key = no prompt = no string_mask = nombstr

req_extensions = v3_req  # The extensions to add to a certificate request

[ req_distinguished_name ] countryName   = NZ

stateOrProvinceName  = Auckland

localityName   = Auckland

0.organizationName  = IT Solutions 2000 Ltd

organizationalUnitName  = IT

commonName   = vsphere-web.homedns.org

emailAddress   = admin@homedns.org

[ req_attributes ]

[ usr_cert ]

basicConstraints=CA:FALSE nsComment   = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer

[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = DNS:vsphere-web.homedns.org, DNS:vcva.homedns.org, DNS:vcweb.homedns.org, DNS:vsphere-web, DNS:vcva, DNS:vcweb

[ v3_ca ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid:always,issuer:always basicConstraints = CA:true

[ crl_ext ] authorityKeyIdentifier=keyid:always,issuer:always

[ proxy_cert_ext ] basicConstraints=CA:FALSE nsComment   = “OpenSSL Generated Certificate” subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer:always proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# End of vSphere Web Client OpenSSL Configuration File

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Advertisements
  1. February 13, 2012 at 6:20 am

    Michael,

    To help simplify updating the SSL certs for the first two paths (WebUI & vCenter) in VCSA:
    /opt/vmware/etc/lighttpd/
    /etc/vmware-vpx/ssl

    You can use “vpxd_servicecfg” utility to update the certificate, take a look at this article for more details
    http://www.virtuallyghetto.com/2012/02/automating-vcenter-server-appliance.html

    vpxd_servicecfg service stop
    vpxd_servicecfg certificate change rui.crt rui.key

    • February 13, 2012 at 9:24 pm

      Thanks for the tip William. I’ll update the article. That will make things a lot easier.

  2. Wan
    February 21, 2012 at 11:29 am

    I followed the steps up to step 31 and got error: cannot conntact vsphere web client service. I started the service and re-run the command. Got same error and the service is stopped again. Please advise.

    • February 21, 2012 at 1:07 pm

      Hi Wan, Have you attempted rebooting your vCenter Server Appliance? If you can’t connect to the vSphere Web Client service then it’s possible something is wrong with the command line as you entered it. Is the vSphere Web Client service running?

      • Wan
        February 27, 2012 at 12:12 pm

        Have reboot a few times. Not good.
        Found another article on changing ssl cert. Follow through and updated cert successfully.

      • February 27, 2012 at 1:31 pm

        Hi Wan, Could you share a link to the article? What was the solution to your particular problem? Most of the problems in my experience have been due to certificates not created with the correct attributes. If you would mind sharing what you did I’d be happy to ensure that this article is updated to ensure accuracy. When I published this I had it tested in my lab and also by a couple of customers. But every environment is different and dealing with PKI and CA’s is very complex by it’s nature.

      • February 27, 2012 at 2:31 pm

        I have now included a warning in the post to use Text transfer mode. Without using Text transfer mode in WinSCP you will corrupt your certificates with additional ASCII characters. The auto mode is not sufficient. I have also included a reference to Doug Baer’s article on the same subject.

  3. March 3, 2012 at 9:24 am

    Thanks for the link Michael.

  4. Doug Baer
    March 6, 2012 at 4:02 am

    Thanks for the great article, and the backlink 🙂

    I totally agree that dealing with PKI is complicated by nature and pretty much misunderstood across the board.

  5. andy franks
    September 14, 2012 at 2:23 am

    Found that you might need to install the root certificate into the java keystore first before the server certificate. I did :
    keytool -import -trustcacerts -alias root -file root.cer -keystore /usr/lib/vmware-vsphere-client/server/config/keystore -storetype JCEKS -storepass testpassword

    HTH someone

    • September 14, 2012 at 10:48 am

      Hi Andy, Yes the root cert and any intermediate CA certs should be pre-trusted in the keystores. They also need to be trusted by the other systems in the PKI.

  6. Doug
    September 16, 2012 at 8:12 am

    Looks like the process is different for 5.1. 5.1 doesn’t have the directory /usr/lib/vmware-vpx/jre/bin/

    • September 16, 2012 at 10:50 am

      Hi Doug,

      There are a lot of changes for 5.1. I haven’t yet taken a look at it. There is a guide as part of the doc set this time around though. Rest assured if necessary i’ll write about it when I’ve had time to cover it.

      • Doug
        September 17, 2012 at 5:05 am

        It looks like I was able to get SSL for the web client working in 5.1 by using the following chain of commands:

        service vmware-vpxd stop
        service vmware-sso stop
        vpxd_servicecfg certificate change newrui.crt newrui.key
        /usr/bin/openssl pkcs12 -export -in /etc/vmware-vpx/ssl/rui.crt -inkey /etc/vmware-vpx/ssl/rui.key -name “rui” -passout pass:testpassword -out /etc/vmware-vpx/ssl/rui.pfx

        And then a reboot.


        The first time I had my new cert just named rui.crt and rui.key and it didn’t take. After renaming them to “newrui” it worked. I’m not sure if that was a fluke but if anyone else has trouble with this script, try renaming your new cert files. Also if you get an error code you can get more info by typing:

        cat /usr/sbin/vpxd_servicecfg | grep

        I’m very close to getting it working. But even though my EXSi host is configured with the same wildcard certificate (and the Windows client can access it via SSL with no popup warning), when I try to add that host via the web client I get this error:

        License file download from to vCenter Server failed due to exception: vim.fault.SSLVerifyFault.

        I’m guessing this means there is another place I need to add in my certificate.

  1. February 24, 2012 at 6:53 am
  2. August 9, 2012 at 10:51 am

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: