Infrastructure as Code

"Infrastructure as Code" or IaC, when you search for this term online you will get loads of information and all this information has a different interpretation of Infrastructure as Code. Even if you search for it in the VMware context it will deliver you loads of different VMware blog posts and product related web pages.
Although the all the different interpretations, there are some things all information has in common. They all have the same basic concept, have IT teams to treat IT infrastructure the same way application developers manage / develop their applications, so treat it as code. Human readable software code that is. This concept is also known as DevOps. And most of the information will also tie this concept to Continuous Integration / Continuous Delivery practises.

If you are looking for additional information on IaC, there are two Forrester reports that are very relevant and therefore are referenced frequently when writing about IaC:

• How A Sysadmin Becomes A Developer written by Chris Gardner and Robert Stroud
• Lead the I&O Software Revolution with Infrastructure-As-Code written by Chris Gardner and Richard Fichera

I mentioned that there is an abundance of information on IaC and most Vendors have multiple product or solution pitches around it. In this blog post I'm not going to focus on the well known solutions, I would like to provide some information on a less known solution, which has the obvious name Infrastructure as Code.

This particular solution is a offering from VMware's Professional Services Center of Excellence (PSCoE) and it basically delivers an Integrated Development Environment (IDE) focusing on getting vRealize Orchestrator (vRO) coding to "the next level". In other words provide a toolchain that provides the tools a Developer needs, but is missing from the standard vRO solution. So what features does this PSCoE IaC solution bring:

• Work Easy – Powerful tools for Management of vRA/vRO content
• Work Together – Share Code, Validate, Do Peer Review
• Quality Control – Quality Gates, Unit Testing
• Improve Efficiency – Solution Modularization, Dependency Management, Versioning, Parallel Releases
• Release – Frequent Releases, Policy Gates, Sites Distribution
• Step Back in Time – Rollback releases, Troubleshooting

As you can see for yourself the PSCoE IaC solution is focused on vRealize Automation, let's look in a bit more detail how a vRealize Automation environment looks when the PSCoE IaC solution is deployed.

Currently this PSCoE IaC solution is only available as an complete PSCoE offering, this offering includes the Installation and configuration of the required software components, Onboarding training for in-house developers and Initial guidance.

The solution is not a one-size-fits-all solution, there are customer / environment specifics that can influence the solution. If you are interested in learning more about this solution offering contact PSCoE or find them during the upcoming VMworld events.

Lastly a small word of caution, this PSCoE IaC solution is not a good fit for all vRealize Automation customers. In general customers that already have adopted the DevOps way of working and/or have in-house (Java) developers would benefit from this solution, most of the time this will be the larger enterprises that already have a (Private) Cloud Solution in Production.

vRealize Orchestrator (vRO) licensing pitfall

vRO is a powerful tool used when you want to automate repetitive tasks on your vSphere environment. It has various plugins which bring integration with other VMware products as well as integration with third party hardware and software solutions.
vRO is a workflow based tool, besides the default workflows you can also create custom workflows. These workflows are stored in the vRO database together with the configuration data, depending on the use case you can choose either between a embedded database that is based on vPostgress or you can choose to use a external database solution based MS SQL or Oracle.

About the use cases which could use the embedded database, VMware documentation is not really clear on this. Because the Install and Configure documentation on vRO version 6.0.1 states that the embedded database should only be used for testing or PoC purposes. But the Install and Configure documentation on vRO version 6.0.3 states that default database option is suitable for small- or medium-scale environments. What is clear is that when you use the embedded database you cannot set up vRO to work in cluster mode, or change any licenses and the server certificate by using the vRO configuration interface.

So why am I writing about the database choice of vRO, well recently at a customer which uses vRO to automate tasks we ran into a operational issue when nobody could logon to the newly deployed vRO instance. It became clear that this vRO instances had been running on the default evaluation license, which stops working after its 90 day evaluation / grace period.
The solution would be simple, just login to the vRO configuration page and update the license key. But as documented in the Install and Configure documentation, when the embedded database is used you cannot update the license key from the configuration page. The only way to do this is to run the license update workflow from within vRO itself.
This was not possible because logon the the vRO client did not work because of the expired evaluation license. A solution to this "chicken and egg" problem was not easily found.

So how do you solve this problem, I first need to mention that I did not find this solution. All credits for the solution described below go to Martijn Went (@MartijnW01).

As mentioned earlier, when your vRO license is expired you cannot logon the the vRO client and run workflows. But you can logon to the vRO configuration page, after you first start the configuration service.

• Start the service and logon to the configuration page
• Go to the Database sub menu
• Change the database type to anything other than embedded, this will make License sub menu become available.

• Go to the License sub menu and go to the vCenter license tab and enter the details about the vCenter that needs to be used to license this vRO instance. Leave the port (443) and path (/sdk) default. Enter vCenter credentials and apply changes.
• After the connection to the vCenter is made the license will be verified, check the license details on the vCenter license tab you are on.
• Go back to the Database sub menu and change the database back to embedded
• Now you should be able to start the vRO service again.
• After the service is started, logon the the vRO client and run the workflow to license vRO to the vCenter license.

Now all should be operational again, the only thing that we had to do to be fully operational was reconnecting vRO to all vCenters within the environment. Not sure if this is related to the vRO configuration actions, but for some reason the vRO instance lost connection to the vCenters.

I'm sure this is not a supported solution, but VMware GSS did not have a solution on hand when they where contacted regarding this issue. The solution they found that might work, was to manually edit the database (replace the evaluation license key).

vCenter Orchestrator loses VM networks after host renaming

Last week I was asked to have a look at a VCO workflow issue. There was a issue with workflows used to deploy VM's, the workflows would fail on specific hosts. One of the customers' VMware administrators found that the workflow stopped at the point where a VM had to be linked to a specific VM port group.
This happened with any selected VM port group (VLAN) available within the workflow, these workflows have a automatic host selection a manual selection can also be made. After running workflows with manual host selection some hosts were found which completed the workflow successfully.
When verifying what the difference was between the hosts, it became clear that the hosts that failed the workflow where recently renamed.
The customer uses a dVswitch for all network traffic across all hosts within the HA-clusters. During the renaming you have to disconnect the ESXi host from vCenter and re-connect after the renaming,  a PowerCLI script was used to automate the renaming process, a similar script can be found here.
During the renaming there had been a issue with the hosts upon reconnecting to vCenter. After renaming hosts reconnected with a dvSwitch error message, to get rid of this error you manually re-add the host to the dvSwitch. After all the hosts network looked OK, nevertheless this was a good reason to take a better look at the network configuration of those renamed hosts.
One detail which stood out, was the color of the dvUplink interfaces. When all is fine they are coloured green, but when for instance the Physical NIC used by the Uplink is disconnected the color turns to white as shown in the picture below for dvUplink2.

 Now with the renamed hosts it was not one dvUplink, but all 4 dvUplinks where coloured white. Strangely enough the VM's hosted on these hosts had a fully functional network connection, so as expected none of the physical NIC's was disconnected.

One of the VMware administrators tried to get all dvUplinks "green"  again by simply removing and re-adding the vmmic from the dvUplink, this seemed to work all dvUplinks came back "green" again. Unfortunately the Orchestrator workflow persisted, after the actions above and none of the VMware administrators (me included) had any idea's on how to solve this issue so a support case was opened with GSS.
After the usual "please upload logfiles" steps, the problem was quickly solved during a Webex session. The solution was to force update the dvSwitch configuration across all hosts connected to this dvSwitch.
So how to you push the configuration or how do you forcefully update the configuration on ESXi hosts, simple just add a temporary new dvPortgroup to the dvSwitch. By adding a dvPortgroup all connected ESXi hosts get a updated dvSwitch configuration.
This solved the Orchestrator workflow issues finally, I can imagine that this updating of the dvSwitch configuration could also be of help in other dvSwitch "out of sync" kind of issues.
I will be trying next time I run into such a issue.

Database redundancy for your vCenter database(s)

The most important database within a vSphere environment is the vCenter database without a doubt. VMware therefore has enclosed detailed instructions on how to setup and configure this database, they have a guide for every supported database type. Recently I ran into a situation which made me believe that VMware "forgot" some details on this database configuration guide, at least when you have your vCenter database running on Oracle.
A customer has chosen to put their vCenter database on Oracle as this was their preferred database knowledge wise. And they set it up to also be resilient, the way they achieved this was by having a active and a standby database placed on 2 different database server in separated datacenters. To me it looked like a very solid solution. On the vCenter part they modified the TNSNames.ora in such a way it now included 2 database server addresses and also contained the parameters for connect-time failover and load balancing.
By doing this they made sure that vCenter could (almost) always connect to one of the two database servers, it would simply do a failover when the connection time would expire. In this case the failover would not have been quick enough to keep vCenter up-and-running but it would need a reboot (or at least a restart of services) to get connection again. But this would not affect the running VMs at all.
For maintenance purposes to the database servers, we had to switch from the active server to the backup server. As this was a planned action, we could first gracefully stop the vCenter services and after switch to the standby database server. After the switch all vCenter services were started again and vCenter went up-and-running like it supposed to do.
One issue that occurred during this database server switch was that VMware Orchestrator, which was installed on a separate server stopped working, logging all kinds of database related error's. With a quick look at the database configuration of Orchestrator I remembered that it could not cope with multiple database server addresses and was set to connect to the database server that now had become the standby. By changing the database server and starting the Orchestrator services again this problem was solved.
At least until the next day when I took a look at the vCenter Operations dashboard and found that the health of vCenter was 0

When I looked into more detail on what caused this I found VMware vCenter Storage Monitoring Service - Service initalization failed on only thing I found that could link this alert to the database failover was the timestamp, it was recorded right at the same time the failover had happened.

Not really knowing where to start investigating on the vCenter server, I first tried to find some information on the VMware KB and the first article that came up described the exact same error message. When reading kb2016472 I quickly found confirmation that this issue was related to the database failover although it refers to vCenter 4.X and 5.0 with the use of a SQL database instead of vCenter 5.1 / Oracle database.
It appears that this vCenter Storage Monitoring Service does not use the TNSNames.ora for the database connection, it has it's own configuration / connection file called vcdb.properties. This file has only the first of the two database server addresses.
Thru the information in the KB article I knew what to change to get the connection set to the backup database server, and after a restart of the vCenter Server service the vCenter Storage Monitoring Service initialized ok and started without any error.
So my conclusion is that even when you have redundancy or failover setup on vCenter database level, there are still some vCenter related products and services that need some manual action to continue to work in case of a (planned) database failover.

VMware vSphere certificates just got a whole lot easier....

When deploying or upgrading to vSphere 5.1 in a customers environment which has the prerequisite to use 3rd party signed certificates, most VMware consultants and admins already know that this is not the easiest part to manage. In fact it is down right pain to setup the vSphere environment with 3rd party certificates.
VMware has a guide to help you with the process of doing the certificate replacements, the document is called "Replacing Default vCenter 5.1 and ESXi Certificates".
If attempting to replace a certificate please do consult this guide and keep in mind that it is not going to be an easy task.
But wait VMware has taken their vCenter Certificate Automation Tool out of Beta and made it publicly available with the introduction of version 1.0
Details on the features you can find on the VMware Blog "VMware Support Insider".
With version 1.0 certificates of the following VMware products can be changed / update with the use of this tool:

1. vCenter Single Sign On
2. vCenter server
3. vCenter Inventory Service
4. vCenter Log Browser
5. vSphere Web Client
6. VMware Update Manager (VUM)
7. vCenter Orchestrator (VCO)

Hopefully in the next releases they will add more products which can be managed by this tool, like the ESXi hypervisor itself, vCloud Director, Site Recovery Manager and vCenter operations.

For details on deploying and using the tool, please have look at the KB article VMware has published under kb2041600