A Christmas present from VMware; vRealize Automation 7.0 is GA as of today !!

VMware announced a new version of vRA (vRealize Automation) during the last VMworld. And as of today vRealize Automation is available for download.

For detailed information around compatibility, release notes as well as the download link please go to the vRealize Automation 7.0 Information Center

Below a highlight list of new features and enhancements:

Streamlined and Automated Wizard-based Installation

• Introduces management agent to automate the installation of Windows components and to collect logs
• Automates the deployment of all vRealize Automation components
• Installation wizards based on deployment needs: Minimal (Express) and Enterprise (Distributed) Installations

Simplified Deployment Architecture and High Availability Configuration

• Embedded authentication service by using VMware Identity Manager
• Converged Application Services in vRealize Automation Appliance
• Reduced minimal number of appliances for HA configuration
• Automated embedded PostgreSQL clustering with manual failover
• Automated embedded vRealize Orchestrator clustering

Enhanced Authentication Service

• Integrated user interface providing a common look and feel
• Enabled multiple features by new authentication service

Simplified Blueprint Authoring for Infrastructure and Applications

• Single unified model for both machine and application blueprints and unified graphical canvas for designing machine and application blueprint with dependencies and network topology
• Software component (formerly software service in Application Services) authoring on vSphere, vCloud Air, vCloud Director, and AWS endpoints)
• Extend or define external integrations in the canvas by using XaaS (formerly Advanced Service Design)
• Enable team collaboration and role segregation by enhancing and introducing fine-grain roles
• Blueprint as code and human-readable which can be created in editor of choice and stored in source control or import and export in the same or multiple vRealize Automation 7.0 instances
• Customer-requested machine and application blueprints provided
• Additional blueprints available on the VMware Solutions Exchange

Simplified and Enhanced NSX Support for Blueprint Authoring and Deployment

• Dynamically configure NSX Network and micro-segmentation unique for each application
• Automated connectivity to existing or on-demand networks
• Micro-segmentation for application stack isolation
• Automated security policy enforcement by using NSX security policies, groups, and tags
• On-demand dedicated NSX load balancer

Enhanced Cloud Support for vCloud Air and AWS

• Software component authoring for vCloud Air, vCloud Director, and Amazon AWS
• Simplified blueprint authoring for vCloud Air and vCloud Director
• Improved vCloud Air endpoint configuration
• Optional proxy configuration

vRealize Orchestrator 7 New Features

• Introduce vRealize Orchestrator Control Center for easy monitoring and troubleshooting
• Significant Smart Client improvements including Workflow tagging UI, Client reconnect options and enhanced search capabilities
• vSphere 6.X  vAPI endpoint support

Updates failing on VSAN hosts

A while back one of my customers ran into a issue when they wanted to install Update 1 for ESXi 6.0. Initially the ESXi hosts updated as expected, except five hosts.
For these five hosts, the first difference was that the hypervisor is installed on rack servers instead of blade servers. Main reason for this was, these hosts needed to accommodate local storage. This local storage is used for VMware Virtual SAN.
The five hosts where in the same VSAN cluster, and this cluster is used as management cluster for the customers entire vSphere environment.
 So these five hosts needed to get updated with Update 1 for ESXi 6.0, but VUM (VMware Update Manager) failed to install this update with a somewhat strange error message. With the help of the VMware Knowledge Base it became clear it had something to do with staging the Update before installing it on the ESXi host.
Because this customer runs ESXi from a USB flash device, the scratch location is redirected to shared storage. This configuration is similar for both the blade and rack servers. My first thought was that there was something wrong with this redirection for these specific five hosts. But after reviewing the advanced settings and verifying the time and date stamps on the various logfiles of the hosts within the .locker folders located on shared, all looked fine.
To my believe the update and patch staging location moved along with the logfile location when the scratch location was changed. A excellent blog post on logfile redirection and VSAN considerations when booting of a USB flash device is written by Cormac Hogan.
So what do VSAN, ESXi hosts booting from a USB flash device and scratch folder redirection have to do with failing updates? I will come to that, please bear with me as first give you some background on the intended use and deployment method of the hosts concerned.
Because these hosts are used as resources to run the management cluster on, these hosts where the first ESXi hosts to be deployed within the customers data center. No shared storage was available at the time of deployment. This imposes a challenge, to be able to use VSAN you need vCenter and to be able to deploy vCenter (as appliance or Windows based) you need storage accessible by one or more ESXi host(s).
A solution is to bootstrap vCenter on a single VSAN node, yes a single ESXi hosts that runs VSAN with only it's own local storage!.
If you want more information on how to bootstrap vCenter on a single VSAN node, please have a look at this 2-part blog post of William Lam on his VirtuallyGhetto blog
With the use of bootstrapping the vCenter onto a single ESXi host using it's local storage to create the VSAN datastore I could build the VSAN cluster and add the remaining four ESXi hosts to complete the cluster.
When the shared NFS storage became available later on in the project the scratch folder was redirected just like with all of the customers other ESXi hosts.
And here is the catch, the time the ESXi hosts have been running without the redirection, they have been logging to the local flash device. Usually this is not a big issue, other then the risks mentioned in Cormacs' blog post. But when you use VSAN, there will be additional log or trace files written (vsantraces). And in this customers case there were also VSAN Observer files written to the local flash device, VSAN Observer is a tool used to monitor and capture performance statistics, originally only used by the VMware VSAN engineering team. More information on VSAN Observer can be found here.
Vsantrace files can grow quickly up to 500 Mb and VSAN Observer trace files even larger, as I explained previously the scratch folder redirection was done some time (days) after the VSAN cluster became operational. When the redirection is done, the various trace files that are on the local flash device are NOT removed, these files do take up a considerable amount of space. In fact they take up so much space that there is not enough space left for staging Update 1 for ESXi 6.0 on the host.
Manually removing the old VSAN related trace files from the local flash devices was what solved the VUM issue. After the files where deleted the remediation of the ESXi hosts using VSAN ran without any issue.

Installing vSphere 6 gotcha

The last couple of weeks I have been involved in the deployment of a vSphere 6.0 environment. This was my first vSphere 6.0 customer deployment.
Most of the implementation plan I wrote and as this was my first customer deployment I had to do some rework on my default implementation, which I have used on vSphere 5.5 implementations. To write this implementation plan I used the official VMware installation and configuration documentation as guideline. At the time of writing all implementation steps seamed logical, but when we came to the actual implementation we ran into a issue when we tried to add Identity Sources to the PSC (Platform Service Controller).
This customer vSphere environment is a large, multi tenant and multi site environment and for this reason together with the need for flexibility and scalability a design choice was made to use external PSC's. Also during the design sessions with the customer the decision had been made to use the vCSA (vCenter Server Appliance) instead of the up until vSphere 5.5 generally chose Windows based vCenter.
The issue we ran in to has to do with in which order you go to the configuration steps after the PSC is deployed. At some point you want to be able to join a PSC to the Active Directory domain to be able to use AD integrated authentication. When using external PSC's like in this particular case you don't join vCenter (vCSA) to the domain but the PSC. The reason for this is simple, SSO (Single Sign On) and Authentication services are provided by the PSC not vCenter.
The implementation plan followed the exact same steps for this part of the configuration as the VMware documentation and we followed the each en every step as they where documented in the implementation plan, but for some reason we still got a error message.
After verifying that I had used the most recent version of the VMware documentation I did a quick check on the VMware communities to see if anyone else had ran into the same issue.
There was a post dated March 27 2015 called "Wrong information in VMware 6.0 documentation vCSA 6.0" which described the exact issue and the writer came to the same conclusion that the VMware documentation has the configuration steps in the wrong order.
This community post was posted two and a half months ago, clearly VMware hadn't been able to update their online documentation.
Below you will find in short the correct steps and order to join a PSC (internal or external) to a Active Directory domain followed by adding Identity Sources to the PSC. For more details please have a look at the community post.

1. Use the vSphere Web Client to log in as administrator@your_domain_name to the vCenter Server instance.
2. Under Deployment, click System Configuration.
3. Under System Configuration, click Nodes.
4. Under Nodes, select a node and click the Manage tab.
5. Under Advanced, select Active Directory, and click Join.
6. Type the Active Directory details (use the administrator@your_domain_name syntax for the user).
7. Reboot the appliance, after the reboot login as described in step 1.
8. Navigate to Administration > Single Sign-On > Configuration.
9. On the Identity Sources tab, click the Add Identity Source icon.
10. Add the Active Directory domain as an identity source, enter the identity source settings, and click OK.
11. Finished, you can now continue with the rest of the configuration like adding AD security groups to the roles defined in Global Permissions.

I am sure that VMware eventually will update their online documentation. In the mean time, if you follow the steps described above the configuration should be easy. Alternatively you still could follow the steps as described in the VMware documentation, but you will run into this issue. The error message has gotten a link to the correct webpage where you can first join your PSC to a Active Directory domain. After a reboot you can then restart your configuration. Although the second option will take more time the end result will be the same.