Infrastructure as Code

"Infrastructure as Code" or IaC, when you search for this term online you will get loads of information and all this information has a different interpretation of Infrastructure as Code. Even if you search for it in the VMware context it will deliver you loads of different VMware blog posts and product related web pages.
Although the all the different interpretations, there are some things all information has in common. They all have the same basic concept, have IT teams to treat IT infrastructure the same way application developers manage / develop their applications, so treat it as code. Human readable software code that is. This concept is also known as DevOps. And most of the information will also tie this concept to Continuous Integration / Continuous Delivery practises.

If you are looking for additional information on IaC, there are two Forrester reports that are very relevant and therefore are referenced frequently when writing about IaC:

• How A Sysadmin Becomes A Developer written by Chris Gardner and Robert Stroud
• Lead the I&O Software Revolution with Infrastructure-As-Code written by Chris Gardner and Richard Fichera

I mentioned that there is an abundance of information on IaC and most Vendors have multiple product or solution pitches around it. In this blog post I'm not going to focus on the well known solutions, I would like to provide some information on a less known solution, which has the obvious name Infrastructure as Code.

This particular solution is a offering from VMware's Professional Services Center of Excellence (PSCoE) and it basically delivers an Integrated Development Environment (IDE) focusing on getting vRealize Orchestrator (vRO) coding to "the next level". In other words provide a toolchain that provides the tools a Developer needs, but is missing from the standard vRO solution. So what features does this PSCoE IaC solution bring:

• Work Easy – Powerful tools for Management of vRA/vRO content
• Work Together – Share Code, Validate, Do Peer Review
• Quality Control – Quality Gates, Unit Testing
• Improve Efficiency – Solution Modularization, Dependency Management, Versioning, Parallel Releases
• Release – Frequent Releases, Policy Gates, Sites Distribution
• Step Back in Time – Rollback releases, Troubleshooting

As you can see for yourself the PSCoE IaC solution is focused on vRealize Automation, let's look in a bit more detail how a vRealize Automation environment looks when the PSCoE IaC solution is deployed.

Currently this PSCoE IaC solution is only available as an complete PSCoE offering, this offering includes the Installation and configuration of the required software components, Onboarding training for in-house developers and Initial guidance.

The solution is not a one-size-fits-all solution, there are customer / environment specifics that can influence the solution. If you are interested in learning more about this solution offering contact PSCoE or find them during the upcoming VMworld events.

Lastly a small word of caution, this PSCoE IaC solution is not a good fit for all vRealize Automation customers. In general customers that already have adopted the DevOps way of working and/or have in-house (Java) developers would benefit from this solution, most of the time this will be the larger enterprises that already have a (Private) Cloud Solution in Production.

Meet the VMworld 2018 Content Catalog

I guess most of us know the moment when the VMworld Content Catalog becomes available. You want search it and get into the sessions as soon as possible, because the more populair the sessions are the quicker they are full and you will be put on the waiting list.
But what if you are new to VMworld, how do you select the sessions that are relevant and of interest to you? Of course there are blog posts out there that will list their favorite sessions, most of them are the obvious sessions presented by known speakers. These listings will get you a good base, for the rest you need to search on your own.
But what type of sessions do you attend?
In previous years VMworld sessions have been categorized in a couple of categories, from Business to Deep technical. With the 2018 edition of VMworld a new way of using / consuming the Content Catalog is being introduced, it's now role based. So you select a role description that best covers your actual professional position, this way the sessions that are (pre-)filtered hopefully align with your goals, this way you should get the most out of your VMworld visit.

If you are afraid you're going to miss out on sessions because they are filtered out based on your role, don't worry there is a possibility to select multiple roles.

For a detailed description on the available roles please check out this VMworld blog post.

VMworld 2018 Call for Papers!!

So you're a VMworld veteran and feel the time is right to submit your own VMworld session, don't wait any longer!
The VMworld 2018 Call for Papers is now open, you can submit your proposal until March 13th. This call for papers is for VMworld US, VMworld EU or both.

If you want more information please checkout the VMworld submission guidelines and also have a look at the FAQ.

Looking for information around submitting a VMworld proposal that was not covered in this blog post, please have a look at this VMworld blog post.

And remember "Possible begins with You"!

NLVMUG Usercon 2018 Call for papers

For the last couple of years the NLVMUG has probably been the biggest thing after VMworld (Europe) if you live in The Netherlands or Belgium.
If you are thinking about presenting, why not give it a try at a VMUG conference, the NLVMUG to be more precise? For less experienced presenters there are Lightning Talks, present to a smaller audience with a time slot of 20 minutes. Or if you want to have a interactive architecture discussion, go for a whiteboard session.

Call for papers is open from October 26 2017 till January 26 2018. Submitting your paper can be done through this link.
If you're not ready to present by yourself, but want to be part of the NLVMUG Usercon on March 20 2018 then register through this link.

The location for this years edition is the same as last year, 1931 Event Center in Den Bosch
And as last year, you can expect next to the well known solution exchange, workshops and breakout sessions, also Lightning Talks, Whiteboard sessions and Group Discussions this year.

For those of you who are attending this NLVMUG Usercon, see you March 20th 2018!

VMworld US and Europe 2018 Dates and Locations

VMware confirmed the dates and locations of the 2018 edition of VMworld in both US and Europe.


VMworld US event:

Mandalay Bay Convention Center in Las Vegas
August 26 - August 30


VMworld Europe event:

Fira Gran Via in Barcelona
November 5 - November 8*


* Please note that the VMworld Europe event is about two months later then this year. So if you already put a blocker in your calendar based on this years dates, make sure to update this ASAP!


I don't have any information why the VMworld Europe event is in November this year. It does not matter for the actual event, however there is less chance that you can enjoy summer days in Barcelona.

NLVMUG Usercon 2017

The NLVMUG is probably the biggest thing after VMworld (Europe) if you live in The Netherlands or Belgium and it has been the biggest Usercon in the world for a couple of years in row now.

The location for this years edition is the same as last year, 1931 Event Center in Den Bosch
The way the sessions are scheduled has changed this year, all to be able to have more sessions during the event. You can expect next to the well known solution exchange, workshops and breakout sessions, also Lightning Talks and Group Discussions this year.

Registration is still possible through this link. And the full event agenda is here!


For those of you who are attending this NLVMUG Usercon, see you March 16th 2017!

Adding an Tenant Administrator in Multi Tenant vRealize Automation environment gotcha!

While configuring the vRealize Operations Manager (vROps) Managment Pack for vRealize Automation (vRA) I ran into a somewhat strange issue.
The vRA environement was multi-tenant, having one additional tenant (Customer-1) next to the default tenant and to have the vROps Management Pack collect data from both tenants we need to have the same user account across both tenants. In my case an AD user account (service account) would be the account to manage this.
The AD account needs to have to correct vRA roles in each of the tenants:

• Infrastructure Administrator
• Tenant Administrator
• Fabric Group Administrator
• Software Architect role (vRA 7.0 and later)

Adding this account to the required roles is easy and went without any issue on the default tenant. When doing the same on the Customer-1 tenant adding the account (svc_vra in my screenshot) to the Infrastructure and Tenant Administrator roles I ran into an issue, to complete the action you need to click "Finish".

When I did this it returned an error message and failed to complete successfully:
Another user has already modified the data. Please reload the form and try again.

The only error that could be found in the vRA logging was the following:
it needs both the  vRA domain user add across all tenants, one tentant with error:

com.vmware.vcac.authentication.service.sso.horizon.HorizonTenantManagement.updateUserAttributeDefinitions:265 - Updating user attribute definitions in the tenant 'Customer-1'...
2016-11-29 08:28:33,186 vcac: [component="cafe:identity" priority="ERROR" thread="tomcat-http--32" tenant="vsphere.local" context="7kprPPeo" parent="" token="7kprPPeo"] com.vmware.vcac.platform.rest.client.error.ResponseErrorHandler.handleRestError:113 - [Rest Error]: {Status code: 409}, {Error code: 7} , {Error Source: null}, {Error Msg: Duplicate user attribute definition "manager" for org.}, {System Msg: vidm.userattributedefinition.duplicate}
2016-11-29 08:28:33,189 vcac: [component="cafe:identity" priority="ERROR" thread="tomcat-http--32" tenant="vsphere.local" context="7kprPPeo" parent="" token="7kprPPeo"] com.vmware.vcac.platform.service.rest.resolver.ApplicationExceptionHandler.handleRestException:610 - [Rest Error]: {Status code: 409}, {Error code: 7} , {Error Source: null}, {Error Msg: Duplicate user attribute definition "manager" for org.}, {System Msg: vidm.userattributedefinition.duplicate}
Reffering to AD user attribute "manager" which is additionally added to the default user attributes:

The error message you see in the partial logging above refers to a duplicate user attribute "manager" screenshot, this user attribute is not a default user attribute but additionally added in each of the tenants. Now having this attribute in multiple tenants looks like the cause of my error.
To be sure I removed the user attribute "manager" from the user attribute in the Customer-1 tenant.

After the user attribute "manager"  was removed I could without any issue add the account to the Infrastructure and Tenant Administrator roles.

Note: Don't forget to add the user attribute "manager" back to the user attribute configuration of the tenant.

VMworld US and Europe 2017 Dates and Locations

VMware confirmed the dates and locations of the 2017 edition of VMworld in both US and Europe.

VMworld US 2017:

 Mandalay Bay Convention Center in Las Vegas

August 27 - August 31

VMworld Europe:

 Fira Gran Via in Barcelona

September 11 - September 14*

* Please note that the VMworld Europe event is about a month earlier compared to the event dates of previous years. So if you already put a blocker in your calendar, make sure to update this ASAP!

I don't have any information why VMworld has advanced the Europe event, but now the event is closer to the US event and the chances are that you can still enjoy real summer days in Spain are a bit higher.

NSX Load balancer advanced HTTP redirect

One of the advanced features of NSX for vSphere is Load Balancing. The NSX for vSphere load balancer enables network traffic to follow multiple paths to a specific destination. It distributes incoming service requests according the selected load balancing algorithm among multiple servers in such a way that the load distribution is transparent to users. The NSX load balancer provides load balancing up to Layer 7.

When configuring a NSX load balancer you will need to map either an external, or public, IP address to a set of internal servers for load balancing. The load balancer accepts TCP, HTTP, or HTTPS requests on the external IP address and decides which internal server to use.

You create an application profile to define the behaviour of a particular type of network traffic. You also create a server pool consisting of backend server members and associate a service monitor with the pool to manage and share the backend servers flexibly and efficiently. The service monitor, monitors health of the server pool members according to a set of defined health check parameters.

During the creation of the application profile you can set up HTTP re-direct, by simply specifying the URL to which you want HTTP traffic to be redirected. For example, you can direct traffic from http://website.com to https://website.com

This kind of redirect works in most standard cases, but redirecting a login page which uses cookies will probably not work. For this you will need to make use of the Application Rule feature, which basically let's you create a rule for the traffic that you specific in this rule. A rule is actually a script, based on the HA-proxy scripting set.

VMware documentation around Application Rules is not extensive, you can find a few examples in the NSX Administration Guide. But if you need more details, most documentation will refer to the HA-proxy documentation. Although the NSX load balancer is based up on HA-proxy the ability to use HA-proxy scripting as a Application Rule is somewhat limited.

Lately I found a customer use case where HTTP redirection was required and where the basic HTTP auto redirection feature proofed not to be a valid solution. The use case for me is when a customer has both vROps (vRealize Operations Manager) and LogInsight. When you integrate these two solutions you get the option to search for syslog events in LogInsight that correlate to the object, error or warning presented in vROps.

Like in the example above when you click "Search for VM logs in vRealize Log Insight" the link will take you to the LogInsight web interface complete with the required filters to only show you the syslog events belonging to that particular object. To do so the link actually contains a search string or query. And good to know, the link is a HTTP link not a HTTPS link!

If you run vROps and LogInsight in an Enterprise scale, you will probably have multiple instances of each solution behind a load balancer, and if this load balancer is an NSX load balancer and you use the basic auto redirection feature for the VIP(s) of LogInsight. Then clicking the link will only take you to the default home screen of the LogInsight web interface, why?
Well the basic HTTP redirection does not take in to account the entire URL, but only FQDN that you have set up on the VIP. So here goes the nice integration between the two solutions....

The solution to keep the integration working between vROps and LogInsight while using NSX load balancers is fairly easy, you "only" need to make use of an Application Role and don't use the auto HTTP redirecting feature.
If you already had a look at the VMware Application Rule scripts or the HA-proxy scripting, you might find it a bit challenging to start with. But in fact not all scripts have to be complex, for the problem discussed in this blog post the script isn't complex or long.

# Redirect all HTTP requests to same URI but HTTPS redirect scheme 
Redirect scheme https if !{ ssl_fc }

As you can see, the solution only takes one single line (and maybe an additional line for description).
This solution will do a HTTP to HTTPS redirection which will respect the entire URL.

All (good) things come to an end. Goodbye C# client

Yesterday VMware announced the retirement of the C# client or vSphere Client for Windows. It will not be available for the next version of vSphere.
The next version of vSphere will get a HTML5 based Web Client, this will not only replace the C# client but will also replace the current Flash based Web Client. Although both web clients will coexist for some time to give (3rd party) plugins time to move from the Flash based Web Client to the new HTML5 based Web Client.
VMware states that the HTML5 Web Client will bring a great user experience. Currently you can already try the new HTML5 "look and feel" when you run ESXi 6.0 Update 2, this includes the Embedded Host Client which started of as a VMWare Lab Fling but made it into the Update 2 release.
And if you want to take it even a step further in your lab (VMware Lab Flings should not be used in a production environment), you can go ahead and get the vSphere HTML5 Web Client Fling!

At this time, it is not sure if the GA version of the HTML5 Web Client is going to look the same as the Fling does at the moment. But for the moment I really like the clean and basic look (called Clarity) of the Fling.

One other important thing to mention, VMware will try to stay on the same support model (supporting the one it’s released with, and one version back for upgrade transitioning) for the new HTML5 Web Client. Due to the amount of changes to the backend API it is not sure if they will be able to make this actually happen.

NSX syslog caveat

When you run a VMware based SDDC solution (in a lab or at a customer site). Which includes VMware NSX for network virtualization, it will probably also include VMware vRealize Operations (vROps) for monitoring and capacity planning. On top of that there is a pretty good change that VMware LogInsight is also included for log management and analytics.

Recently I ran into a syslog issue, using the combination of the three solutions mentioned above. When going through the final checks before handing over to the customer I saw that the Syslog server setting of NSX manager was incorrect, it showed the FQDN of one of the LogInsight nodes instead of the Load Balancer FQDN that is in front of the LogInsight nodes.
This Load Balancer was placed in front of the LogInsight nodes at a later stage during the project, so at the time of configuring syslog server in NSX manager UI one of the LogInsight nodes was used.
Therefore I thought at first it was a configuration change that had been forgotten, so I updated the Syslog server field in NSX manager UI and checked if LogInsight was still receiving syslog information from NSX manager. All looked just fine, it was still receiving syslog information and the NSX manager UI displayed the correct FQDN in the Syslog server field after refreshing the browser.

But when the customer validated this Syslog server field the next day, it was again pointing to one of the LogInsight nodes instead of the Load Balancer FQDN!
To be sure I again updated the Syslog server field and after rebooted the NSX manager appliance, after the reboot the Syslog server field still had the Load Balancer FQDN. A final check was to verify the field at Closing Of Business that day. At the end of the day, I checked the Syslog server field and it had again reverted back to the FQDN of one of the LogInsight nodes.

After doing some searching I found someone that had a similar issue with Syslog settings on ESXi hosts and this information pointed me into the direction of the vROps Management Pack for NSX-v. Apparently with the Management Pack installed, vROps pushes monitoring configurations instead of only just receiving monitoring data.
At the time vROps was using version 2.0 of the Management Pack which is fully compatible with the used NSX version 6.1.5.
Now I'm no vROps expert, but apparently there is an checkbox that you can check when installing the vROps Management Pack "Enable LogInsight Integration". The information I found also described a "solution" to the problem I was having, basically you can solve it by deselecting the LogInsight Integration checkbox. This supposed to solve the issue, but also disables the vROps and LogInsight integration for NSX. And the checkbox solution will only work if you update to version 3.x of the vROps Management Pack for NSX-v. Besides this, this is not what you want if you ask me, there is a reason why you are using vROps and LogInsight inside your environment right?!

The way that we (I got help from a LogInsight expert) got this fixed, is the following. Update the vROps Management Pack for NSX-v to version 3.x, it's probably not needed for this fix but it fixes a whole lot of other issues and future proofs vROps monitoring NSX-v by being able to distinct local from universal logical switches (please see release notes for full details on the Management Pack page).
After the upgrade, make sure that the LogInsight Integration checkbox is enabled.
Then go to the vROps web UI Home and select Administration.

After go to Inventory Explorer.

When you're in the Inventory Explorer you will have 2 panes, the left pane will have kind of a index and the right pane will have a lot of items. To get to the correct items you need to change you probably want to use the filter box on the top right within the right pane to filter only LogInsight related items.

For me there were two items I needed to change, this will probably be the same for other environments. The first Item that needs to be changed is named "Log Insight Server+FQDN", highlight it and then select the pencil icon to edit it.

Within the item you need to update the "HOST" field, currently it will reflect the wrong value / FQDN. Just update it so it reflects the correct FQDN and press OK.

The other item I needed to change is named "Log Insight Server Authentication+FQDN", again highlight it and select the pencil icon to edit it.

And again within the item you need to update the "HOST" field, currently it will reflect the wrong value / FQDN. Just update it so it reflects the correct FQDN and press OK.

That's it, it you give it some time (anywhere between one to two hours) and you will see that the Syslog server field within NSX manager is updated and now shows the correct FQDN.

VMworld 2016 public session voting

The content catalog for the upcoming VMworld 2016 in Las Vegas is live!
Session Voting is open to everyone. The only thing you need is a vmworld.com account to be able to participate. If you do not have a vmworld.com account, you can set one up for free.

So make sure you have a look at the content catalog, there are over 1500 submissions to vote for. You're vote does make a difference, it not only impact internal committee decisions, the VMworld 2016 program reserves space on the agenda (up to 5%) for “Customer Choice” Sessions.
Sessions with the most public votes will be guaranteed a spot for VMworld 2016.

And last but not least, if you are voting please take the following sessions in to consideration:

•  A session submitted by a colleague and myself

Security as a Service inside Software-Defined Data Center with VMware NSX, Palo Alto Networks and Network Micro-Segmentation. A Technical Deep Dive. [9062]

• Sessions submitted on a similar topic but from the business point of view

Security as a Service inside Software-Defined Data Center with VMware NSX, Palo Alto Networks and Network Micro-Segmentation. A Solution Deep Dive. [8811]

Customer Case Study: Empowering DevOps with a Fully Automated Prevention Focused Security Posture. [9212]

Thanks in advance for your time!

VMware announcements for 2016

Last year in January VMware launched vSphere 6, this year February is going to be the month for VMware's announcements for 2016.
This time around VMware has it's announcements divided in to two different tracks, each track is available in three different time zones. Track 1 is going to be all about EUC (End User Computing) and mobility, while track 2 is all about CMP (Cloud Management Platform) and HCI (Hyper-Converged Infrastructure).

You need to register to attend the launch event, so please do so here and find out the latest news.
The news will be brought to you by; Pat Gelsinger, Sanjay Poonen and Raghu Raghuram. On registration site you can also find more details and when each track is live in your region.

A Christmas present from VMware; vRealize Automation 7.0 is GA as of today !!

VMware announced a new version of vRA (vRealize Automation) during the last VMworld. And as of today vRealize Automation is available for download.

For detailed information around compatibility, release notes as well as the download link please go to the vRealize Automation 7.0 Information Center

Below a highlight list of new features and enhancements:

Streamlined and Automated Wizard-based Installation

• Introduces management agent to automate the installation of Windows components and to collect logs
• Automates the deployment of all vRealize Automation components
• Installation wizards based on deployment needs: Minimal (Express) and Enterprise (Distributed) Installations

Simplified Deployment Architecture and High Availability Configuration

• Embedded authentication service by using VMware Identity Manager
• Converged Application Services in vRealize Automation Appliance
• Reduced minimal number of appliances for HA configuration
• Automated embedded PostgreSQL clustering with manual failover
• Automated embedded vRealize Orchestrator clustering

Enhanced Authentication Service

• Integrated user interface providing a common look and feel
• Enabled multiple features by new authentication service

Simplified Blueprint Authoring for Infrastructure and Applications

• Single unified model for both machine and application blueprints and unified graphical canvas for designing machine and application blueprint with dependencies and network topology
• Software component (formerly software service in Application Services) authoring on vSphere, vCloud Air, vCloud Director, and AWS endpoints)
• Extend or define external integrations in the canvas by using XaaS (formerly Advanced Service Design)
• Enable team collaboration and role segregation by enhancing and introducing fine-grain roles
• Blueprint as code and human-readable which can be created in editor of choice and stored in source control or import and export in the same or multiple vRealize Automation 7.0 instances
• Customer-requested machine and application blueprints provided
• Additional blueprints available on the VMware Solutions Exchange

Simplified and Enhanced NSX Support for Blueprint Authoring and Deployment

• Dynamically configure NSX Network and micro-segmentation unique for each application
• Automated connectivity to existing or on-demand networks
• Micro-segmentation for application stack isolation
• Automated security policy enforcement by using NSX security policies, groups, and tags
• On-demand dedicated NSX load balancer

Enhanced Cloud Support for vCloud Air and AWS

• Software component authoring for vCloud Air, vCloud Director, and Amazon AWS
• Simplified blueprint authoring for vCloud Air and vCloud Director
• Improved vCloud Air endpoint configuration
• Optional proxy configuration

vRealize Orchestrator 7 New Features

• Introduce vRealize Orchestrator Control Center for easy monitoring and troubleshooting
• Significant Smart Client improvements including Workflow tagging UI, Client reconnect options and enhanced search capabilities
• vSphere 6.X  vAPI endpoint support

NLVMUG Usercon 2016

The NLVMUG is probably the biggest thing after VMworld (Europe) if you live in The Netherlands or Belgium and it is the biggest Usercon outside the US.

The location of this edition is the same as last year, 1931 Event Center in Den Bosch
Although the agenda is to be determent, you can expect a solution exchange, workshops and breakout sessions. Registration is already possible through this link. Once the agenda is available I will post an update to this blog post.

Or if you are up for it, submit your own presentation through this "call for papers" link.

For those who are attending this NLVMUG Usercon, see you March 27th 2016!

Updates failing on VSAN hosts

A while back one of my customers ran into a issue when they wanted to install Update 1 for ESXi 6.0. Initially the ESXi hosts updated as expected, except five hosts.
For these five hosts, the first difference was that the hypervisor is installed on rack servers instead of blade servers. Main reason for this was, these hosts needed to accommodate local storage. This local storage is used for VMware Virtual SAN.
The five hosts where in the same VSAN cluster, and this cluster is used as management cluster for the customers entire vSphere environment.
 So these five hosts needed to get updated with Update 1 for ESXi 6.0, but VUM (VMware Update Manager) failed to install this update with a somewhat strange error message. With the help of the VMware Knowledge Base it became clear it had something to do with staging the Update before installing it on the ESXi host.
Because this customer runs ESXi from a USB flash device, the scratch location is redirected to shared storage. This configuration is similar for both the blade and rack servers. My first thought was that there was something wrong with this redirection for these specific five hosts. But after reviewing the advanced settings and verifying the time and date stamps on the various logfiles of the hosts within the .locker folders located on shared, all looked fine.
To my believe the update and patch staging location moved along with the logfile location when the scratch location was changed. A excellent blog post on logfile redirection and VSAN considerations when booting of a USB flash device is written by Cormac Hogan.
So what do VSAN, ESXi hosts booting from a USB flash device and scratch folder redirection have to do with failing updates? I will come to that, please bear with me as first give you some background on the intended use and deployment method of the hosts concerned.
Because these hosts are used as resources to run the management cluster on, these hosts where the first ESXi hosts to be deployed within the customers data center. No shared storage was available at the time of deployment. This imposes a challenge, to be able to use VSAN you need vCenter and to be able to deploy vCenter (as appliance or Windows based) you need storage accessible by one or more ESXi host(s).
A solution is to bootstrap vCenter on a single VSAN node, yes a single ESXi hosts that runs VSAN with only it's own local storage!.
If you want more information on how to bootstrap vCenter on a single VSAN node, please have a look at this 2-part blog post of William Lam on his VirtuallyGhetto blog
With the use of bootstrapping the vCenter onto a single ESXi host using it's local storage to create the VSAN datastore I could build the VSAN cluster and add the remaining four ESXi hosts to complete the cluster.
When the shared NFS storage became available later on in the project the scratch folder was redirected just like with all of the customers other ESXi hosts.
And here is the catch, the time the ESXi hosts have been running without the redirection, they have been logging to the local flash device. Usually this is not a big issue, other then the risks mentioned in Cormacs' blog post. But when you use VSAN, there will be additional log or trace files written (vsantraces). And in this customers case there were also VSAN Observer files written to the local flash device, VSAN Observer is a tool used to monitor and capture performance statistics, originally only used by the VMware VSAN engineering team. More information on VSAN Observer can be found here.
Vsantrace files can grow quickly up to 500 Mb and VSAN Observer trace files even larger, as I explained previously the scratch folder redirection was done some time (days) after the VSAN cluster became operational. When the redirection is done, the various trace files that are on the local flash device are NOT removed, these files do take up a considerable amount of space. In fact they take up so much space that there is not enough space left for staging Update 1 for ESXi 6.0 on the host.
Manually removing the old VSAN related trace files from the local flash devices was what solved the VUM issue. After the files where deleted the remediation of the ESXi hosts using VSAN ran without any issue.

Joining VMware

A while ago I got the opportunity to join VMware, I always thought that working with a vendor would be a great chance to get as close to the "fire" as possible. I am sure that moving to a vendor will definitely have it's advantages, both professional and personal.

I joined VMware as of november 2nd 2015, I am now working as a senior consultant SDDC with VMWare Professional Services Organisation for the Northern EMEA region.

Currently I am in the middle of the onboarding process, which includes a lot of (mandatory) training. Especially the training around  (cloud) automation and network virtualization is very interesting, but leaves little time for blogging at the moment. I hope to start blogging more frequently again when I am finished onboarding and used to my new role at VMware.

vRealize Orchestrator (vRO) licensing pitfall

vRO is a powerful tool used when you want to automate repetitive tasks on your vSphere environment. It has various plugins which bring integration with other VMware products as well as integration with third party hardware and software solutions.
vRO is a workflow based tool, besides the default workflows you can also create custom workflows. These workflows are stored in the vRO database together with the configuration data, depending on the use case you can choose either between a embedded database that is based on vPostgress or you can choose to use a external database solution based MS SQL or Oracle.

About the use cases which could use the embedded database, VMware documentation is not really clear on this. Because the Install and Configure documentation on vRO version 6.0.1 states that the embedded database should only be used for testing or PoC purposes. But the Install and Configure documentation on vRO version 6.0.3 states that default database option is suitable for small- or medium-scale environments. What is clear is that when you use the embedded database you cannot set up vRO to work in cluster mode, or change any licenses and the server certificate by using the vRO configuration interface.

So why am I writing about the database choice of vRO, well recently at a customer which uses vRO to automate tasks we ran into a operational issue when nobody could logon to the newly deployed vRO instance. It became clear that this vRO instances had been running on the default evaluation license, which stops working after its 90 day evaluation / grace period.
The solution would be simple, just login to the vRO configuration page and update the license key. But as documented in the Install and Configure documentation, when the embedded database is used you cannot update the license key from the configuration page. The only way to do this is to run the license update workflow from within vRO itself.
This was not possible because logon the the vRO client did not work because of the expired evaluation license. A solution to this "chicken and egg" problem was not easily found.

So how do you solve this problem, I first need to mention that I did not find this solution. All credits for the solution described below go to Martijn Went (@MartijnW01).

As mentioned earlier, when your vRO license is expired you cannot logon the the vRO client and run workflows. But you can logon to the vRO configuration page, after you first start the configuration service.

• Start the service and logon to the configuration page
• Go to the Database sub menu
• Change the database type to anything other than embedded, this will make License sub menu become available.

• Go to the License sub menu and go to the vCenter license tab and enter the details about the vCenter that needs to be used to license this vRO instance. Leave the port (443) and path (/sdk) default. Enter vCenter credentials and apply changes.
• After the connection to the vCenter is made the license will be verified, check the license details on the vCenter license tab you are on.
• Go back to the Database sub menu and change the database back to embedded
• Now you should be able to start the vRO service again.
• After the service is started, logon the the vRO client and run the workflow to license vRO to the vCenter license.

Now all should be operational again, the only thing that we had to do to be fully operational was reconnecting vRO to all vCenters within the environment. Not sure if this is related to the vRO configuration actions, but for some reason the vRO instance lost connection to the vCenters.

I'm sure this is not a supported solution, but VMware GSS did not have a solution on hand when they where contacted regarding this issue. The solution they found that might work, was to manually edit the database (replace the evaluation license key).

vSphere 6 template issue

A couple of weeks ago I was on-site at a customer. While working I needed to copy a couple of templates from their current environment to their new environment. For all except one template this worked as expected. The one template had some actions grayed out.

To complete my work I needed first the Rename action and second Remove form Inventory, it looked like this could possibly be a issue.
My first thought was that the template had become orphaned for some reason, but this would result in having all actions grayed out. To verify if this template was still usable I deployed a VM from it, this worked without any issues. The deployed VM looked fine.
After this action I took another look at the available actions for the template and all actions were available again, apparently the deployment action "fixed" the issue.
The real cause and why deploying a VM from it fixed it is still not clear to me, but I guess the deployment does something to either the vCenter inventory or the database that clears up the issue.

Deprecated VMFS volume(s) found on the host warning message

Last week I ran into a VMFS version warning message while building a new vSphere 6 environment for a enterprise customer. This environment is a green field deployment, so everything is new from SAN infrastructure, Network, Compute and Storage.
When I started adding payload ESXi hosts to their respective HA clusters, the only warning message I received was about the System logs being stored on non-persistent storage. This was expected as the hosts where Blades (without any local harddisks) which booted the hypervisor of a embedded USB device and there was no shared storage available yet.
After presenting shared storage and adding it to the ESXi hosts, being formatted in the process as VMFS 5 (VMFS 5.61 to be exact) all looked fine.
But when I rebooted the ESXi hosts, they came back online within vCenter and there was a extra warning message: Deprecated VMFS volume(s) found on the host. Please consider upgrading volume(s) to the latest version

The only action I had done on the environment between adding the shared storage and rebooting the hosts was deploying a OVA. This was the EMC VSI (Virtual Storage Integrator) appliance, deployment, configuration and installing of the plug-in went without any trouble.
Not sure if deploying the EMC OVA had anything to do with the warning message. In any case VMware has a KB article about the false positive deprecated VMFS warning.
KB2109735 does not disclose what is causing this false positive, but it does suggest a simple yet effective solution to the problem. At least it did solve the problem in my case.
By restarting the hosts management services the warning message is gone, the easiest way to do this is to run services.sh restart from either the DCUI or through a SSH session.