18 December, 2015

A Christmas present from VMware; vRealize Automation 7.0 is GA as of today !!

VMware announced a new version of vRA (vRealize Automation) during the last VMworld. And as of today vRealize Automation is available for download.

For detailed information around compatibility, release notes as well as the download link please go to the vRealize Automation 7.0 Information Center

Below a highlight list of new features and enhancements:

Streamlined and Automated Wizard-based Installation

  • Introduces management agent to automate the installation of Windows components and to collect logs
  • Automates the deployment of all vRealize Automation components
  • Installation wizards based on deployment needs: Minimal (Express) and Enterprise (Distributed) Installations

Simplified Deployment Architecture and High Availability Configuration

  • Embedded authentication service by using VMware Identity Manager
  • Converged Application Services in vRealize Automation Appliance
  • Reduced minimal number of appliances for HA configuration
  • Automated embedded PostgreSQL clustering with manual failover
  • Automated embedded vRealize Orchestrator clustering

Enhanced Authentication Service

  • Integrated user interface providing a common look and feel
  • Enabled multiple features by new authentication service

Simplified Blueprint Authoring for Infrastructure and Applications

  • Single unified model for both machine and application blueprints and unified graphical canvas for designing machine and application blueprint with dependencies and network topology
  • Software component (formerly software service in Application Services) authoring on vSphere, vCloud Air, vCloud Director, and AWS endpoints)
  • Extend or define external integrations in the canvas by using XaaS (formerly Advanced Service Design)
  • Enable team collaboration and role segregation by enhancing and introducing fine-grain roles
  • Blueprint as code and human-readable which can be created in editor of choice and stored in source control or import and export in the same or multiple vRealize Automation 7.0 instances
  • Customer-requested machine and application blueprints provided
  • Additional blueprints available on the VMware Solutions Exchange

Simplified and Enhanced NSX Support for Blueprint Authoring and Deployment

  • Dynamically configure NSX Network and micro-segmentation unique for each application
  • Automated connectivity to existing or on-demand networks
  • Micro-segmentation for application stack isolation
  • Automated security policy enforcement by using NSX security policies, groups, and tags
  • On-demand dedicated NSX load balancer

Enhanced Cloud Support for vCloud Air and AWS

  • Software component authoring for vCloud Air, vCloud Director, and Amazon AWS
  • Simplified blueprint authoring for vCloud Air and vCloud Director
  • Improved vCloud Air endpoint configuration
  • Optional proxy configuration

vRealize Orchestrator 7 New Features

  • Introduce vRealize Orchestrator Control Center for easy monitoring and troubleshooting
  • Significant Smart Client improvements including Workflow tagging UI, Client reconnect options and enhanced search capabilities
  • vSphere 6.X  vAPI endpoint support

09 December, 2015

NLVMUG Usercon 2016

The NLVMUG is probably the biggest thing after VMworld (Europe) if you live in The Netherlands or Belgium and it is the biggest Usercon outside the US.

The location of this edition is the same as last year, 1931 Event Center in Den Bosch
Although the agenda is to be determent, you can expect a solution exchange, workshops and breakout sessions. Registration is already possible through this link. Once the agenda is available I will post an update to this blog post.

Or if you are up for it, submit your own presentation through this "call for papers" link.

For those who are attending this NLVMUG Usercon, see you March 27th 2016!

07 December, 2015

Updates failing on VSAN hosts

A while back one of my customers ran into a issue when they wanted to install Update 1 for ESXi 6.0. Initially the ESXi hosts updated as expected, except five hosts.
For these five hosts, the first difference was that the hypervisor is installed on rack servers instead of blade servers. Main reason for this was, these hosts needed to accommodate local storage. This local storage is used for VMware Virtual SAN.
The five hosts where in the same VSAN cluster, and this cluster is used as management cluster for the customers entire vSphere environment.
 So these five hosts needed to get updated with Update 1 for ESXi 6.0, but VUM (VMware Update Manager) failed to install this update with a somewhat strange error message. With the help of the VMware Knowledge Base it became clear it had something to do with staging the Update before installing it on the ESXi host.
Because this customer runs ESXi from a USB flash device, the scratch location is redirected to shared storage. This configuration is similar for both the blade and rack servers. My first thought was that there was something wrong with this redirection for these specific five hosts. But after reviewing the advanced settings and verifying the time and date stamps on the various logfiles of the hosts within the .locker folders located on shared, all looked fine.
To my believe the update and patch staging location moved along with the logfile location when the scratch location was changed. A excellent blog post on logfile redirection and VSAN considerations when booting of a USB flash device is written by Cormac Hogan.
So what do VSAN, ESXi hosts booting from a USB flash device and scratch folder redirection have to do with failing updates? I will come to that, please bear with me as first give you some background on the intended use and deployment method of the hosts concerned.
Because these hosts are used as resources to run the management cluster on, these hosts where the first ESXi hosts to be deployed within the customers data center. No shared storage was available at the time of deployment. This imposes a challenge, to be able to use VSAN you need vCenter and to be able to deploy vCenter (as appliance or Windows based) you need storage accessible by one or more ESXi host(s).
A solution is to bootstrap vCenter on a single VSAN node, yes a single ESXi hosts that runs VSAN with only it's own local storage!.
If you want more information on how to bootstrap vCenter on a single VSAN node, please have a look at this 2-part blog post of William Lam on his VirtuallyGhetto blog
With the use of bootstrapping the vCenter onto a single ESXi host using it's local storage to create the VSAN datastore I could build the VSAN cluster and add the remaining four ESXi hosts to complete the cluster.
When the shared NFS storage became available later on in the project the scratch folder was redirected just like with all of the customers other ESXi hosts.
And here is the catch, the time the ESXi hosts have been running without the redirection, they have been logging to the local flash device. Usually this is not a big issue, other then the risks mentioned in Cormacs' blog post. But when you use VSAN, there will be additional log or trace files written (vsantraces). And in this customers case there were also VSAN Observer files written to the local flash device, VSAN Observer is a tool used to monitor and capture performance statistics, originally only used by the VMware VSAN engineering team. More information on VSAN Observer can be found here.
Vsantrace files can grow quickly up to 500 Mb and VSAN Observer trace files even larger, as I explained previously the scratch folder redirection was done some time (days) after the VSAN cluster became operational. When the redirection is done, the various trace files that are on the local flash device are NOT removed, these files do take up a considerable amount of space. In fact they take up so much space that there is not enough space left for staging Update 1 for ESXi 6.0 on the host.
Manually removing the old VSAN related trace files from the local flash devices was what solved the VUM issue. After the files where deleted the remediation of the ESXi hosts using VSAN ran without any issue.

Joining VMware

A while ago I got the opportunity to join VMware, I always thought that working with a vendor would be a great chance to get as close to the "fire" as possible. I am sure that moving to a vendor will definitely have it's advantages, both professional and personal.

I joined VMware as of november 2nd 2015, I am now working as a senior consultant SDDC with VMWare Professional Services Organisation for the Northern EMEA region.

Currently I am in the middle of the onboarding process, which includes a lot of (mandatory) training. Especially the training around  (cloud) automation and network virtualization is very interesting, but leaves little time for blogging at the moment. I hope to start blogging more frequently again when I am finished onboarding and used to my new role at VMware.

17 September, 2015

vRealize Orchestrator (vRO) licensing pitfall

vRO is a powerful tool used when you want to automate repetitive tasks on your vSphere environment. It has various plugins which bring integration with other VMware products as well as integration with third party hardware and software solutions.
vRO is a workflow based tool, besides the default workflows you can also create custom workflows. These workflows are stored in the vRO database together with the configuration data, depending on the use case you can choose either between a embedded database that is based on vPostgress or you can choose to use a external database solution based MS SQL or Oracle.

About the use cases which could use the embedded database, VMware documentation is not really clear on this. Because the Install and Configure documentation on vRO version 6.0.1 states that the embedded database should only be used for testing or PoC purposes. But the Install and Configure documentation on vRO version 6.0.3 states that default database option is suitable for small- or medium-scale environments. What is clear is that when you use the embedded database you cannot set up vRO to work in cluster mode, or change any licenses and the server certificate by using the vRO configuration interface.

So why am I writing about the database choice of vRO, well recently at a customer which uses vRO to automate tasks we ran into a operational issue when nobody could logon to the newly deployed vRO instance. It became clear that this vRO instances had been running on the default evaluation license, which stops working after its 90 day evaluation / grace period.
The solution would be simple, just login to the vRO configuration page and update the license key. But as documented in the Install and Configure documentation, when the embedded database is used you cannot update the license key from the configuration page. The only way to do this is to run the license update workflow from within vRO itself.
This was not possible because logon the the vRO client did not work because of the expired evaluation license. A solution to this "chicken and egg" problem was not easily found.

So how do you solve this problem, I first need to mention that I did not find this solution. All credits for the solution described below go to Martijn Went (@MartijnW01).

As mentioned earlier, when your vRO license is expired you cannot logon the the vRO client and run workflows. But you can logon to the vRO configuration page, after you first start the configuration service.

  • Start the service and logon to the configuration page
  • Go to the Database sub menu
  • Change the database type to anything other than embedded, this will make License sub menu become available.

  • Go to the License sub menu and go to the vCenter license tab and enter the details about the vCenter that needs to be used to license this vRO instance. Leave the port (443) and path (/sdk) default. Enter vCenter credentials and apply changes.
  • After the connection to the vCenter is made the license will be verified, check the license details on the vCenter license tab you are on.
  • Go back to the Database sub menu and change the database back to embedded
  • Now you should be able to start the vRO service again.
  • After the service is started, logon the the vRO client and run the workflow to license vRO to the vCenter license.

Now all should be operational again, the only thing that we had to do to be fully operational was reconnecting vRO to all vCenters within the environment. Not sure if this is related to the vRO configuration actions, but for some reason the vRO instance lost connection to the vCenters.

I'm sure this is not a supported solution, but VMware GSS did not have a solution on hand when they where contacted regarding this issue. The solution they found that might work, was to manually edit the database (replace the evaluation license key).

03 September, 2015

vSphere 6 template issue

A couple of weeks ago I was on-site at a customer. While working I needed to copy a couple of templates from their current environment to their new environment. For all except one template this worked as expected. The one template had some actions grayed out.

To complete my work I needed first the Rename action and second Remove form Inventory, it looked like this could possibly be a issue.
My first thought was that the template had become orphaned for some reason, but this would result in having all actions grayed out. To verify if this template was still usable I deployed a VM from it, this worked without any issues. The deployed VM looked fine.
After this action I took another look at the available actions for the template and all actions were available again, apparently the deployment action "fixed" the issue.
The real cause and why deploying a VM from it fixed it is still not clear to me, but I guess the deployment does something to either the vCenter inventory or the database that clears up the issue.

19 June, 2015

Deprecated VMFS volume(s) found on the host warning message

Last week I ran into a VMFS version warning message while building a new vSphere 6 environment for a enterprise customer. This environment is a green field deployment, so everything is new from SAN infrastructure, Network, Compute and Storage.
When I started adding payload ESXi hosts to their respective HA clusters, the only warning message I received was about the System logs being stored on non-persistent storage. This was expected as the hosts where Blades (without any local harddisks) which booted the hypervisor of a embedded USB device and there was no shared storage available yet.
After presenting shared storage and adding it to the ESXi hosts, being formatted in the process as VMFS 5 (VMFS 5.61 to be exact) all looked fine.
But when I rebooted the ESXi hosts, they came back online within vCenter and there was a extra warning message: Deprecated VMFS volume(s) found on the host. Please consider upgrading volume(s) to the latest version
The only action I had done on the environment between adding the shared storage and rebooting the hosts was deploying a OVA. This was the EMC VSI (Virtual Storage Integrator) appliance, deployment, configuration and installing of the plug-in went without any trouble.
Not sure if deploying the EMC OVA had anything to do with the warning message. In any case VMware has a KB article about the false positive deprecated VMFS warning.
KB2109735 does not disclose what is causing this false positive, but it does suggest a simple yet effective solution to the problem. At least it did solve the problem in my case.
By restarting the hosts management services the warning message is gone, the easiest way to do this is to run services.sh restart from either the DCUI or through a SSH session.

Installing vSphere 6 gotcha

The last couple of weeks I have been involved in the deployment of a vSphere 6.0 environment. This was my first vSphere 6.0 customer deployment.
Most of the implementation plan I wrote and as this was my first customer deployment I had to do some rework on my default implementation, which I have used on vSphere 5.5 implementations. To write this implementation plan I used the official VMware installation and configuration documentation as guideline. At the time of writing all implementation steps seamed logical, but when we came to the actual implementation we ran into a issue when we tried to add Identity Sources to the PSC (Platform Service Controller).
This customer vSphere environment is a large, multi tenant and multi site environment and for this reason together with the need for flexibility and scalability a design choice was made to use external PSC's. Also during the design sessions with the customer the decision had been made to use the vCSA (vCenter Server Appliance) instead of the up until vSphere 5.5 generally chose Windows based vCenter.
The issue we ran in to has to do with in which order you go to the configuration steps after the PSC is deployed. At some point you want to be able to join a PSC to the Active Directory domain to be able to use AD integrated authentication. When using external PSC's like in this particular case you don't join vCenter (vCSA) to the domain but the PSC. The reason for this is simple, SSO (Single Sign On) and Authentication services are provided by the PSC not vCenter.
The implementation plan followed the exact same steps for this part of the configuration as the VMware documentation and we followed the each en every step as they where documented in the implementation plan, but for some reason we still got a error message.
After verifying that I had used the most recent version of the VMware documentation I did a quick check on the VMware communities to see if anyone else had ran into the same issue.
There was a post dated March 27 2015 called "Wrong information in VMware 6.0 documentation vCSA 6.0" which described the exact issue and the writer came to the same conclusion that the VMware documentation has the configuration steps in the wrong order.
This community post was posted two and a half months ago, clearly VMware hadn't been able to update their online documentation.
Below you will find in short the correct steps and order to join a PSC (internal or external) to a Active Directory domain followed by adding Identity Sources to the PSC. For more details please have a look at the community post.

  1. Use the vSphere Web Client to log in as administrator@your_domain_name to the vCenter Server instance.
  2. Under Deployment, click System Configuration.
  3. Under System Configuration, click Nodes.
  4. Under Nodes, select a node and click the Manage tab.
  5. Under Advanced, select Active Directory, and click Join.
  6. Type the Active Directory details (use the administrator@your_domain_name syntax for the user).
  7. Reboot the appliance, after the reboot login as described in step 1.
  8. Navigate to Administration > Single Sign-On > Configuration.
  9. On the Identity Sources tab, click the Add Identity Source icon.
  10. Add the Active Directory domain as an identity source, enter the identity source settings, and click OK.
  11. Finished, you can now continue with the rest of the configuration like adding AD security groups to the roles defined in Global Permissions.

I am sure that VMware eventually will update their online documentation. In the mean time, if you follow the steps described above the configuration should be easy. Alternatively you still could follow the steps as described in the VMware documentation, but you will run into this issue. The error message has gotten a link to the correct webpage where you can first join your PSC to a Active Directory domain. After a reboot you can then restart your configuration. Although the second option will take more time the end result will be the same.

19 May, 2015

VMworld 2015 information

The largest VMware conference, VMworld is again having two conferences in 2015. As usual the first will be in San Francisco US and the second will be in Barcelona, exactly like previous year. Both conferences have a vast session catalog, but before a session even makes it to be hosted at the conference (US, Europe or both) there is a public voting possibility. This public voting gives you the opportunity to have some influence on what sessions will be hosted, it also gives a preview on what kind of sessions are being submitted.
So give it a try and vote for the sessions you find interesting, a VMworld account is needed to vote, if you do not have one you can register and create a VMworld account.

The closer we get to the actual conference dates, more and more information is published. Currently there already is some information regarding the key dates related to both of the conferences.

Conference key dates:

VMworld 2015 US (San Francisco): 
  • Conference is 30 August - 3 September
  • Registration opened 5th of May
  • Content Catalog is live 9th of June
  • Schedule Builder is live 21st of July
VMworld 2015 Europe (Barcelona):
  • Conference is 12 - 15 October
  • Registration opens 9th of June
  • Content Catalog is live 23rd of June
  • Schedule Builder is live 18th of August
But the most important is that registration for VMworld US is already open! So go and register !! 

Hope to see you at VMworld !

06 February, 2015

vExpert 2015!

Yesterday one of my fellow 2014 vExperts mentioned me in a tweet, I received vExpert 2015 status! That was a great way to finish the day, thanks Niels for the mention!
 For me it's the second year in a row I have been awarded the vExpert status. When I received it the first time I didn't really understand what it was all about, I just liked to be active in the VMware community and blog about all I find interesting within the world of virtualisation and cloud computing. But having vExpert status does really open doors for you, entering beta programs, pre-release webinars and briefings, NFR licenses and most important a lot of colleague vExperts. Especially those vExperts and also the whole VMware community is what it is all about, it's great to get recognised for your contribution to this community.

I’m thankful to be awarded the vExpert award in 2015. The VMware vExpert program acknowledges the people within the community that have contributed into evangelising virtualization as whole. It feels great to be part of that group and in my day to day role as Consultant as well as my role as blogger I will hopefully be able continue to contribute to the community.

For a complete list of vExperts 2015 please checkout the following link:

02 February, 2015

VMware reveals vSphere 6 !

Most people in the Virtualization industry already knew, today was the day of the big VMware event. There was enough speculation about what VMware would be revealing, but generally most rumors were telling vSphere 6 (or vSphere next). The rumors were right, VMware launched vSphere 6 today.

So why does VMware call this the biggest launch in VMware history?
Well let's first look at the highlights of vSphere 6 and after we take a quick look under the covers of vSphere 6.

With the new release VMware is focusing more and more on vCenter as a virtual appliance, no longer the appliance is in any way inferior to the Windows based vCenter. Both have the same maximums, although Vmware Update Manager is embedded within the vCenter virtual appliance (vCSA) a separate Windows (virtual) server is still needed for this component.
New to the vCSA is a guided installer, instead of getting a OVF file from the vCenter installer ISO and importing it in your vSphere environment it now get's a separate ISO and a more wizard like deployment.
For most deployments and customers, I think that the vCSA is the way to go from now on. Easier to deploy and no reboots regarding Microsoft security patches and updates. More secure and build-in upgrade/patch functionality. Although there will still be some doubt around troubleshooting vCenter as a virtual appliance, in addition there are also some vSphere features not integrated in the vCSA (yet).

The configuration maximums have been updated to further improve scalability, clusters can now support up to 64 nodes, Hosts support up to 12 TB of RAM and Virtual Machines support up to 128 vCPUs and 4 TB of RAM. Creating Virtual Machines of this size you will need to use the new hardware compatibility version 11.
Personally I have not seen many customers that have a use for Monster VMs, but it is good for the customers that do have this kind of VMs to be able to scale when needed.

In addition to the already existing reservations for CPU and Memory, vSphere 6 introduces Storage IO Controls this supports per VM reservations for Storage. With this you can set reservation in IOPS per VM, in other words a guaranteed minimum in IOPS.

Now the feature everybody was waiting for (I know I was), Virtual Volumes ( VVol). During the last couple of VMworlds VMware has been very busy promoting and explaining what VVol is about and why we need it. There is already so much information published around VVol, please checkout this blog post of Cormac Hogan is want to read up on it, that I will spare you the details. VVols is VMwares' way of making (existing) SAN/NAS systems VM-aware. The granularity of the current storage systems is mostly at LUN level, with VVol this is brought down to VM level. Every VM is comprised out of a minimum of 3 files, VMX (VM configuration), VMDK (hard disk file) and a swap file. With VVol each of these files becomes a separate volume a VVol.
A VMware product already leveraging VVol is VSAN (Virtual SAN); this has been fun testing during the Beta program. The most important points I found exploring VSAN, it is simple to configure and manage. On top of that, depending on the hardware, it's incredibly high performance. Another new addition in the storage area is the introduction of NFSv4.1 support (with Kerberos).

The next important enhancement is around vMotion, possibly one of the most "famous" vSphere features. Up until this new release of vSphere, vMotion was bound within a vCenter, or actually bound within a logical datacenter object. With the new release this is no longer the case, you could almost state that you now can vMotion across anything. Being able to vMotion across different vCenters and long distance vMotion, which will work up to 100 ms RTT are the two enhancements most important to me as most of the customers I work with have multi site environments. With these enhancements multi-site load balancing and "follow the sun" principles become possible.

Other enhancements around availability are, vSphere FT is going SMP. Fault Tolerant is now able to protect VMs with up to 4 vCPUs. vSphere Replication is also enhanced and it will now support replication up to 2000 VMs per vCenter.

The biggest complaint about vSphere 5.X was without doubt the web client, the look and feel of it was not good, performance was even worse. Along the vSphere 5.X versions it got better, but still not good enough for VM admins to stop using the C# client. To no surprise the web client has gotten a lot of enhancements on the performance part, login, action menus, page loading time, they all have been improved. Also an important enhancement is that the home screen layout looks again like the C# client, the recent tasks pane is back to the location it belongs, at the bottom center of the screen. Also all menus are flattened for quicker access to all menu actions.

Next to the enhancements there are also some new features added to vCenter, the most important in my opinion is the Content Library. This is basically a place where you can effectively manage all your VM templates, vApps and ISOs. The content can then be synchronized / replicated across sites and or vCenters.
One new feature that is added, will be greatly appreciated by anyone who ever had to manually replace all SSL certificates across their vSphere environment(s). The CLI interface called VMware Certificate Lifecycle Management, is going to be the way of managing both VMware and 3rd party certificates.

One of the most important differences, in my opinion between vCenter in vSphere 6 compared to vCenter in vSphere 5.5 is the components it is comprised from. With the introduction of vSphere 5.1 we got SSO (Single Sign On), Inventory Service and Web Client as additional components that you could install on separate (virtual) servers next to vCenter itself. With 5.1 the general advice was to use a separate server for each of these components, especially in large enterprise environments to keep up  performance and scalability. With vSphere 5.5 this advice was replaced with "keep the components together as much as you possibly can". Also the SSO component was rewritten,  now with vSphere 6 all known components have been replaced. vCenter is now comprised out of only two main components, Management (vCenter server) and Platform Services Controller (PSC). This is similar for both versions (VCSA and Windows based). The management part handles vSphere management, Infrastructure monitoring and management and API's. And the PSC handles SSO, Licensing, Certificate Management (VMCA) and Identity Management.
And more important, the installation or deployment options have become a lot simpler. You either install both components on 1 (virtual) Windows server (embedded node option) or you install the components on separate servers (individual node option). The same options are available for the vCSA. For now the general rule is to use the embedded option for standard vSphere environments. For more complex environments, which consists of more then one SSO enabled solution (example: vRealize Automation Center) and/or future need to do PSC replication across sites use the individual node deployment.