VMware reveals vSphere 6 !

Most people in the Virtualization industry already knew, today was the day of the big VMware event. There was enough speculation about what VMware would be revealing, but generally most rumors were telling vSphere 6 (or vSphere next). The rumors were right, VMware launched vSphere 6 today.

So why does VMware call this the biggest launch in VMware history?

Well let's first look at the highlights of vSphere 6 and after we take a quick look under the covers of vSphere 6.

With the new release VMware is focusing more and more on vCenter as a virtual appliance, no longer the appliance is in any way inferior to the Windows based vCenter. Both have the same maximums, although Vmware Update Manager is embedded within the vCenter virtual appliance (vCSA) a separate Windows (virtual) server is still needed for this component.
New to the vCSA is a guided installer, instead of getting a OVF file from the vCenter installer ISO and importing it in your vSphere environment it now get's a separate ISO and a more wizard like deployment.
For most deployments and customers, I think that the vCSA is the way to go from now on. Easier to deploy and no reboots regarding Microsoft security patches and updates. More secure and build-in upgrade/patch functionality. Although there will still be some doubt around troubleshooting vCenter as a virtual appliance, in addition there are also some vSphere features not integrated in the vCSA (yet).

The configuration maximums have been updated to further improve scalability, clusters can now support up to 64 nodes, Hosts support up to 12 TB of RAM and Virtual Machines support up to 128 vCPUs and 4 TB of RAM. Creating Virtual Machines of this size you will need to use the new hardware compatibility version 11.
Personally I have not seen many customers that have a use for Monster VMs, but it is good for the customers that do have this kind of VMs to be able to scale when needed.

In addition to the already existing reservations for CPU and Memory, vSphere 6 introduces Storage IO Controls this supports per VM reservations for Storage. With this you can set reservation in IOPS per VM, in other words a guaranteed minimum in IOPS.

Now the feature everybody was waiting for (I know I was), Virtual Volumes ( VVol). During the last couple of VMworlds VMware has been very busy promoting and explaining what VVol is about and why we need it. There is already so much information published around VVol, please checkout this blog post of Cormac Hogan is want to read up on it, that I will spare you the details. VVols is VMwares' way of making (existing) SAN/NAS systems VM-aware. The granularity of the current storage systems is mostly at LUN level, with VVol this is brought down to VM level. Every VM is comprised out of a minimum of 3 files, VMX (VM configuration), VMDK (hard disk file) and a swap file. With VVol each of these files becomes a separate volume a VVol.
A VMware product already leveraging VVol is VSAN (Virtual SAN); this has been fun testing during the Beta program. The most important points I found exploring VSAN, it is simple to configure and manage. On top of that, depending on the hardware, it's incredibly high performance. Another new addition in the storage area is the introduction of NFSv4.1 support (with Kerberos).

The next important enhancement is around vMotion, possibly one of the most "famous" vSphere features. Up until this new release of vSphere, vMotion was bound within a vCenter, or actually bound within a logical datacenter object. With the new release this is no longer the case, you could almost state that you now can vMotion across anything. Being able to vMotion across different vCenters and long distance vMotion, which will work up to 100 ms RTT are the two enhancements most important to me as most of the customers I work with have multi site environments. With these enhancements multi-site load balancing and "follow the sun" principles become possible.

Other enhancements around availability are, vSphere FT is going SMP. Fault Tolerant is now able to protect VMs with up to 4 vCPUs. vSphere Replication is also enhanced and it will now support replication up to 2000 VMs per vCenter.

The biggest complaint about vSphere 5.X was without doubt the web client, the look and feel of it was not good, performance was even worse. Along the vSphere 5.X versions it got better, but still not good enough for VM admins to stop using the C# client. To no surprise the web client has gotten a lot of enhancements on the performance part, login, action menus, page loading time, they all have been improved. Also an important enhancement is that the home screen layout looks again like the C# client, the recent tasks pane is back to the location it belongs, at the bottom center of the screen. Also all menus are flattened for quicker access to all menu actions.

Next to the enhancements there are also some new features added to vCenter, the most important in my opinion is the Content Library. This is basically a place where you can effectively manage all your VM templates, vApps and ISOs. The content can then be synchronized / replicated across sites and or vCenters.
One new feature that is added, will be greatly appreciated by anyone who ever had to manually replace all SSL certificates across their vSphere environment(s). The CLI interface called VMware Certificate Lifecycle Management, is going to be the way of managing both VMware and 3rd party certificates.

One of the most important differences, in my opinion between vCenter in vSphere 6 compared to vCenter in vSphere 5.5 is the components it is comprised from. With the introduction of vSphere 5.1 we got SSO (Single Sign On), Inventory Service and Web Client as additional components that you could install on separate (virtual) servers next to vCenter itself. With 5.1 the general advice was to use a separate server for each of these components, especially in large enterprise environments to keep up  performance and scalability. With vSphere 5.5 this advice was replaced with "keep the components together as much as you possibly can". Also the SSO component was rewritten,  now with vSphere 6 all known components have been replaced. vCenter is now comprised out of only two main components, Management (vCenter server) and Platform Services Controller (PSC). This is similar for both versions (VCSA and Windows based). The management part handles vSphere management, Infrastructure monitoring and management and API's. And the PSC handles SSO, Licensing, Certificate Management (VMCA) and Identity Management.
And more important, the installation or deployment options have become a lot simpler. You either install both components on 1 (virtual) Windows server (embedded node option) or you install the components on separate servers (individual node option). The same options are available for the vCSA. For now the general rule is to use the embedded option for standard vSphere environments. For more complex environments, which consists of more then one SSO enabled solution (example: vRealize Automation Center) and/or future need to do PSC replication across sites use the individual node deployment.

vSphere 5.1 All Paths Down (APD) warning messages Part-2

In my previous post I wrote about troubleshooting APD and finding the root cause. At that time the root cause turned out to be a IP address conflict which bound 2 different NFS networks to 1 vmkernel interface, causing random APD events on all hosts connected to either of the NFS networks.
Besides these random APD events the same customer also had APD events that appeared to be a set times, mainly around the time backups where running (Netapp backups).
So for these APD events it looked like the NAS head becomes overloaded when it has to run a backup task on top of the normal load. When talking to the storage admin's, they wanted to see if any other background process (like deduplication) could trigger the same issue.
When looking into this we found that indeed deduplication could trigger APD events on vSphere and write "NFS slow" events in the Netapp logs. Not sure why this happened, a support case was opened with Netapp. Within this case all known performance where looked at and from the perfstat captures they could tell there were misaligned VM's. There is only one way to measure the effect of misaligned IO on your Netapp is by looking at the pw.over_limit counter. This counter is only available in priv set advanced command line mode.
So we ran the Scan Manager from the Netapp plugin for vCenter to see how many VM's where misaligned and we found there where a lot, due to the vCloud environment which has a lot of misaligned base VM's with multiple linked clones (which are automatically also misaligned).
During the search one of the storage admin's found a relation between a deduplication task on NAS head "A" in Datacenter A causing not only load on NAS head "A" in Datacenter A but also on NAS head "B" in Datacenter B. This was caused by a feature called "alternated write" this combined with the used storage design had a negative effect on the load.
All these factors turned a more then capable storage system into a stressed out, overloaded storage system. Like they say "The devil is in the details"
The vSphere environment suffered from these storage performance issues, in a massive way as you can imagine. This specific customer had Enterprise+ licenses and had SIOC (Storage I/O Control) enabled on all datastores, but even with SIOC they still experienced unresponsive and crashing VM's.

vSphere 5.1 All Paths Down (APD) warning messages Part-1

As you might know there has been some changes to APD and PDL behaviour in vSphere version 5.1.
APD got a new way of handling I/O's during APD scenarios and by setting a advanced option you can now even choose between the old way and the new I/O's "fast fail" way.
In short, the old way attempted to retry failed I/O's for a period of 140 seconds. After it would stop, the new way is to stop all I/O's directly. There are situations that the old way could cause the host(s) to disconnect from vCenter or even become unresponsive, this is something you want to prevent. By using the new way it will prevent these issues.
If you want to know more about ADP and PDL behaviour you should read the different articles on these subjects on Duncan Epping's blog Yellow Bricks or to be more specific start with this blog post.

So way I am writing a post about this subject when there is already a lot of good information out there ? A couple of days ago I have been asked to troubleshoot APD warning events in a vCenter log of a customer and I found that it was very difficult to pinpoint the cause of these APD messages.

There where multiple factors that made the troubleshooting difficult, one being that the customer has a stretched metro-cluster setup and a other one being that a part of the messages would appear on recurring times and the other part at "random" times.
The origin of the APD messages on recurring times was quickly found, when these would only occur if there where "extra" background processes running on the Netapp NAS heads. With background processes I mean processes like backup or de-duplication tasks. And the timeslot these APD events occurred made it a not so urgent issue. On how the APD events related to the background processes where solved I will write a other blog post as soon as all is double checked and confirmed both by VMware and Netapp.
On the other-hand the "random" APD events where a lot more difficult to pin point and the issue was a lot more high-profile as customers where complaining about slowness and unresponsiveness of their VM's and vApp's during the APD events. The customer used HP blades and Flex 10 modules for connecting the enclosures to the core network and NFS network. After troubleshooting and ruling-out all enclosure and network related possible causes, only the Netapp NAS heads or ESXi hosts could be the root cause of the APD events. These APD events occurred at random times, and most of these times the Netapp NAS heads didn't have any background processes running, nor did we find any information in the system logs of the Netapp pointing to the cause. Last place to look where the ESXi hosts, first checked all physical NIC's (which are actually virtual NIC's as they are presented to the Blade by the Flex 10 module) no issues there. Next checked network config of all hosts, luckily 1 host within a HA cluster assigned to a vCloud environment wrote warning messages of a duplicate IP address being used on 1 of it's VMkernel interfaces.
When I checked the network config of this host I saw nothing strange, so I started checking all other hosts with in the same cluster finding nothing... I continued checking a other HA cluster assigned to the same vCloud environment, finally I found a other host which had a VMkernel interface configured with the same IP address. Both VMkernel interfaces where used for NFS, but this IP address was not from the same subnet the Netapp NAS heads where in. They where in a seperate subnet in which a other NFS NAS was connected, this was used by only 1 of the 2 hosts. But on the hosts is was not used it was configured on the same dVswitch as the NFS network to the Netap NAS heads.
I updated the network config for the unused VMkernel interface and the "random" APD events disappeared. So I guess having a IP address conflict on a interface not used but within the same (d)Vswitch with a interface that is being used for NFS could cause APD events for multiple hosts and even multiple HA clusters, in fact it even affected hosts outside the vCloud environment the only thing in common was that they all where connected to the same Netapp NAS heads.