vSphere 5.1 All Paths Down (APD) warning messages Part-2

In my previous post I wrote about troubleshooting APD and finding the root cause. At that time the root cause turned out to be a IP address conflict which bound 2 different NFS networks to 1 vmkernel interface, causing random APD events on all hosts connected to either of the NFS networks.
Besides these random APD events the same customer also had APD events that appeared to be a set times, mainly around the time backups where running (Netapp backups).
So for these APD events it looked like the NAS head becomes overloaded when it has to run a backup task on top of the normal load. When talking to the storage admin's, they wanted to see if any other background process (like deduplication) could trigger the same issue.
When looking into this we found that indeed deduplication could trigger APD events on vSphere and write "NFS slow" events in the Netapp logs. Not sure why this happened, a support case was opened with Netapp. Within this case all known performance where looked at and from the perfstat captures they could tell there were misaligned VM's. There is only one way to measure the effect of misaligned IO on your Netapp is by looking at the pw.over_limit counter. This counter is only available in priv set advanced command line mode.
So we ran the Scan Manager from the Netapp plugin for vCenter to see how many VM's where misaligned and we found there where a lot, due to the vCloud environment which has a lot of misaligned base VM's with multiple linked clones (which are automatically also misaligned).
During the search one of the storage admin's found a relation between a deduplication task on NAS head "A" in Datacenter A causing not only load on NAS head "A" in Datacenter A but also on NAS head "B" in Datacenter B. This was caused by a feature called "alternated write" this combined with the used storage design had a negative effect on the load.
All these factors turned a more then capable storage system into a stressed out, overloaded storage system. Like they say "The devil is in the details"
The vSphere environment suffered from these storage performance issues, in a massive way as you can imagine. This specific customer had Enterprise+ licenses and had SIOC (Storage I/O Control) enabled on all datastores, but even with SIOC they still experienced unresponsive and crashing VM's.

The downside of "Fast Provisioned" vApps in vCloud

When you or your company is replacing their existing SAN on which their vSphere and/or vCloud environment is running, you are probably going to replace it with a VAAI capable storage solution.
VAAI (vStorage APIs for Array Integration) was introduced with ESX(i) 4.1 and it basically provides a way to offload storage related tasks to the storage device. This reduces the load on the ESX(i) hosts and vCenter, it also speeds up those tasks.
When looking at vCloud Director, it is since version 5.1 also supporting VAAI, it is possible to offload any cloning of VM's "Fast Provisoning" as it is called in vCloud.
This speeds up the deployment of vApps considerably, it also helps to be very storage efficient when combined with deduplication and thin provisioning.
So far still no downside, well the point is that when you need to move vApps to a datastore that is on a different storage system or you need to move your entire vCloud environment to a new storage solution then that is the moment you could run into some unpleasant suprises (limitations) that VAAI brings.

A customer where I was recently working on their vSphere / vCloud environment where running both these environments on the same storage solution, but due to the explosive growth of the vCloud part they where starting to experience performance issues. The storage solution could not deliver the needed IOPS. Their current storage solution is a Netapp VAAI capable one, the purchased a new storage solution which would be only used for their vCloud environment. This also is a Netapp VAAI capable one, only a more high performance model.
They had enabled "Fast Provisioning" within vCloud from the get go, so all vApps where deployed like this. This means all linked clones where actually "Flex clones". When the new high performance storage solution was setup and ready to be used they needed to move / migrate their deployed vApps to the new datastores. They knew that this would cause the linked clones to be consolidated and become full clones as this is the only way a SvMotion could move the VM's within the vApps.
But a unexpected error stopped this, vSphere vCenter reported a error explaining that it could not consolidate and therefore not migrate the VM's. When I had a look at it I first thought the error could be caused by a compatibility mismatch between vSphere and Netapp Ontap, so to gather more info on this I opened up a support case with VMWare GSS. After providing vCloud logs and some additional information the SR became a PR and went to the engineering department, quickly after this had happened I got a answer from VMWare, which basically said that we where trying to do something that is not supported.
Not the answer I was looking for ! And when I continued to read thru the list of unsupported actions regarding vCloud and VAAI storage I even became more unhappy no consolidate or Storage vMotion are permitted on VAAI enabled storage (Fast Provisioned VM's and vApps that is). Luckily at the bottom of the email it read that it move could be done by the use of the "relocate" methode which is not present in any UI but only thru the vCD API. So there was a way of accomplishing the move, but it would take me some time to figure out how to do this. The mail provided a link to vCloud 5.1 API guide.

Unsupported / not permitted actions on "Fast provisioned" or "VAAI" clones:

• Consolidate operations are not permitted for VMs created through the VAAI cloning process, even for powered-off VMs.
• Using Storage vMotion through VC UI to move VMs created through the VAAI cloning process, is not permitted. However it is possible to relocate VAAI clones through the VCD API
• Reporting on capacity remaining on a given datastore may be inaccurate. 
• This release supports a maximum VAAI chain length of 256. Once VAAI clones reach this limit, further clones will be full-copy, handled by vCloud Director. This maximum is configurable using the db flag, Config.VirtualMachine.AllowedMaxVaaiChainLength
• Source VMs, when cloned, have a REDO log attached to them, which, if they are running, may cause a negative read performance (compared to non-cloned vm) impact.  
• No explicit way to prevent an admin from turning on Vaai flag for NAS datastores that do not support Vaai.
• Vaai clones will not work for vSphere  < 5.0 (Linked clones in VCD only work for VC versions after 5.0).
• Relocate of vaai clones (VMs residing on vaai enabled volumes) will not work if the vm has user created snapshot. The snapshots will need to be removed for clones to work.
• There maybe additional constraints on Vaai Clone support imposed by arrays. Please contact vendors. 

On page 18 of the guide you find the information about authentication and headers required and on page 231 there is information on relocating a VM to a different datastore, example code is provided.

With this new information I started to figure out what this relocate actually does, when I was looking for some additional information I stumbled upon the blog of Matt Vogt he wrote a article on VMware opening up a lot of API's with the 5.1 release of vCloud Director, but there was still a lot more to be desired. One of the things being able to change the storage profile of VM's and for this you need to know the Href for the storage profile you want to change to.

When you change the storage profile of a VM in vCloud it will trigger a relocate action, because the different storage profile refers to a different datastore. 

Matt used a script from Jake Robinson posted on the VMware Community which could retrieve the Href of storage profiles and created his own script which could change storage profiles of all VM's within a vApp. I took his script and adjusted it to my needs, this resulted in a script which can do the following.

It can change the storage profile of VM's within the same vApp in one go (sequentially). When the VM has a chain length lower then 2 it can be powered on, when the chain length is equal or greater then 2 the VM needs to be powered off to successfully complete the change. In any case the linked clone will be consolidated to a full/thick clone, so it becomes independent of it's base disk(s). 

When I tried the script it did not work for some reason it would not retrieve the Href of the new storage profile. For my purpose I did not spend any time on solving this I just hard coded the Href in the script, be sure to retrieve it yourself and update the script before use. This can be done easily with a Powercli one-liner.

Steps to get Href of destination (new) storage profile:

1. Manually change the storage profile of a VM to the new storage profile, this VM will be relocated to the corresponding datastore(s).
2. Setup a Powercli connection to vCloud director
3. Run $VM = Get-CIVApp "vApp name" | Get-CIVM VM name
4. Run $VM.ExtensionData.storageprofile.name (verify the name reflects the name of the destination storage profile)
5. Run $VM.ExtensionData.storageprofile.Href

This last line gives you a output that should look like: 

https://vCD-server/api/vdcStorageProfile/610a9da9-9970-49e5-bf58-9a7c89fea3a0

This is the Href of the storage profile, update this in the script at $profileHref line and you are ready to relocate vApp's.

Script code

Of course you can also use the script when relocating traditional/vSphere linked clones.

VM's grayed out (Status Unknown) after a APD (All Paths Down) event on NFS datastores

Last week during a change on one of the core switches of the NFS storage network at a customer, we ran into a big problem causing a outage of 50 % of all VM's for around 4 hours.
The problem started with a network related error, on which I will not elaborate other then the result was a unstable NFS network causing random disconnected NFS datastores on the majority of the customers' ESXi hosts. On top of that it also caused latencies, which triggered vMotion actions causing to ramp-up the latencies even more and resulting in a storm of failing vMotions.
In theory this would never have happend as the NFS network of the customer is completely redundant, but in real life it turned out completely different in this particular case.
After putting DRS into "partially automated" the vMotion storm stopped, the latency continued on the NFS network and this also had it's effect on the responsiveness of the ESXi hosts. Only after powering down the core switch (the one which had the change) all returned to normal status, datastores were connected to ESXi hosts again and latency disappeared. When looking into the vSphere client I found lots and lots of VMs that had a inaccesible or invalid status. When trying to power-on such a VM it would not work and you would get a "action not allowed in this state" message. The only way I knew to get them accessible again at the time was to unregister the VMs from vCenter (Remove from Inventory) and add them again to browsing to the .vmx file with the Datastore Browser and selecting "Add to Inventory". This was time consuming and tedious work, but the only quick fix in getting those VMs back into vCenter. Mind you, most of the VMs where still up-and-running but in no way manageable thru vCenter.
By the time I had all VMs registered again, some also needed a reboot as their OS crashed thru to high disk latencies. I was contacted by the vCloud admin, he had also lost around 100 VMs from his vCloud environment. It looked to be a other long task of getting those VMs back, but we faced a extra problem.  vCloud relies heavily on MoRef Id's for identification of VMs, in other words if the MoRef Id changes vCloud will no longer recognise this VM as it cannot match it to anything in its database.
But removing a VM from Inventory and re-adding it changes / updates its MoRef Id, so even if we wanted this quick fix I had could not be used on the VMs in vCloud. Luckily the vCloud admin found VMware kb1026043 it looked like VMware had the solution to our problem, but for some reason this solution was not working for us and it needed to have the host of the affected VMs in maintenance mode. It did help us with the search for a working solution, which was quickly after found by the vCloud admin on www.hypervisor.fr a French VMware related blog of Raphael Schitz. He wrote a article "Reload du vmx en Powershell" (Reload a vmx with Powershell) on how to reload VMs into Inventory without having the need for maintenance mode on your host(s), it all comes down to a PowerCLI one-liner that does the trick. You can alter the command to run it against entire Datacenter or just a Cluster.
In the end it saved our day by reloading all inaccesible and invalid VMs within just 5 minutes, this is a very useful one-liner as NFS is getting more and more used as preferred storage.

vCloud loses sync with the vCenter Inventory Service

Yesterday the vCloud admin of a customer I am working for on a other project, had a strange problem. He told me that it became impossible to deploy new vApps from the catalog, this proces would stop with all kinds of error's.
A few days before when we where testing the deployment of vApps on NFS datastores that where on new storage devices he also ran into a strange problem which looked quite similar. When deploying vCloud would first generate a other set of "shadow VMs" before actually deploying the vApp. This was strange because the vApp already had a set of running "shadow VMs" and it should have been "using" these.
Because the issue of yesterday had stopped production on the vCloud environment, the vCloud admin opened a support request with VMware GSS.
Once they had a look at the issue, it became quite quickly clear what was causing these strange problems. It looked like the vCloud Director cell had lost sync with the vCenter Inventory Service, this is not uncommon and you can find several "solutions" to this problem when searching thru some blogs.
In short the steps you need to take to re-start the syncing process again (If you are running a multi-cell environment):

1. First disable the cell and pass the active jobs to the other cells.

2. Display the current state of the cell to view any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --status

3. Then Quiesce the active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --quiesce true

4. Confirm the cell isn’t processing any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --status

5. Now shut the cell down to prevent any other jobs from becoming active on the cell.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --shutdown

6. Then restart the services.
# service vmware-vcd restart

If you are not running a multi-cell environment, you can just restart the services but it will incorporate loss of service for a couple of minutes. If you want to keep the loss of services to a minimum, you can "monitor" or tail the cell.log file (when it reads 100%, it's done)

# tail -f /opt/vmware/vcloud-director/logs/cell.log

In case that the above does not work, you can also reboot the complete cell (In a multi-cell environment first pass all active tasks to other cells). Upon reboot the vCloud cell will reconnect and will sync again. 

Ok back to the issue, in this case this did not work, vCloud did not start the sync again in either way tried.

The support engineer wanted to restart all vCenter service to make sure that they were all running ok. Unfortunately this did not help, but with in this specific environment the Inventory Service is run from a separate server (VM) and after restarting the Inventory Service and after a other restart of the cell services vCloud did sync upon starting.

After this when talking to the vCloud admin, he told me that he had found other minor issues that probably could be interpreted as signs that vCloud and Inventory Service were getting out of sync again.

He found that some VMs where present in vCloud but not in vCenter, hence you could not find them when using the search feature (which is driven by Inventory Service) in the vSphere client. And he found that the "VMs and Clusters" view of the vSphere client had become very very slow upon to unresponsive. All other views in the client were working as usual.

As this issue can occur again, we decided to keep an eye out if we would detect either of these "signs" and when we do, do a restart of the Inventory Service ASAP.

Better to be safe then sorry.

Is rescanning storage adapters something you can do during production hours ?

First off, why do you initiate a rescan action ? Usually after you have made modifications on the storage / datastore part of your vSphere environment, like adding a new datastore. In general we can presume that a "Rescan All" can be done during production hours. In theory you could do this action without disrupting your environment, therefore it is acceptable to allow it during production hours. But in reality this is not always the case, so as with many decisions and actions on vSphere (and vSphere design) the correct answer to the question should be; It depends....on numerous factors.
I will try to clarify my "statement" by the hand of a use case combined with information gathered from a real life incident at one of the customers I worked for.
All I am writing in this article is related to vSphere 5.1 and up, why not the older versions ? For 2 reasons, first I did not investigate it on older versions and second starting version 5.1 the new Inventory Service was introduced. This service basically is there to make vCenter perform better (on several parts, for instance on the search feature) as it has it's own database which is used by the search feature. The Inventory Service is closely linked with Profile-Driven Storage, this manages the feature know as "Storage Profiles". If Inventory Service is not working then "Storage Profiles" will not be usable, please keep this in mind.
Back to the rescanning actions on storage adapters, this usually not something you do a daily basis. It will concentrate around changes to the storage area of your vSphere environment as mentioned before. Now you can imagine that it will be heavily used when move from one storage device to a other storage device. A action I recently worked on, the customer needed to change his SAN for performance and stability reasons. They decided to no longer go with FC storage but instead go and start use NFS storage, this resulted in a large migration action of all VM's in their vSphere environment.
When all VM's where moved to the new NFS backed datastores, the next step was to remove all FC backed datastores. One of the VM admins started this action deleting the FC datastores from the vSphere client host by host, what happend was that with every removal automatically a rescan was initiated on that host.
During this time the vCloud environment (runs on the same backend and vCenter as the "normal" vSphere environment) started to show very strange behaviour, It was not possible to deploy a vApp from the catalog, adding aVM to a vApp did not work also. And on the vSphere / vCenter part the search feature stopped working, deploying a new VM failed on most occasions.
With the search feature not working, there should be something wrong with the Inventory Service. I opened up my vSphere webclient to see if I could use search from there and I was presented with a error notification "Connection lost to Inventory Service". I check the services on the server that was running Inventory Service and was showing "started". After a reboot of the server (Inventory Service ran on a separate VM because of the size of the environment) and a long wait before the Inventory Service went form "starting" to "started" (about 40 min.) the search feature started working. But vCloud still did not operate normally and all Storage Profiles where gone from vCloud aswel as from vCenter. After the Profile-driven Storage service was restarted they re-appeared and vCloud started acting normal again.
Every admin returned to his normal work after this incident, so again one of the VM admins went back to removing the FC datastores and within a short time the strange behaviour on vCloud and vCenter appeared again. After opening a support request with VMWare GSS and having a talk with one of the support engineers, it became clear that the rescan action was putting to much stress on both Inventory Service and Profile-Driven Storage. In such a way that it slowed down all other tasks and eventually crashing the Inventory Service completely.
I had never seen or heard of such a incident with other customers or colleagues, so I started investigating what combination of factors are responsible for this to happen and soon it became clear and started to make sense.
The customer uses Storage Profiles on both vCloud and on vCenter (for all VM's +/- 1700) and it is known that vCloud can create a fair amount of load on Inventory Service by itself. Rescanning puts a extra load on Inventory Service and also puts a strain on the host agents aswel this combined generates a load of storage traffic which results in a service to fail / crash.
Advice of VMware GSS was to reduce rescanning and datastore manipulation to a minimum during production hours and at best to execute these actions during quietest hours of the day, or even better during a maintenance window.
And if rescanning is needed, to do it more localised instead of host, cluster or even datacenter wide. This last part is not possible thru the vSphere client, but on the DCUI of a host you can run a rescan per host and even per storage adapter. This will reduce the load by a big amount and makes it possible to run rescan actions during production hours without disrupting your vCloud and vSphere environment.

Example:
esxcli storage core adapter rescan -A vmhba0 
This will rescan vmhba0 on this specific host, leaving all other adapters (and hosts) alone.

So you can do a rescan action during production hours but you can never doing this without thinking it over and without knowing your environment. So please use it with caution and only execute it during production hours when it's absolute needed.

More information on rescanning can be found at VMware KB1003988

Real life benefits and caveats of NFS storage with VAAI

I recently was involved in a storage replacement project. This storage was used for a fairly large vSphere and vCloud environment amongst other.
During the replacement the customer went from FC storage to NFS storage, the FC storage did not incorporate VAAI (actually it did, but was disabled for stability reasons). The new NFS storage did incorporate VAAI and the design and sizing was done by involving all size reduction technics available on both the storage and vSphere. Therefore thin provisioning and deduplication were must haves to not run out of storage space before the end of the project.
In the initial phase of the project, right after the new NFS datastores where presented to the vSphere environment the customer ran into a big surprise. The VMware administrators had started to move VM's from the FC storage to the NFS storage and they forced the disk provisioning from these VM's from "thin" to "thick lazy zero" as they where convinced running "thin on thin" was very dangerous and could lead into a out of space issue without receiving any warnings or alerts from vCenter.
Not very well know but luckily recently very well documented by Cormac Hogan in his blog is that from the three disk provisioning types "thin, lazy zero thick and eager zero thick", only two are usable on NFS storage. Please read his blog posts on NFS best practices to get all the details, but in short NFS does not use the "lazy zero thick" type it will set it as "eager zero thick"! But they thought the deduplication feature of the storage device would straighten this out by deduping all "zero's". But what actually does happen with VAAI enabled, vSphere and the storage device become aware of each other and because vSphere has the disks provisioned as "eager zero thick" the storage device will not touch these zero'd blocks as they are reserved by vSphere (Reserve space feature of VAAI). So there was no dedupelication at all on these eager zero thick provisioned VM's.
After the storage admin's alerted them that they were about to run out of space on their datastores they only got up to 25% dedupe, I started looking at this and found the reason why.
The issue was solved without any uptime lost on the VM's, by creating and using a Powercli script that would Storage vMotion all VM's from 1 datastore to a other (temp.) datastore and back to the original location after the first action was completed. On the way back the disks where converted from "eager zero thick" to "thin".
After all VM's where back on there original datastore with "thin" provisioned disks the dedupe factor went up to almost 70%.

VMware Inventory Service causes vCenter server to run out of diskspace

Recently I ran into a issue with a vCenter server, this server apparently crashed because it ran out of diskspace on it's system disk. This particular vCenter server was running on physical hardware. When the system disk was investigated in a attempt to find what caused the disk to fill up 18 GB of free space in less then 48 hours, 4 large log files where found in the Inventory service database logs
 folder. After a second look, they where all created within 20 minutes and for some reason the logfiles weren't flushed to the database. Usually there is a new log file created every 10 minutes or so and / or if the file size is around 4 GB.
In search for a explanation and a solution a good look in the VMware vCenter documentation, resulted in finding a part in the "After you install the vCenter server" chapter called "Back up the Inventory Service Database on Windows" and in this part a "scripts"  folder is mentioned. One of the scripts is intended to backup the Inventory Database.
I presume everyone that has something to do vSphere knows how important the vCenter database is and I hope that all backup this accordingly. But I personally never read any article or best practice that stated to also backup the Inventory Service database separately, this includes the VMware documentation that is provided as a guideline to upgrade from vSphere 5.0 to 5.1.

Additional information:
After an other day of testing, the excessive growth of the logs could be leaded back to vCloud deploying vApps. For test we had 5 vApps deployed and monitored the xhive log files, this started growing during these deployment at a rate of 2 GB per minute.
If this can be addressed as "normal" behaviour....I am not sure, neither is GSS. The case which investigates this is still open at the time of me writing this article.

So the big question is / was, what caused the log files to grow this fast and why weren't these logs written to the database as it normally does? For now the only explanation is hardware malfunction.
This particular vCenter server was a physical server and it had a malfunctioning disk controller cache battery, therefore it had disabled the write cache to prevent data corruption.
This limited the write performance for the usual 100 MB/s to only 5MB/s, the server just could not keep up with writing log entries during the vCloud deployments.
The battery has been replaced and for now vCloud keeps on generating huge amount of log entries during deploy actions, but for now the server is able to keep up and write them to the database is a timely fashion.

Disabling VMware DRS with vCloud in your environment

So there is a commonly used practice when 1 (or multiple) host have a issue with applying DRS settings. Usually the quick fix is to disable DRS altogether on the Cluster containing the host(s), this fix is also used as a troubleshooting step by VMware GSS.

But if you run a environment which includes vCloud Director then this fix is a absolute No No !! By disabling DRS you (temporarily) remove all resource pools, and vCloud Director relies heavily on these. In fact by disabling you destroy your vCloud environment completely, all VM's running will keep on running as usual but you will not be able to do any vCloud actions.

The complete explanation as why this happens can be found here in a excellent blog post of Chris Colotti.

If you find yourself in the situation as described above your best bet is to contact VMware GSS and have a skilled DBA on standby. Because fixing this problem will ask for a lot of manual editing of records in the vCenter and vCloud databases.

After experiencing this issue with a customer, I found out there is a other way of fixing this issue. Please note this solution is not the recommended solution, but it is also used by GSS in smaller environments or as a last resort.
In my case I was very lucky to be in a environment that had the vCenter database located on Oracle and this Oracle system had what is called a "flashback" feature. It means a DBA can "rewind"  the transaction log very precise (up until the exact second) and almost instant. As we knew the exact time DRS had been disabled (vCenter logging) we could use "flashback" to go back to exactly 1 second before it was turned off. Off course all vCenter related services where stopped and after the database restore the vCenter server was rebooted.
When the server came back online again all was back to the state as we had before the disabling of DRS, so we had a working vCloud Director again (although it also needed a reboot to start functioning properly) and we had 1 host with DRS issues. But the mean problem was solved, vCloud was working again.

Because the Oracle "flashback" function was available and we acted quickly after the problem occurred, we minimised the potential loss of vCenter changes and actions getting lost / erased.

It still took some work to get the DRS issue solved on 1 host of the vCloud cluster, but it saved a lot of work by not having to manually editing the databases.