vSphere 5.1 All Paths Down (APD) warning messages Part-2

In my previous post I wrote about troubleshooting APD and finding the root cause. At that time the root cause turned out to be a IP address conflict which bound 2 different NFS networks to 1 vmkernel interface, causing random APD events on all hosts connected to either of the NFS networks.
Besides these random APD events the same customer also had APD events that appeared to be a set times, mainly around the time backups where running (Netapp backups).
So for these APD events it looked like the NAS head becomes overloaded when it has to run a backup task on top of the normal load. When talking to the storage admin's, they wanted to see if any other background process (like deduplication) could trigger the same issue.
When looking into this we found that indeed deduplication could trigger APD events on vSphere and write "NFS slow" events in the Netapp logs. Not sure why this happened, a support case was opened with Netapp. Within this case all known performance where looked at and from the perfstat captures they could tell there were misaligned VM's. There is only one way to measure the effect of misaligned IO on your Netapp is by looking at the pw.over_limit counter. This counter is only available in priv set advanced command line mode.
So we ran the Scan Manager from the Netapp plugin for vCenter to see how many VM's where misaligned and we found there where a lot, due to the vCloud environment which has a lot of misaligned base VM's with multiple linked clones (which are automatically also misaligned).
During the search one of the storage admin's found a relation between a deduplication task on NAS head "A" in Datacenter A causing not only load on NAS head "A" in Datacenter A but also on NAS head "B" in Datacenter B. This was caused by a feature called "alternated write" this combined with the used storage design had a negative effect on the load.
All these factors turned a more then capable storage system into a stressed out, overloaded storage system. Like they say "The devil is in the details"
The vSphere environment suffered from these storage performance issues, in a massive way as you can imagine. This specific customer had Enterprise+ licenses and had SIOC (Storage I/O Control) enabled on all datastores, but even with SIOC they still experienced unresponsive and crashing VM's.

vSphere 5.1 All Paths Down (APD) warning messages Part-1

As you might know there has been some changes to APD and PDL behaviour in vSphere version 5.1.
APD got a new way of handling I/O's during APD scenarios and by setting a advanced option you can now even choose between the old way and the new I/O's "fast fail" way.
In short, the old way attempted to retry failed I/O's for a period of 140 seconds. After it would stop, the new way is to stop all I/O's directly. There are situations that the old way could cause the host(s) to disconnect from vCenter or even become unresponsive, this is something you want to prevent. By using the new way it will prevent these issues.
If you want to know more about ADP and PDL behaviour you should read the different articles on these subjects on Duncan Epping's blog Yellow Bricks or to be more specific start with this blog post.

So way I am writing a post about this subject when there is already a lot of good information out there ? A couple of days ago I have been asked to troubleshoot APD warning events in a vCenter log of a customer and I found that it was very difficult to pinpoint the cause of these APD messages.

There where multiple factors that made the troubleshooting difficult, one being that the customer has a stretched metro-cluster setup and a other one being that a part of the messages would appear on recurring times and the other part at "random" times.
The origin of the APD messages on recurring times was quickly found, when these would only occur if there where "extra" background processes running on the Netapp NAS heads. With background processes I mean processes like backup or de-duplication tasks. And the timeslot these APD events occurred made it a not so urgent issue. On how the APD events related to the background processes where solved I will write a other blog post as soon as all is double checked and confirmed both by VMware and Netapp.
On the other-hand the "random" APD events where a lot more difficult to pin point and the issue was a lot more high-profile as customers where complaining about slowness and unresponsiveness of their VM's and vApp's during the APD events. The customer used HP blades and Flex 10 modules for connecting the enclosures to the core network and NFS network. After troubleshooting and ruling-out all enclosure and network related possible causes, only the Netapp NAS heads or ESXi hosts could be the root cause of the APD events. These APD events occurred at random times, and most of these times the Netapp NAS heads didn't have any background processes running, nor did we find any information in the system logs of the Netapp pointing to the cause. Last place to look where the ESXi hosts, first checked all physical NIC's (which are actually virtual NIC's as they are presented to the Blade by the Flex 10 module) no issues there. Next checked network config of all hosts, luckily 1 host within a HA cluster assigned to a vCloud environment wrote warning messages of a duplicate IP address being used on 1 of it's VMkernel interfaces.
When I checked the network config of this host I saw nothing strange, so I started checking all other hosts with in the same cluster finding nothing... I continued checking a other HA cluster assigned to the same vCloud environment, finally I found a other host which had a VMkernel interface configured with the same IP address. Both VMkernel interfaces where used for NFS, but this IP address was not from the same subnet the Netapp NAS heads where in. They where in a seperate subnet in which a other NFS NAS was connected, this was used by only 1 of the 2 hosts. But on the hosts is was not used it was configured on the same dVswitch as the NFS network to the Netap NAS heads.
I updated the network config for the unused VMkernel interface and the "random" APD events disappeared. So I guess having a IP address conflict on a interface not used but within the same (d)Vswitch with a interface that is being used for NFS could cause APD events for multiple hosts and even multiple HA clusters, in fact it even affected hosts outside the vCloud environment the only thing in common was that they all where connected to the same Netapp NAS heads.

VM's grayed out (Status Unknown) after a APD (All Paths Down) event on NFS datastores

Last week during a change on one of the core switches of the NFS storage network at a customer, we ran into a big problem causing a outage of 50 % of all VM's for around 4 hours.
The problem started with a network related error, on which I will not elaborate other then the result was a unstable NFS network causing random disconnected NFS datastores on the majority of the customers' ESXi hosts. On top of that it also caused latencies, which triggered vMotion actions causing to ramp-up the latencies even more and resulting in a storm of failing vMotions.
In theory this would never have happend as the NFS network of the customer is completely redundant, but in real life it turned out completely different in this particular case.
After putting DRS into "partially automated" the vMotion storm stopped, the latency continued on the NFS network and this also had it's effect on the responsiveness of the ESXi hosts. Only after powering down the core switch (the one which had the change) all returned to normal status, datastores were connected to ESXi hosts again and latency disappeared. When looking into the vSphere client I found lots and lots of VMs that had a inaccesible or invalid status. When trying to power-on such a VM it would not work and you would get a "action not allowed in this state" message. The only way I knew to get them accessible again at the time was to unregister the VMs from vCenter (Remove from Inventory) and add them again to browsing to the .vmx file with the Datastore Browser and selecting "Add to Inventory". This was time consuming and tedious work, but the only quick fix in getting those VMs back into vCenter. Mind you, most of the VMs where still up-and-running but in no way manageable thru vCenter.
By the time I had all VMs registered again, some also needed a reboot as their OS crashed thru to high disk latencies. I was contacted by the vCloud admin, he had also lost around 100 VMs from his vCloud environment. It looked to be a other long task of getting those VMs back, but we faced a extra problem.  vCloud relies heavily on MoRef Id's for identification of VMs, in other words if the MoRef Id changes vCloud will no longer recognise this VM as it cannot match it to anything in its database.
But removing a VM from Inventory and re-adding it changes / updates its MoRef Id, so even if we wanted this quick fix I had could not be used on the VMs in vCloud. Luckily the vCloud admin found VMware kb1026043 it looked like VMware had the solution to our problem, but for some reason this solution was not working for us and it needed to have the host of the affected VMs in maintenance mode. It did help us with the search for a working solution, which was quickly after found by the vCloud admin on www.hypervisor.fr a French VMware related blog of Raphael Schitz. He wrote a article "Reload du vmx en Powershell" (Reload a vmx with Powershell) on how to reload VMs into Inventory without having the need for maintenance mode on your host(s), it all comes down to a PowerCLI one-liner that does the trick. You can alter the command to run it against entire Datacenter or just a Cluster.
In the end it saved our day by reloading all inaccesible and invalid VMs within just 5 minutes, this is a very useful one-liner as NFS is getting more and more used as preferred storage.

vCloud loses sync with the vCenter Inventory Service

Yesterday the vCloud admin of a customer I am working for on a other project, had a strange problem. He told me that it became impossible to deploy new vApps from the catalog, this proces would stop with all kinds of error's.
A few days before when we where testing the deployment of vApps on NFS datastores that where on new storage devices he also ran into a strange problem which looked quite similar. When deploying vCloud would first generate a other set of "shadow VMs" before actually deploying the vApp. This was strange because the vApp already had a set of running "shadow VMs" and it should have been "using" these.
Because the issue of yesterday had stopped production on the vCloud environment, the vCloud admin opened a support request with VMware GSS.
Once they had a look at the issue, it became quite quickly clear what was causing these strange problems. It looked like the vCloud Director cell had lost sync with the vCenter Inventory Service, this is not uncommon and you can find several "solutions" to this problem when searching thru some blogs.
In short the steps you need to take to re-start the syncing process again (If you are running a multi-cell environment):

1. First disable the cell and pass the active jobs to the other cells.

2. Display the current state of the cell to view any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --status

3. Then Quiesce the active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --quiesce true

4. Confirm the cell isn’t processing any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --status

5. Now shut the cell down to prevent any other jobs from becoming active on the cell.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <username> cell --shutdown

6. Then restart the services.
# service vmware-vcd restart

If you are not running a multi-cell environment, you can just restart the services but it will incorporate loss of service for a couple of minutes. If you want to keep the loss of services to a minimum, you can "monitor" or tail the cell.log file (when it reads 100%, it's done)

# tail -f /opt/vmware/vcloud-director/logs/cell.log

In case that the above does not work, you can also reboot the complete cell (In a multi-cell environment first pass all active tasks to other cells). Upon reboot the vCloud cell will reconnect and will sync again. 

Ok back to the issue, in this case this did not work, vCloud did not start the sync again in either way tried.

The support engineer wanted to restart all vCenter service to make sure that they were all running ok. Unfortunately this did not help, but with in this specific environment the Inventory Service is run from a separate server (VM) and after restarting the Inventory Service and after a other restart of the cell services vCloud did sync upon starting.

After this when talking to the vCloud admin, he told me that he had found other minor issues that probably could be interpreted as signs that vCloud and Inventory Service were getting out of sync again.

He found that some VMs where present in vCloud but not in vCenter, hence you could not find them when using the search feature (which is driven by Inventory Service) in the vSphere client. And he found that the "VMs and Clusters" view of the vSphere client had become very very slow upon to unresponsive. All other views in the client were working as usual.

As this issue can occur again, we decided to keep an eye out if we would detect either of these "signs" and when we do, do a restart of the Inventory Service ASAP.

Better to be safe then sorry.

Is rescanning storage adapters something you can do during production hours ?

First off, why do you initiate a rescan action ? Usually after you have made modifications on the storage / datastore part of your vSphere environment, like adding a new datastore. In general we can presume that a "Rescan All" can be done during production hours. In theory you could do this action without disrupting your environment, therefore it is acceptable to allow it during production hours. But in reality this is not always the case, so as with many decisions and actions on vSphere (and vSphere design) the correct answer to the question should be; It depends....on numerous factors.
I will try to clarify my "statement" by the hand of a use case combined with information gathered from a real life incident at one of the customers I worked for.
All I am writing in this article is related to vSphere 5.1 and up, why not the older versions ? For 2 reasons, first I did not investigate it on older versions and second starting version 5.1 the new Inventory Service was introduced. This service basically is there to make vCenter perform better (on several parts, for instance on the search feature) as it has it's own database which is used by the search feature. The Inventory Service is closely linked with Profile-Driven Storage, this manages the feature know as "Storage Profiles". If Inventory Service is not working then "Storage Profiles" will not be usable, please keep this in mind.
Back to the rescanning actions on storage adapters, this usually not something you do a daily basis. It will concentrate around changes to the storage area of your vSphere environment as mentioned before. Now you can imagine that it will be heavily used when move from one storage device to a other storage device. A action I recently worked on, the customer needed to change his SAN for performance and stability reasons. They decided to no longer go with FC storage but instead go and start use NFS storage, this resulted in a large migration action of all VM's in their vSphere environment.
When all VM's where moved to the new NFS backed datastores, the next step was to remove all FC backed datastores. One of the VM admins started this action deleting the FC datastores from the vSphere client host by host, what happend was that with every removal automatically a rescan was initiated on that host.
During this time the vCloud environment (runs on the same backend and vCenter as the "normal" vSphere environment) started to show very strange behaviour, It was not possible to deploy a vApp from the catalog, adding aVM to a vApp did not work also. And on the vSphere / vCenter part the search feature stopped working, deploying a new VM failed on most occasions.
With the search feature not working, there should be something wrong with the Inventory Service. I opened up my vSphere webclient to see if I could use search from there and I was presented with a error notification "Connection lost to Inventory Service". I check the services on the server that was running Inventory Service and was showing "started". After a reboot of the server (Inventory Service ran on a separate VM because of the size of the environment) and a long wait before the Inventory Service went form "starting" to "started" (about 40 min.) the search feature started working. But vCloud still did not operate normally and all Storage Profiles where gone from vCloud aswel as from vCenter. After the Profile-driven Storage service was restarted they re-appeared and vCloud started acting normal again.
Every admin returned to his normal work after this incident, so again one of the VM admins went back to removing the FC datastores and within a short time the strange behaviour on vCloud and vCenter appeared again. After opening a support request with VMWare GSS and having a talk with one of the support engineers, it became clear that the rescan action was putting to much stress on both Inventory Service and Profile-Driven Storage. In such a way that it slowed down all other tasks and eventually crashing the Inventory Service completely.
I had never seen or heard of such a incident with other customers or colleagues, so I started investigating what combination of factors are responsible for this to happen and soon it became clear and started to make sense.
The customer uses Storage Profiles on both vCloud and on vCenter (for all VM's +/- 1700) and it is known that vCloud can create a fair amount of load on Inventory Service by itself. Rescanning puts a extra load on Inventory Service and also puts a strain on the host agents aswel this combined generates a load of storage traffic which results in a service to fail / crash.
Advice of VMware GSS was to reduce rescanning and datastore manipulation to a minimum during production hours and at best to execute these actions during quietest hours of the day, or even better during a maintenance window.
And if rescanning is needed, to do it more localised instead of host, cluster or even datacenter wide. This last part is not possible thru the vSphere client, but on the DCUI of a host you can run a rescan per host and even per storage adapter. This will reduce the load by a big amount and makes it possible to run rescan actions during production hours without disrupting your vCloud and vSphere environment.

Example:
esxcli storage core adapter rescan -A vmhba0 
This will rescan vmhba0 on this specific host, leaving all other adapters (and hosts) alone.

So you can do a rescan action during production hours but you can never doing this without thinking it over and without knowing your environment. So please use it with caution and only execute it during production hours when it's absolute needed.

More information on rescanning can be found at VMware KB1003988

Sphere client shows "wrong" disk provisioning type in VM properties after deployment from template

Last week I got a question from a senior VM administrator at a customer where I had been working on several vSphere and storage related projects.
They use templates to deploy new VM's and they where created to use "thin" provisioned disks when the VM had been deployed. Now he told me that one of the other admin's had deployed a VM on the recently replaced storage (NFS storage) and this VM got "eager zero thick" disks when it was deployed.
To rule out a human error, he tried a other deployment from the same template himself and found that the VM also displayed "eager zero thick" in the VM properties after the deployment.
I asked me if I knew why this happend or if I maybe had altered their templates. On both question I had to tell him no, but it got me curious. As this customer had some disk provisioning issues before when moving from FC to NFS storage (read blog article :Real life benefits and caveats of NFS storage with VAAI) they where very keen to find out what was the cause of this to prevent any issues in the future.
When I started looking at this issue I knew they had the problem with a template containing Windows 2008 R2 SP1, so I thought I would clone this template and have a deeper look at it.
After cloning the template I converted the template to a VM and looked at the provisioning of the disks of this VM and to my surprise I found the disk where "thin"! I converted the VM back to a template and deployed a new VM from this template. When completed I again looked at the VM properties and found the disk provisioning was telling me "eager zero thick", now I was getting confused.
When I took at the VMDK discriptor file to check the virtual disk details I found just below The Disk Data Base line the following ddb.thinProvisioned = "1" so even if the vSphere client GUI stated the disk was "eager zero thick" the disk was actually "thin".
But I still needed to "convince" the customer that all was ok and it just was a flaw in the GUI. At the time I wanted to show the customer what was actually happening I ran over my test VM's once more.
In the mean time I had powered on some of the VM's and when I looked at their properties I found the disk provisioning now was "thin" to verify I did a deployment from template again.
After deployment I looked at the VM properties and the disk was "eager zero thick" I checked the VMDK discriptor and there it was "thin" I powered on the VM and went back to the properties screen of the VM and saw that the disk provisioning had changed to "thin".
For me it looks like if the vSphere client "chooses" a provisioning type after deployment but only reads the actual value from the discriptor file upon powering on / boot and then updates the VM properties.

Real life benefits and caveats of NFS storage with VAAI

I recently was involved in a storage replacement project. This storage was used for a fairly large vSphere and vCloud environment amongst other.
During the replacement the customer went from FC storage to NFS storage, the FC storage did not incorporate VAAI (actually it did, but was disabled for stability reasons). The new NFS storage did incorporate VAAI and the design and sizing was done by involving all size reduction technics available on both the storage and vSphere. Therefore thin provisioning and deduplication were must haves to not run out of storage space before the end of the project.
In the initial phase of the project, right after the new NFS datastores where presented to the vSphere environment the customer ran into a big surprise. The VMware administrators had started to move VM's from the FC storage to the NFS storage and they forced the disk provisioning from these VM's from "thin" to "thick lazy zero" as they where convinced running "thin on thin" was very dangerous and could lead into a out of space issue without receiving any warnings or alerts from vCenter.
Not very well know but luckily recently very well documented by Cormac Hogan in his blog is that from the three disk provisioning types "thin, lazy zero thick and eager zero thick", only two are usable on NFS storage. Please read his blog posts on NFS best practices to get all the details, but in short NFS does not use the "lazy zero thick" type it will set it as "eager zero thick"! But they thought the deduplication feature of the storage device would straighten this out by deduping all "zero's". But what actually does happen with VAAI enabled, vSphere and the storage device become aware of each other and because vSphere has the disks provisioned as "eager zero thick" the storage device will not touch these zero'd blocks as they are reserved by vSphere (Reserve space feature of VAAI). So there was no dedupelication at all on these eager zero thick provisioned VM's.
After the storage admin's alerted them that they were about to run out of space on their datastores they only got up to 25% dedupe, I started looking at this and found the reason why.
The issue was solved without any uptime lost on the VM's, by creating and using a Powercli script that would Storage vMotion all VM's from 1 datastore to a other (temp.) datastore and back to the original location after the first action was completed. On the way back the disks where converted from "eager zero thick" to "thin".
After all VM's where back on there original datastore with "thin" provisioned disks the dedupe factor went up to almost 70%.