The downside of "Fast Provisioned" vApps in vCloud

When you or your company is replacing their existing SAN on which their vSphere and/or vCloud environment is running, you are probably going to replace it with a VAAI capable storage solution.
VAAI (vStorage APIs for Array Integration) was introduced with ESX(i) 4.1 and it basically provides a way to offload storage related tasks to the storage device. This reduces the load on the ESX(i) hosts and vCenter, it also speeds up those tasks.
When looking at vCloud Director, it is since version 5.1 also supporting VAAI, it is possible to offload any cloning of VM's "Fast Provisoning" as it is called in vCloud.
This speeds up the deployment of vApps considerably, it also helps to be very storage efficient when combined with deduplication and thin provisioning.
So far still no downside, well the point is that when you need to move vApps to a datastore that is on a different storage system or you need to move your entire vCloud environment to a new storage solution then that is the moment you could run into some unpleasant suprises (limitations) that VAAI brings.

A customer where I was recently working on their vSphere / vCloud environment where running both these environments on the same storage solution, but due to the explosive growth of the vCloud part they where starting to experience performance issues. The storage solution could not deliver the needed IOPS. Their current storage solution is a Netapp VAAI capable one, the purchased a new storage solution which would be only used for their vCloud environment. This also is a Netapp VAAI capable one, only a more high performance model.
They had enabled "Fast Provisioning" within vCloud from the get go, so all vApps where deployed like this. This means all linked clones where actually "Flex clones". When the new high performance storage solution was setup and ready to be used they needed to move / migrate their deployed vApps to the new datastores. They knew that this would cause the linked clones to be consolidated and become full clones as this is the only way a SvMotion could move the VM's within the vApps.
But a unexpected error stopped this, vSphere vCenter reported a error explaining that it could not consolidate and therefore not migrate the VM's. When I had a look at it I first thought the error could be caused by a compatibility mismatch between vSphere and Netapp Ontap, so to gather more info on this I opened up a support case with VMWare GSS. After providing vCloud logs and some additional information the SR became a PR and went to the engineering department, quickly after this had happened I got a answer from VMWare, which basically said that we where trying to do something that is not supported.
Not the answer I was looking for ! And when I continued to read thru the list of unsupported actions regarding vCloud and VAAI storage I even became more unhappy no consolidate or Storage vMotion are permitted on VAAI enabled storage (Fast Provisioned VM's and vApps that is). Luckily at the bottom of the email it read that it move could be done by the use of the "relocate" methode which is not present in any UI but only thru the vCD API. So there was a way of accomplishing the move, but it would take me some time to figure out how to do this. The mail provided a link to vCloud 5.1 API guide.

Unsupported / not permitted actions on "Fast provisioned" or "VAAI" clones:

• Consolidate operations are not permitted for VMs created through the VAAI cloning process, even for powered-off VMs.
• Using Storage vMotion through VC UI to move VMs created through the VAAI cloning process, is not permitted. However it is possible to relocate VAAI clones through the VCD API
• Reporting on capacity remaining on a given datastore may be inaccurate. 
• This release supports a maximum VAAI chain length of 256. Once VAAI clones reach this limit, further clones will be full-copy, handled by vCloud Director. This maximum is configurable using the db flag, Config.VirtualMachine.AllowedMaxVaaiChainLength
• Source VMs, when cloned, have a REDO log attached to them, which, if they are running, may cause a negative read performance (compared to non-cloned vm) impact.  
• No explicit way to prevent an admin from turning on Vaai flag for NAS datastores that do not support Vaai.
• Vaai clones will not work for vSphere  < 5.0 (Linked clones in VCD only work for VC versions after 5.0).
• Relocate of vaai clones (VMs residing on vaai enabled volumes) will not work if the vm has user created snapshot. The snapshots will need to be removed for clones to work.
• There maybe additional constraints on Vaai Clone support imposed by arrays. Please contact vendors. 

On page 18 of the guide you find the information about authentication and headers required and on page 231 there is information on relocating a VM to a different datastore, example code is provided.

With this new information I started to figure out what this relocate actually does, when I was looking for some additional information I stumbled upon the blog of Matt Vogt he wrote a article on VMware opening up a lot of API's with the 5.1 release of vCloud Director, but there was still a lot more to be desired. One of the things being able to change the storage profile of VM's and for this you need to know the Href for the storage profile you want to change to.

When you change the storage profile of a VM in vCloud it will trigger a relocate action, because the different storage profile refers to a different datastore. 

Matt used a script from Jake Robinson posted on the VMware Community which could retrieve the Href of storage profiles and created his own script which could change storage profiles of all VM's within a vApp. I took his script and adjusted it to my needs, this resulted in a script which can do the following.

It can change the storage profile of VM's within the same vApp in one go (sequentially). When the VM has a chain length lower then 2 it can be powered on, when the chain length is equal or greater then 2 the VM needs to be powered off to successfully complete the change. In any case the linked clone will be consolidated to a full/thick clone, so it becomes independent of it's base disk(s). 

When I tried the script it did not work for some reason it would not retrieve the Href of the new storage profile. For my purpose I did not spend any time on solving this I just hard coded the Href in the script, be sure to retrieve it yourself and update the script before use. This can be done easily with a Powercli one-liner.

Steps to get Href of destination (new) storage profile:

1. Manually change the storage profile of a VM to the new storage profile, this VM will be relocated to the corresponding datastore(s).
2. Setup a Powercli connection to vCloud director
3. Run $VM = Get-CIVApp "vApp name" | Get-CIVM VM name
4. Run $VM.ExtensionData.storageprofile.name (verify the name reflects the name of the destination storage profile)
5. Run $VM.ExtensionData.storageprofile.Href

This last line gives you a output that should look like: 

https://vCD-server/api/vdcStorageProfile/610a9da9-9970-49e5-bf58-9a7c89fea3a0

This is the Href of the storage profile, update this in the script at $profileHref line and you are ready to relocate vApp's.

Script code

Of course you can also use the script when relocating traditional/vSphere linked clones.

Real life benefits and caveats of NFS storage with VAAI

I recently was involved in a storage replacement project. This storage was used for a fairly large vSphere and vCloud environment amongst other.
During the replacement the customer went from FC storage to NFS storage, the FC storage did not incorporate VAAI (actually it did, but was disabled for stability reasons). The new NFS storage did incorporate VAAI and the design and sizing was done by involving all size reduction technics available on both the storage and vSphere. Therefore thin provisioning and deduplication were must haves to not run out of storage space before the end of the project.
In the initial phase of the project, right after the new NFS datastores where presented to the vSphere environment the customer ran into a big surprise. The VMware administrators had started to move VM's from the FC storage to the NFS storage and they forced the disk provisioning from these VM's from "thin" to "thick lazy zero" as they where convinced running "thin on thin" was very dangerous and could lead into a out of space issue without receiving any warnings or alerts from vCenter.
Not very well know but luckily recently very well documented by Cormac Hogan in his blog is that from the three disk provisioning types "thin, lazy zero thick and eager zero thick", only two are usable on NFS storage. Please read his blog posts on NFS best practices to get all the details, but in short NFS does not use the "lazy zero thick" type it will set it as "eager zero thick"! But they thought the deduplication feature of the storage device would straighten this out by deduping all "zero's". But what actually does happen with VAAI enabled, vSphere and the storage device become aware of each other and because vSphere has the disks provisioned as "eager zero thick" the storage device will not touch these zero'd blocks as they are reserved by vSphere (Reserve space feature of VAAI). So there was no dedupelication at all on these eager zero thick provisioned VM's.
After the storage admin's alerted them that they were about to run out of space on their datastores they only got up to 25% dedupe, I started looking at this and found the reason why.
The issue was solved without any uptime lost on the VM's, by creating and using a Powercli script that would Storage vMotion all VM's from 1 datastore to a other (temp.) datastore and back to the original location after the first action was completed. On the way back the disks where converted from "eager zero thick" to "thin".
After all VM's where back on there original datastore with "thin" provisioned disks the dedupe factor went up to almost 70%.