vExpert 2015!

Yesterday one of my fellow 2014 vExperts mentioned me in a tweet, I received vExpert 2015 status! That was a great way to finish the day, thanks Niels for the mention!
 For me it's the second year in a row I have been awarded the vExpert status. When I received it the first time I didn't really understand what it was all about, I just liked to be active in the VMware community and blog about all I find interesting within the world of virtualisation and cloud computing. But having vExpert status does really open doors for you, entering beta programs, pre-release webinars and briefings, NFR licenses and most important a lot of colleague vExperts. Especially those vExperts and also the whole VMware community is what it is all about, it's great to get recognised for your contribution to this community.

I’m thankful to be awarded the vExpert award in 2015. The VMware vExpert program acknowledges the people within the community that have contributed into evangelising virtualization as whole. It feels great to be part of that group and in my day to day role as Consultant as well as my role as blogger I will hopefully be able continue to contribute to the community.

For a complete list of vExperts 2015 please checkout the following link:
http://blogs.vmware.com/vmtn/2015/02/vexpert-2014-announcement-2.html

VMware reveals vSphere 6 !

Most people in the Virtualization industry already knew, today was the day of the big VMware event. There was enough speculation about what VMware would be revealing, but generally most rumors were telling vSphere 6 (or vSphere next). The rumors were right, VMware launched vSphere 6 today.

So why does VMware call this the biggest launch in VMware history?

Well let's first look at the highlights of vSphere 6 and after we take a quick look under the covers of vSphere 6.

With the new release VMware is focusing more and more on vCenter as a virtual appliance, no longer the appliance is in any way inferior to the Windows based vCenter. Both have the same maximums, although Vmware Update Manager is embedded within the vCenter virtual appliance (vCSA) a separate Windows (virtual) server is still needed for this component.
New to the vCSA is a guided installer, instead of getting a OVF file from the vCenter installer ISO and importing it in your vSphere environment it now get's a separate ISO and a more wizard like deployment.
For most deployments and customers, I think that the vCSA is the way to go from now on. Easier to deploy and no reboots regarding Microsoft security patches and updates. More secure and build-in upgrade/patch functionality. Although there will still be some doubt around troubleshooting vCenter as a virtual appliance, in addition there are also some vSphere features not integrated in the vCSA (yet).

The configuration maximums have been updated to further improve scalability, clusters can now support up to 64 nodes, Hosts support up to 12 TB of RAM and Virtual Machines support up to 128 vCPUs and 4 TB of RAM. Creating Virtual Machines of this size you will need to use the new hardware compatibility version 11.
Personally I have not seen many customers that have a use for Monster VMs, but it is good for the customers that do have this kind of VMs to be able to scale when needed.

In addition to the already existing reservations for CPU and Memory, vSphere 6 introduces Storage IO Controls this supports per VM reservations for Storage. With this you can set reservation in IOPS per VM, in other words a guaranteed minimum in IOPS.

Now the feature everybody was waiting for (I know I was), Virtual Volumes ( VVol). During the last couple of VMworlds VMware has been very busy promoting and explaining what VVol is about and why we need it. There is already so much information published around VVol, please checkout this blog post of Cormac Hogan is want to read up on it, that I will spare you the details. VVols is VMwares' way of making (existing) SAN/NAS systems VM-aware. The granularity of the current storage systems is mostly at LUN level, with VVol this is brought down to VM level. Every VM is comprised out of a minimum of 3 files, VMX (VM configuration), VMDK (hard disk file) and a swap file. With VVol each of these files becomes a separate volume a VVol.
A VMware product already leveraging VVol is VSAN (Virtual SAN); this has been fun testing during the Beta program. The most important points I found exploring VSAN, it is simple to configure and manage. On top of that, depending on the hardware, it's incredibly high performance. Another new addition in the storage area is the introduction of NFSv4.1 support (with Kerberos).

The next important enhancement is around vMotion, possibly one of the most "famous" vSphere features. Up until this new release of vSphere, vMotion was bound within a vCenter, or actually bound within a logical datacenter object. With the new release this is no longer the case, you could almost state that you now can vMotion across anything. Being able to vMotion across different vCenters and long distance vMotion, which will work up to 100 ms RTT are the two enhancements most important to me as most of the customers I work with have multi site environments. With these enhancements multi-site load balancing and "follow the sun" principles become possible.

Other enhancements around availability are, vSphere FT is going SMP. Fault Tolerant is now able to protect VMs with up to 4 vCPUs. vSphere Replication is also enhanced and it will now support replication up to 2000 VMs per vCenter.

The biggest complaint about vSphere 5.X was without doubt the web client, the look and feel of it was not good, performance was even worse. Along the vSphere 5.X versions it got better, but still not good enough for VM admins to stop using the C# client. To no surprise the web client has gotten a lot of enhancements on the performance part, login, action menus, page loading time, they all have been improved. Also an important enhancement is that the home screen layout looks again like the C# client, the recent tasks pane is back to the location it belongs, at the bottom center of the screen. Also all menus are flattened for quicker access to all menu actions.

Next to the enhancements there are also some new features added to vCenter, the most important in my opinion is the Content Library. This is basically a place where you can effectively manage all your VM templates, vApps and ISOs. The content can then be synchronized / replicated across sites and or vCenters.
One new feature that is added, will be greatly appreciated by anyone who ever had to manually replace all SSL certificates across their vSphere environment(s). The CLI interface called VMware Certificate Lifecycle Management, is going to be the way of managing both VMware and 3rd party certificates.

One of the most important differences, in my opinion between vCenter in vSphere 6 compared to vCenter in vSphere 5.5 is the components it is comprised from. With the introduction of vSphere 5.1 we got SSO (Single Sign On), Inventory Service and Web Client as additional components that you could install on separate (virtual) servers next to vCenter itself. With 5.1 the general advice was to use a separate server for each of these components, especially in large enterprise environments to keep up  performance and scalability. With vSphere 5.5 this advice was replaced with "keep the components together as much as you possibly can". Also the SSO component was rewritten,  now with vSphere 6 all known components have been replaced. vCenter is now comprised out of only two main components, Management (vCenter server) and Platform Services Controller (PSC). This is similar for both versions (VCSA and Windows based). The management part handles vSphere management, Infrastructure monitoring and management and API's. And the PSC handles SSO, Licensing, Certificate Management (VMCA) and Identity Management.
And more important, the installation or deployment options have become a lot simpler. You either install both components on 1 (virtual) Windows server (embedded node option) or you install the components on separate servers (individual node option). The same options are available for the vCSA. For now the general rule is to use the embedded option for standard vSphere environments. For more complex environments, which consists of more then one SSO enabled solution (example: vRealize Automation Center) and/or future need to do PSC replication across sites use the individual node deployment.

Microsoft Remote Desktop Services issue

This is clearly not a usual virtualisation related blog post. But every now and then I'm asked by either a customer or colleague to have a look at a non virtualisation issue they run into, mostly because the issue is occurring on a virtualised server.
In this particular case for some unknown reason, overnight all RemoteApp programs serviced from their "Terminal Services" server suddenly stopped working.
When users started a RemoteApp they where prompted for their username and password, after entering these nothing happened. Looking in the Application event log two events were logged for each RemoteApp start attempt.

When looking at the first information event (ID 9003) it looks like the Windows Desktop Experience feature is not installed or the service is not started.

The second event (ID 9009) is a logical result from the first event, the Window Desktop Manager exits.
When searching for the cause of event ID 9003 all I found was that indeed the Windows Desktop Manager Experience probably was not installed or the RemoteApp was being started within a Remote Desktop Session.
When looking at the installed Windows features it turned that the feature was installed, I also verified that the service belonging to this were also started and no Remote Desktop sessions were used.
The next obvious thing was to recreate one of the RemoteApp programs, to see if this would solve this issue. Unfortunately this did not solve it.
Then I started thinking about the description of event ID 9003 "The Desktop Window Manager was unable to start because a composited theme is not in use" and the fact all Internet resources linked this to either the Window Desktop Feature or a not supported desktop theme on the client.
For this customer there were no changes to the desktops, although a small change due to a Windows Update / Patch could be possible considering the RemoteApp Programs stopped working at the same time on all workstations.
I started searching on the server part where I could define the settings for the Desktop Experience or for the allowed desktop theme's.

I started at the configuration of the Remote Desktop Session Host server, I opened the properties.

And going through the tabs I ended up at the Client Settings tab, were you find the Color Depth limit setting. This setting was set to 32-bit per pixel, I removed this limit by deselecting it.

As a result all RemoteApp Programs started working again on all workstations. About the root cause I still have no clue, for now I will keep it at a patch or update on the "Terminal Services" server.

Virtual Ethernet card "Virtual Ethernet Adapter" is of type which is not supported

Last week I was asked if I could solve a issue at a customer, they where patching all of their ESXi hosts. Only they ran into a host which refused to go in to maintenance mode, the reason for this was a VM which didn't vMotion to a other host successfully.
Basically, the vMotion task timed-out. When they investigated the error details a more specific error description.

It looked like there was something wrong with the network adapter installed in this specific VM. But when the configuration of this VM was checked it turned out that it had the customers default network adapter installed, VMXNET3.

Having a second look at the VM configuration, the Guest OS was set at Windows 2003 standard for some odd reason. After all this VM was running Windows 2008 R2 Enterprise.

After shutting down the VM and correcting the Guest OS, the VM was powered-on again. Now the VM could be vMotioned without any issue.

Already knowing that it is important to select the proper Guest OS to have VMtools install correctly and to have the right default virtual hardware presented to the VMs OS. There is a extra reason to pay extra attention to choosing the right Guest OS for a VM.

If you want to read more about the impact of a mismatched Guest OS selection, please read this blog post of Frank Denneman.

In this blog post you can also find a PowerCLI "one-liner" by Alan Renouf to quickly scan your environment for mismatched Guest OS.

A personal introduction to Diablo Technologies @VMworld

A couple of weeks before VMworld Europe, I got introduced to Diablo Technologies. Before that I had heard of the company but did not really know what they did.
I got the opportunity to have a meeting with Kevin Wagner, VP Marketing during VMworld to talk about Diablo Technologies, their current solutions and yet to be released solutions.
After I got introduced, I wanted to find out what Diablo Technologies is all about and what their relation is to virtualization. Looking into their website I found out they have a technology called Memory Channel Storage, which sounds great but I could not really figure out how this technology is actually used. I found it was Flash storage that sits in a DRAM slot, but it was not clear to me how you could leverage this Flash storage.
This changed quickly after meeting with Kevin, I was expecting a marketing talk which hopefully also would bring some technical details along. This was not the case at all, of course there was a fair amount of marketing info. After all I did not know the company nor their technologies really well, the meeting was for the largest part technical.
So what's Memory Channel Storage all about then, the technology is based on a combination of hardware and software (driver). The hardware, this looks somewhat like a DDR3 DRAM module with a additional heatsink attached to it.

Only this DRAM module there is no DDR3 RAM on it, but their is 19 nm MLC NAND (Flash) memory on it accompanied by some controllers, NAND and controllers are provided by Sandisk. These controllers make it possible to present this Flash memory as block-based storage within the servers OS, additionally a driver within the OS is needed. Currently drivers are available for Microsoft Windows, VMware ESXi and a variety of Linux distributions.
So two questions popped to mind, how much Flash is on such a module ? And Flash memory in a DRAM slot presented as block-based storage, how does the server (mainboard) recognise this correctly ?
The MCS modules currently are available in 2 sizes, 200 GB and 400 GB. There are ideas to also add a 800 GB version to this. So 200 GB and 400 GB Flash modules are adequate sized for using as a cashing solution, would it also be possible to use it as a real storage solution like you can do with SSDs ? More on this later, because the other question was how is it presented as block-based storage.
Well to be able to do this Diablo Technologies has incorporated controllers on the module which act as a disk controller, also a driver within the servers OS is needed to control the way the module(s) work. But you are still left with some module plugged into a DRAM slot on the mainboard, for this to be detected as MCS module instead of a DRAM module a minor change is needed to the mainboards UEFI (Unified Extensible Firmware Interface) which is the newer version of the main mainboards BIOS.
My two questions were answered, but a now other questions came up.
A change to the UEFI/BIOS of a server means you can not buy MCS modules and start plugging them in to your existing server hardware ! So what hardware do I need to buy, which OEM's does Diablo Technology work with ? And how do I fit these MCS modules together with DRAM modules in a server ?
The biggest OEM partners of Diablo Technology are IBM and SuperMicro, there are others and the OEM base is growing. It would be great to see the possibility for MCS in HP, Dell or even Fujitsu compute hardware if you ask me!
MCS module placement, as previously mentioned the modules are plugged into DDR3 DRAM slots and the way you can place them is exactly like you can place DRAM modules. This means a DRAM slot can either be occupied by a DRAM module or a MCS module, in other words you will need to have a server mainboard with enough DRAM slots to accommodate both types of modules.
There are a couple of things you need to take in account, so why would you choose a MCS solution over a SSD disk or PCIe SSD solution ? Simple, the memory bus is very close to the CPU and therefore performs with a higher speed / lower latency then on a PCIe or SATA / SAS bus connection. On top of that each MCS module uses only one memory channel, just like a DRAM module does. Because of this it can make use of parallelism, so instead of having 800 GB of flash available when you plug-in two 400 GB MCS modules you only have 400 GB of flash with both of the modules communicating through separate memory channels, guarantying a very low response times even when servicing heavy I/O loads.
 So SSD's result in low latencies when used in servers, we all know this. This is in the <1ms ranges in the most optimal situations. Latencies when using MCS technology are in the 4μs to 5μs range, but these numbers don't tell the complete story. The Diablo Technologies MCS is best at keeping ultra low latencies even with high throughput.
When looking at VM latencies running on a vSphere hypervisor it's not only the use of the memory bus that get's the very low latencies, when you look at how a traditional storage stack is buildup compared to a MCS storage stack you can easily see that the I/O path through the MCS storage stack is much shorter and passing less components. This reduction also reduces the total latency.

Now if we take a new storage technology based on local storage, VMware's VSAN. This uses SSDs for caching and spinning disks for persistent storage, so with MCS being presented as block-based storage you could use MCS for caching. This possibly could improve the already impressive VSAN performance.
But if we look a bit further down the road, VMware has announced it will probably support full flash VSAN configurations in the "2.0" release. Just imagine the performance you could get when you would use MCS in a full flash VSAN.
Also further down the road for Diablo Technologies, they have Carbon2 just around the corner this is their MCS solution based on DDR4 memory channel interface, which will bring MCS to the newest server architectures.

I must say I was really impressed with the technology and would recommend you to have a look at Diablo Technologies  if you are searching for a high throughput / ultra low latency storage solution. Diablo TTechnologies offers a possibility to "Test Drive" MCS, they will set you up with a time slot to try out MCS running in their own servers based at their HQ in Canada.
Both IBM and Sandisk have their own name for the MCS technology, at IBM it's called eXFlash DIMM and Sandisk runs with ULLtra DIMM.

21st VMUGBE+ Meeting

Over the last couple of years I have been attending VMUG meetings, they are all interesting although some have more interesting sessions than others. I always found the Belgium VMUG to have very good sessions.
The upcoming meeting promises also to be a good one, especially if you are into EVO:RAIL, VSAN and/or NSX and missed VMworld. Please check out the details below !
For those who are attending this VMUG meeting, see you November 21!

VMUG BE Agenda:

If you are a VMUG member you can register here for this free meeting. You can also find the location details on that page.

vSphere Distributed Switch health check gotcha !

Last week at a customer I was asked to have a look at a issue that they recently been having regarding the Distributed Switch health check feature.
This customer uses this feature to regularly check if all physical network connections are providing all the VLAN's configured on the Distributed Switches in their vSphere environment. About two weeks ago the health check suddenly started notifying errors on all hosts connected to a specific Distributed Switch. The error was a missing VLAN and a MTU size mismatch, the VLAN missing on the physical switch ports which connected to the physical NIC's of the hosts.
When I looked at the details of the error, I saw the missing VLAN was VLAN 0. This got me interested as usually you don't use VLAN 0 for standard network traffic and therefore you don't see it often in vSphere on a Distributed Switch which handles Virtual Machine network traffic.
When I checked the Distributed Switch configuration, more specific the dvPortgroups configured on it I found a dvPortGroup named "name_test" there where no VM's using it and when I looked at it's configuration I found it had the VLAN type set to "None".
After asking I learned that this dvPortGroup was created as a test during troubleshooting an other issue, after troubleshooting it was left in place. So there was no reason for leaving it there, I removed this dvPortGroup and after a refresh I checked the health status of this Distributed Switch.
To no surprise the health check showed up without any error's this time, so not only the missing VLAN error got solved but also the MTU size mismatch got solved in this.
So my take away on this is, when you as a vSphere Admin use the Distributed Switch health check feature to keep tabs on the status of your virtual to physical network please keep a tight procedure when it comes to changes on the Distributed Switches! And if you have to add a dvPortGroup for testing purposes please do not add it with the VLAN type set to "None", but add it with a VLAN that is available on your physical network interfaces.