ESXi 5.1 host cannot synchronize due to an incorrect user name or password

Recently at a customer I found a issue with 8 out of a 10 host HA cluster, the 8 hosts reported a warning as seen below.

Usually this error is seen when something is changed to the vpxuser user account, as also described in VMware's KB2017460.

The solution to the problem is pretty straight forward, you disconnect the host from vCenter and use the DCUI or a ssh session to remove the vpxuser user account. After you re-connect the host again to vCenter, you ignore the warning about the bad user name and password and enter the root credentials of the host when prompted. The host will now be re-connected and vCenter will create a new vpxuser user account. After this process is finished the host should be as it was before the issue.

In this particular case I found something strange, of the 8 hosts with this issue 7 hosts did not have a vpxuser account at all. When I tried to delete the user account it just reported back "unknown" only the 8th host (I know it's always the last one) did have a vpxuser user account.

The important part to know is that when there is no vpxuser user account just a disconnect followed by a re-connect will solve the issue.

Resetting (local) Windows 2008 and 2012 Administrator password

Although not a actual virtualization topic, I think it's something a lot of Admin's have to deal with every now and then. For me it's occasionally when working with templates of Windows server based VM's, when they are created a long time ago some Admin's just tend to forget what local administrator password was used. Because it is usually changed after a VM has been deployed to a different password.

When you then need to work on such a template for some reason you will get stuck when you do not remember the password. In the past I used the "Offline NT Password & Registry Editor" to reset the administrator password, but recently a Admin at a customer informed me there was a other way to reset the password and this way would also work on Windows server 2012.

You need to have the Windows server install dvd, or if you're like me into virtualization a ISO of that dvd. For both Windows server versions you need to go to the "Repair your computer" option on the Setup menu after booting the VM (if it's a template, first convert to VM) from this dvd.

After selecting this you need to select the OS instance you want to repair in the "System Recovery Options". Next select "Command Prompt". Once you are in the command prompt you need to go to the disk that holds the system32 folder, default this is C:

Then run the following commands:

• cd windows\system32
• ren Utilman.exe Utilman.exe.old
• copy cmd.exe Utilman.exe

After reboot the server (not from dvd!). At the logon screen press Windows-key + U this will open a command prompt.

Now run the following command "net user administrator Password" where password stands for you the password of your choice. After closing the command prompt you will be able to login to the server as local administrator with the password you previously entered.

You're able to logon again but you are not completely done yet, there is some clean-up to be done.

Again boot your VM from the Windows server dvd and follow the previous mentioned steps to get to the Command Prompt and go to the disk that holds the system32 folder.

• cd windows\system32
• ren utilman.exe utilman.exe.new
• copy utilman.exe.old utilman.exe

When finished close the Command prompt and click "Continue" now you're VM should be running as it should and you should be able to logon as local administrator.

If it was a template do not forget to convert your VM back to a template. And store the local administrator password in some sort of password-vault !

How to manage Hardware Version 10 VM's on a standalone (free) ESXi ?

As you might already know, to manage or configure VM's at hardware level 10 which comes with ESXi 5.5 and vSphere 5.5 you will need the vSphere web client (and vCenter) to do so. So what about a standalone or free ESXi 5.5 host ? The easiest way is NOT to upgrade the hardware version to version 10, but to simply keep it at any version below by this way you can fully manage and configure the VM with the C# client (full vSphere client).
I got caught by my enthusiasm to try the new features of vSphere 5.5 and I went and upgraded a VM to version 10 in a lab environment not connected to a vCenter. The quickest way I found to get out of this "unmanaged" situation was to create a new VM with hardware version 9 without disks, after creation I moved the disks of the unmanageable version 10 VM to this new VM and powered it on.
This will give you back control, but you still won't have any of the new features of 5.5.
A other way to manage these version 10 VM's is to install VMware Workstation 10 on you computer and connect to either your vCenter or to your standalone host, by this way you will be able to manage you VM's including the hardware version 10 VM's

More information and screenshot will follow soon in upcoming blog posts

20th Belgium VMUG meeting (agenda updated !)

VMUG meetings are always interessting and I think that the Belgium VMUG have very good sessions scheduled for their 20th VMUG meeting.
Please check out the details below ! For those who are attending this VMUG, see you December 5th !

If you are a VMUG member you can register here for this free meeting. You can also find the location details on that page.

Belgium VMUG 20th meeting agenda:

	ROOM 1		ROOM 2
	SPEAKER	SESSION	SPEAKER	SESSION
9:00	Erik Schils	Welcome
9:15	Chad Sakac	Software Defined Storage – State of the Art, and the Art of the Possible (EMC Gold Sponsor)
10:00	Lee Dilworth	Stretched Storage Clusters, Best Practices	Seb Hakiel	User Case: A 4K VDI
10:45	COFFEE
11:00	Dheeraj Pandey	“Bringing Google-like Infrastructure to the Enterprise “ (Nutanix Gold Sponsor)
11:45	Chris Van Den Abbeele	Turning the Tables on Cyber Attacks (Trend Micro Silver Sponsor)	Gilles Chekroun	Application Centric Infrastructure: Redefine the Power of IT (Cisco Silver Sponsor)
12:30	LUNCH
13:45	Joe Baguley	Thoughts from a CTO - Where's it all going?" (VMware Gold Sponsor)
14:30	Luc Dekens	Manage and Automate VMware View with PowerCLI	Viktor van den Berg and Arnim van Lieshout	vCac
15:15	COFFEE
15:30	Vaughn Stewart	Accelerate vSphere with Flash Storage Technologies	Bouke Groenescheij	Practical Vscsistats Result Analysis
16:15	Hans & Guests	Panel Discussion: "How Does the Future of Enterprise IT Impact Us?" (Pure Storage Silver Sponsor)	Kris Vandermeulen	Everything you always wanted to know about Microsoft licensing on VMware but were afraid to ask
16:45	Cormac Hogan	vSAN	Peter vandermeulen	User Case: Datacenters, Branch Office en Process Controle MIZ with VSA and How We Want to Integrate
17:15	vBEERS

VCAP5 exam experiences

I have been looking into getting VCAP5-DCA and VCAP5-DCD certified for some time now, when I was making plans to attend VMworld 2013 in Barcelona those 2 came together. I was lucky enough to have the opportunity to get some study time beside my work for a couple of weeks just before VMworld.
For the DCA exam I thought I did not have to study that much, so I just did some fleshing up on my vMA "skills" (don't use it regularly) and on configuring SATP rules by CLI.
On the other hand I did study a fair amount on the DCD exam as I knew it would be a real challenge.
I planned my DCA exam on Monday (Partner / TAM day) in the afternoon, because in the morning I was still travelling to Barcelona. And I planned my DCD exam on Tuesday morning, to be sure I could spend the rest of my time at VMworld attending sessions (and going to the various parties).
When I sat for my DCA exam I found it a very good exam, I would almost say I enjoyed taking it. I think that for enterprise admin's it is the same as it is within your comfort zone. Nevertheless it is a challenging exam where you are really tested for your hands-on knowledge. I did manage to complete all questions / tasks within the given time. I knew that I would not receive the results immediately after completing the last question it could take up to 15 days before you got the results back by email. But to my surprise VMware has recently changed this and they will get you the results within 8 hours ! As I did not know this I was surprised to find a email "Notice of VMware Technical Certification" the next morning and the best part I passed !
That day started very good, and with a boost in confidence I sat for my DCD exam. Now this is a whole different kind of exam, I didn't like it as much as the DCA exam for mainly 2 reasons. The first is that the design questions you get you need to do in a environment somewhat similar to MS Visio, but is feels like it is in Beta. It just isn't user friendly especially if you need to make modifications along the way, I was struggling more with the user interface as I was with the actual content of the question itself.
And the second reason for me is that a lot of questions have vague descriptions, I understand that when you are designing in real life there will also be points that are not really clear or vague but in this case you could ask to get clarification. This is not a option during the exam of course.
I have to say that the DCD exam is a difficult exam, in which time is your biggest enemy. It's just a huge amount of questions that you need to answer and the questions do cover every object in the blueprint !
When I got to the last question I had 5 minutes left, luckily I was able to answer within a blink of an eye. After answering the last question I had some exciting seconds before the results where shown, I passed this exam as well.
Needless to say that this has been the best VMworld ever for me !

Both exams are difficult and need some sort of exam strategy if you ask me, but both strategies are very different. If you are pursuing either or both of them, please take a look at my VCAP5-DCA and VCAP5-DCD study pages for useful links and tips.

vCenter 5.1 takes ESXi 5.1 host automatically out of maintenance mode

A couple of days ago I wrote about changing the scratch location on ESXi hosts in relation to the use of SNMP "Enabling SNMP on ESXI 5.1 host results in "The ramdisk 'root' is full" events".
In addition to this, the same customer ran into a other issue which also relates back to having the scratch location on non-persistent storage.
For hardware maintenance hosts where put into maintenance mode and "handed over" to datacenter engineers, they needed to update firmwares and bios of these hosts. When they finished the first host, they powered it on and it booted up ESXi as normal.
When they came up to find out if all was OK with this host, the VM admin looked up the host and found it in fully operational state ! And not in maintenance mode as expected, when looking at the host tasks and events it looked like "system" had taken the host out of maintenance mode after it came back online in vCenter. This could cause some serious issues, if the host is a member of a Cluster with HA and DRS (full automated) enabled but lacks the network uplinks that provide the VMnetwork(s). VM's would be vMotioned to this hosts, these VM's will lose their network connection !
For this to happen, the host has to be in operational state before it re-connected to vCenter. So why did this host "forget" it was in maintenance mode during the hardware maintenance ?
When I saw this happening I remembered that in the past with patching ESXi 4.1 hosts similar events happened and this was caused by that during the patching the scratch location (/tmp/scratch) got damaged / not accessible. When this happened the host booted normally, re-connected again to vCenter and got taken out of maintenance mode by the system account.
So a quick check learned that the host we where working on now had it's scratch location on non-persistent storage, next I checked in what way the datacenter engineers shutdown or reboot a ESXi host. I learned that as they don't have the rights to do shutdown or reboot a host through the vSphere client (either connected directly to the host or connected to vCenter), they used the out-of-band management (iLO, Drac, iRMC etc.). They always tried to do a graceful shutdown or reboot, but for this agents need to be installed on the host. As this is not the case this does not work for them, the other option is hard reset or power cycle. Let's be clear this is not a good thing, in my opinion you always need to do a clean and graceful shutdown or reboot ! Especially if the host concerned has its scratch location on non-persistent storage (read ramdisk), as this type of storage will act like if the host is experiencing power failure. And therefore not writing anything to a persistent location as it would during a clean reboot or shutdown.

VMware KB on changing the scratch location KB1033696
    

Enabling SNMP on ESXI 5.1 host results in "The ramdisk 'root' is full" events

Recently I ran into a issue when doing some work on a ESXI 5.1 Cluster, I needed to put the hosts in maintenance mode one by one. When I put the first host in maintenance I assumed that the host would be evacuated by migrating all VM's with the use of vMotion as Enterprise plus licenses where in place. But when the progress bar hit 13% the vMotion process stopped with a error. The error referred to "ramdisk (root) is full". When I checked with the customer they told me that this started happening after they configured snmp. They found that it would for some reason fill up the disk containing /var and they also found that sometimes those hosts became unresponsive to SSH and or DCUI.
After looking up the error message it quickly became clear what the relation was between snmp and "ramdisk (root) is full", the snmp service generated a .TRP file for every snmp trap sent. I believe this is not normal behavior, when I checked the functionality of snmp "esxcli system snmp test" it reported a error "Agent not responding, connect uds socket(/var/run/snmp.ctl) failed 2, err= No such file or directory" this proved my assumption was right (a successful test should result in "Comments: There is 1 target configured, send warmStart requested, test completed normally.". These files are stored in /var/spool/snmp and this location was located on non-persistent storage, in fact it was located on a 4GB ramdisk. Please check VMware KB2042772 for details on the error related to scratch location.
The snmp service will write a maximum of 8191 .TRP files, if the /var/spool/snmp location runs out of space before hitting this number you will have a host which is no longer able to vMotion, it can also become disconnected / unresponsive. And in some cases you are not able to start DCUI as there are no free inodes on the host. In this case connect to the host console (iLO, DRAC,....) and make sure you can login, then stop the vpxa service this will free up a inode and you will be able to start DCUI from the host's console (Troubleshooting options).
Now you need to remove the files that fill up the ramdisk, but first be sure that snmp is the cause of the issue by checking the file count in /var/spool/snmp "ls /var/spool/snmp | wc -l" if the result is above 2000 files snmp is most likely the cause.
To remove the files you can go 2 ways, move to the /var/spool/snmp dir and remove all .TRP files "for i in $(ls | grep trp); do rm -f $i; done" but I also found that stopping the snmp service "esxcli system snmp -e No" also clears the dir most of the times.
When the files are removed the host will start responding normally again, you will be able to start the vpxa service again "/etc/init.d/vpxa start" if you had to stop it previously.
The permanently fix this issue you need, as stated earlier to change to scratch location preferably to a local or shared datastore (VMFS or NFS). You can do this by editing the advanced settings (software) of the host “ScratchConfig -> ScratchConfig.ConfiguredLocation” after changing a reboot is mandatory to apply the change.
If you have to go thru a number of hosts , you might want to do this by using PowerCLI. If your lucky and the naming convention of the (local) datastores is uniform you will be able to automate all actions. If not (like in my particular case) you either go it host by host with the use of the vSphere (web) client. Or you could use a small script to look up all datastores, let you select the (local) datastore and update the advanced settings for you. sample script below.




VMware KB used as reference : KB2001550 KB2040707 KB1010837