Adding an Tenant Administrator in Multi Tenant vRealize Automation environment gotcha!

While configuring the vRealize Operations Manager (vROps) Managment Pack for vRealize Automation (vRA) I ran into a somewhat strange issue.
The vRA environement was multi-tenant, having one additional tenant (Customer-1) next to the default tenant and to have the vROps Management Pack collect data from both tenants we need to have the same user account across both tenants. In my case an AD user account (service account) would be the account to manage this.
The AD account needs to have to correct vRA roles in each of the tenants:

• Infrastructure Administrator
• Tenant Administrator
• Fabric Group Administrator
• Software Architect role (vRA 7.0 and later)

Adding this account to the required roles is easy and went without any issue on the default tenant. When doing the same on the Customer-1 tenant adding the account (svc_vra in my screenshot) to the Infrastructure and Tenant Administrator roles I ran into an issue, to complete the action you need to click "Finish".

When I did this it returned an error message and failed to complete successfully:
Another user has already modified the data. Please reload the form and try again.

The only error that could be found in the vRA logging was the following:
it needs both the  vRA domain user add across all tenants, one tentant with error:

com.vmware.vcac.authentication.service.sso.horizon.HorizonTenantManagement.updateUserAttributeDefinitions:265 - Updating user attribute definitions in the tenant 'Customer-1'...
2016-11-29 08:28:33,186 vcac: [component="cafe:identity" priority="ERROR" thread="tomcat-http--32" tenant="vsphere.local" context="7kprPPeo" parent="" token="7kprPPeo"] com.vmware.vcac.platform.rest.client.error.ResponseErrorHandler.handleRestError:113 - [Rest Error]: {Status code: 409}, {Error code: 7} , {Error Source: null}, {Error Msg: Duplicate user attribute definition "manager" for org.}, {System Msg: vidm.userattributedefinition.duplicate}
2016-11-29 08:28:33,189 vcac: [component="cafe:identity" priority="ERROR" thread="tomcat-http--32" tenant="vsphere.local" context="7kprPPeo" parent="" token="7kprPPeo"] com.vmware.vcac.platform.service.rest.resolver.ApplicationExceptionHandler.handleRestException:610 - [Rest Error]: {Status code: 409}, {Error code: 7} , {Error Source: null}, {Error Msg: Duplicate user attribute definition "manager" for org.}, {System Msg: vidm.userattributedefinition.duplicate}
Reffering to AD user attribute "manager" which is additionally added to the default user attributes:

The error message you see in the partial logging above refers to a duplicate user attribute "manager" screenshot, this user attribute is not a default user attribute but additionally added in each of the tenants. Now having this attribute in multiple tenants looks like the cause of my error.
To be sure I removed the user attribute "manager" from the user attribute in the Customer-1 tenant.

After the user attribute "manager"  was removed I could without any issue add the account to the Infrastructure and Tenant Administrator roles.

Note: Don't forget to add the user attribute "manager" back to the user attribute configuration of the tenant.

VMworld US and Europe 2017 Dates and Locations

VMware confirmed the dates and locations of the 2017 edition of VMworld in both US and Europe.

VMworld US 2017:

 Mandalay Bay Convention Center in Las Vegas

August 27 - August 31

VMworld Europe:

 Fira Gran Via in Barcelona

September 11 - September 14*

* Please note that the VMworld Europe event is about a month earlier compared to the event dates of previous years. So if you already put a blocker in your calendar, make sure to update this ASAP!

I don't have any information why VMworld has advanced the Europe event, but now the event is closer to the US event and the chances are that you can still enjoy real summer days in Spain are a bit higher.

NSX Load balancer advanced HTTP redirect

One of the advanced features of NSX for vSphere is Load Balancing. The NSX for vSphere load balancer enables network traffic to follow multiple paths to a specific destination. It distributes incoming service requests according the selected load balancing algorithm among multiple servers in such a way that the load distribution is transparent to users. The NSX load balancer provides load balancing up to Layer 7.

When configuring a NSX load balancer you will need to map either an external, or public, IP address to a set of internal servers for load balancing. The load balancer accepts TCP, HTTP, or HTTPS requests on the external IP address and decides which internal server to use.

You create an application profile to define the behaviour of a particular type of network traffic. You also create a server pool consisting of backend server members and associate a service monitor with the pool to manage and share the backend servers flexibly and efficiently. The service monitor, monitors health of the server pool members according to a set of defined health check parameters.

During the creation of the application profile you can set up HTTP re-direct, by simply specifying the URL to which you want HTTP traffic to be redirected. For example, you can direct traffic from http://website.com to https://website.com

This kind of redirect works in most standard cases, but redirecting a login page which uses cookies will probably not work. For this you will need to make use of the Application Rule feature, which basically let's you create a rule for the traffic that you specific in this rule. A rule is actually a script, based on the HA-proxy scripting set.

VMware documentation around Application Rules is not extensive, you can find a few examples in the NSX Administration Guide. But if you need more details, most documentation will refer to the HA-proxy documentation. Although the NSX load balancer is based up on HA-proxy the ability to use HA-proxy scripting as a Application Rule is somewhat limited.

Lately I found a customer use case where HTTP redirection was required and where the basic HTTP auto redirection feature proofed not to be a valid solution. The use case for me is when a customer has both vROps (vRealize Operations Manager) and LogInsight. When you integrate these two solutions you get the option to search for syslog events in LogInsight that correlate to the object, error or warning presented in vROps.

Like in the example above when you click "Search for VM logs in vRealize Log Insight" the link will take you to the LogInsight web interface complete with the required filters to only show you the syslog events belonging to that particular object. To do so the link actually contains a search string or query. And good to know, the link is a HTTP link not a HTTPS link!

If you run vROps and LogInsight in an Enterprise scale, you will probably have multiple instances of each solution behind a load balancer, and if this load balancer is an NSX load balancer and you use the basic auto redirection feature for the VIP(s) of LogInsight. Then clicking the link will only take you to the default home screen of the LogInsight web interface, why?
Well the basic HTTP redirection does not take in to account the entire URL, but only FQDN that you have set up on the VIP. So here goes the nice integration between the two solutions....

The solution to keep the integration working between vROps and LogInsight while using NSX load balancers is fairly easy, you "only" need to make use of an Application Role and don't use the auto HTTP redirecting feature.
If you already had a look at the VMware Application Rule scripts or the HA-proxy scripting, you might find it a bit challenging to start with. But in fact not all scripts have to be complex, for the problem discussed in this blog post the script isn't complex or long.

# Redirect all HTTP requests to same URI but HTTPS redirect scheme 
Redirect scheme https if !{ ssl_fc }

As you can see, the solution only takes one single line (and maybe an additional line for description).
This solution will do a HTTP to HTTPS redirection which will respect the entire URL.

All (good) things come to an end. Goodbye C# client

Yesterday VMware announced the retirement of the C# client or vSphere Client for Windows. It will not be available for the next version of vSphere.
The next version of vSphere will get a HTML5 based Web Client, this will not only replace the C# client but will also replace the current Flash based Web Client. Although both web clients will coexist for some time to give (3rd party) plugins time to move from the Flash based Web Client to the new HTML5 based Web Client.
VMware states that the HTML5 Web Client will bring a great user experience. Currently you can already try the new HTML5 "look and feel" when you run ESXi 6.0 Update 2, this includes the Embedded Host Client which started of as a VMWare Lab Fling but made it into the Update 2 release.
And if you want to take it even a step further in your lab (VMware Lab Flings should not be used in a production environment), you can go ahead and get the vSphere HTML5 Web Client Fling!

At this time, it is not sure if the GA version of the HTML5 Web Client is going to look the same as the Fling does at the moment. But for the moment I really like the clean and basic look (called Clarity) of the Fling.

One other important thing to mention, VMware will try to stay on the same support model (supporting the one it’s released with, and one version back for upgrade transitioning) for the new HTML5 Web Client. Due to the amount of changes to the backend API it is not sure if they will be able to make this actually happen.

NSX syslog caveat

When you run a VMware based SDDC solution (in a lab or at a customer site). Which includes VMware NSX for network virtualization, it will probably also include VMware vRealize Operations (vROps) for monitoring and capacity planning. On top of that there is a pretty good change that VMware LogInsight is also included for log management and analytics.

Recently I ran into a syslog issue, using the combination of the three solutions mentioned above. When going through the final checks before handing over to the customer I saw that the Syslog server setting of NSX manager was incorrect, it showed the FQDN of one of the LogInsight nodes instead of the Load Balancer FQDN that is in front of the LogInsight nodes.
This Load Balancer was placed in front of the LogInsight nodes at a later stage during the project, so at the time of configuring syslog server in NSX manager UI one of the LogInsight nodes was used.
Therefore I thought at first it was a configuration change that had been forgotten, so I updated the Syslog server field in NSX manager UI and checked if LogInsight was still receiving syslog information from NSX manager. All looked just fine, it was still receiving syslog information and the NSX manager UI displayed the correct FQDN in the Syslog server field after refreshing the browser.

But when the customer validated this Syslog server field the next day, it was again pointing to one of the LogInsight nodes instead of the Load Balancer FQDN!
To be sure I again updated the Syslog server field and after rebooted the NSX manager appliance, after the reboot the Syslog server field still had the Load Balancer FQDN. A final check was to verify the field at Closing Of Business that day. At the end of the day, I checked the Syslog server field and it had again reverted back to the FQDN of one of the LogInsight nodes.

After doing some searching I found someone that had a similar issue with Syslog settings on ESXi hosts and this information pointed me into the direction of the vROps Management Pack for NSX-v. Apparently with the Management Pack installed, vROps pushes monitoring configurations instead of only just receiving monitoring data.
At the time vROps was using version 2.0 of the Management Pack which is fully compatible with the used NSX version 6.1.5.
Now I'm no vROps expert, but apparently there is an checkbox that you can check when installing the vROps Management Pack "Enable LogInsight Integration". The information I found also described a "solution" to the problem I was having, basically you can solve it by deselecting the LogInsight Integration checkbox. This supposed to solve the issue, but also disables the vROps and LogInsight integration for NSX. And the checkbox solution will only work if you update to version 3.x of the vROps Management Pack for NSX-v. Besides this, this is not what you want if you ask me, there is a reason why you are using vROps and LogInsight inside your environment right?!

The way that we (I got help from a LogInsight expert) got this fixed, is the following. Update the vROps Management Pack for NSX-v to version 3.x, it's probably not needed for this fix but it fixes a whole lot of other issues and future proofs vROps monitoring NSX-v by being able to distinct local from universal logical switches (please see release notes for full details on the Management Pack page).
After the upgrade, make sure that the LogInsight Integration checkbox is enabled.
Then go to the vROps web UI Home and select Administration.

After go to Inventory Explorer.

When you're in the Inventory Explorer you will have 2 panes, the left pane will have kind of a index and the right pane will have a lot of items. To get to the correct items you need to change you probably want to use the filter box on the top right within the right pane to filter only LogInsight related items.

For me there were two items I needed to change, this will probably be the same for other environments. The first Item that needs to be changed is named "Log Insight Server+FQDN", highlight it and then select the pencil icon to edit it.

Within the item you need to update the "HOST" field, currently it will reflect the wrong value / FQDN. Just update it so it reflects the correct FQDN and press OK.

The other item I needed to change is named "Log Insight Server Authentication+FQDN", again highlight it and select the pencil icon to edit it.

And again within the item you need to update the "HOST" field, currently it will reflect the wrong value / FQDN. Just update it so it reflects the correct FQDN and press OK.

That's it, it you give it some time (anywhere between one to two hours) and you will see that the Syslog server field within NSX manager is updated and now shows the correct FQDN.

VMworld 2016 public session voting

The content catalog for the upcoming VMworld 2016 in Las Vegas is live!
Session Voting is open to everyone. The only thing you need is a vmworld.com account to be able to participate. If you do not have a vmworld.com account, you can set one up for free.

So make sure you have a look at the content catalog, there are over 1500 submissions to vote for. You're vote does make a difference, it not only impact internal committee decisions, the VMworld 2016 program reserves space on the agenda (up to 5%) for “Customer Choice” Sessions.
Sessions with the most public votes will be guaranteed a spot for VMworld 2016.

And last but not least, if you are voting please take the following sessions in to consideration:

•  A session submitted by a colleague and myself

Security as a Service inside Software-Defined Data Center with VMware NSX, Palo Alto Networks and Network Micro-Segmentation. A Technical Deep Dive. [9062]

• Sessions submitted on a similar topic but from the business point of view

Security as a Service inside Software-Defined Data Center with VMware NSX, Palo Alto Networks and Network Micro-Segmentation. A Solution Deep Dive. [8811]

Customer Case Study: Empowering DevOps with a Fully Automated Prevention Focused Security Posture. [9212]

Thanks in advance for your time!

VMware announcements for 2016

Last year in January VMware launched vSphere 6, this year February is going to be the month for VMware's announcements for 2016.
This time around VMware has it's announcements divided in to two different tracks, each track is available in three different time zones. Track 1 is going to be all about EUC (End User Computing) and mobility, while track 2 is all about CMP (Cloud Management Platform) and HCI (Hyper-Converged Infrastructure).

You need to register to attend the launch event, so please do so here and find out the latest news.
The news will be brought to you by; Pat Gelsinger, Sanjay Poonen and Raghu Raghuram. On registration site you can also find more details and when each track is live in your region.