Adding an Tenant Administrator in Multi Tenant vRealize Automation environment gotcha!

While configuring the vRealize Operations Manager (vROps) Managment Pack for vRealize Automation (vRA) I ran into a somewhat strange issue.
The vRA environement was multi-tenant, having one additional tenant (Customer-1) next to the default tenant and to have the vROps Management Pack collect data from both tenants we need to have the same user account across both tenants. In my case an AD user account (service account) would be the account to manage this.
The AD account needs to have to correct vRA roles in each of the tenants:

• Infrastructure Administrator
• Tenant Administrator
• Fabric Group Administrator
• Software Architect role (vRA 7.0 and later)

Adding this account to the required roles is easy and went without any issue on the default tenant. When doing the same on the Customer-1 tenant adding the account (svc_vra in my screenshot) to the Infrastructure and Tenant Administrator roles I ran into an issue, to complete the action you need to click "Finish".

When I did this it returned an error message and failed to complete successfully:
Another user has already modified the data. Please reload the form and try again.

The only error that could be found in the vRA logging was the following:
it needs both the  vRA domain user add across all tenants, one tentant with error:

com.vmware.vcac.authentication.service.sso.horizon.HorizonTenantManagement.updateUserAttributeDefinitions:265 - Updating user attribute definitions in the tenant 'Customer-1'...
2016-11-29 08:28:33,186 vcac: [component="cafe:identity" priority="ERROR" thread="tomcat-http--32" tenant="vsphere.local" context="7kprPPeo" parent="" token="7kprPPeo"] com.vmware.vcac.platform.rest.client.error.ResponseErrorHandler.handleRestError:113 - [Rest Error]: {Status code: 409}, {Error code: 7} , {Error Source: null}, {Error Msg: Duplicate user attribute definition "manager" for org.}, {System Msg: vidm.userattributedefinition.duplicate}
2016-11-29 08:28:33,189 vcac: [component="cafe:identity" priority="ERROR" thread="tomcat-http--32" tenant="vsphere.local" context="7kprPPeo" parent="" token="7kprPPeo"] com.vmware.vcac.platform.service.rest.resolver.ApplicationExceptionHandler.handleRestException:610 - [Rest Error]: {Status code: 409}, {Error code: 7} , {Error Source: null}, {Error Msg: Duplicate user attribute definition "manager" for org.}, {System Msg: vidm.userattributedefinition.duplicate}
Reffering to AD user attribute "manager" which is additionally added to the default user attributes:

The error message you see in the partial logging above refers to a duplicate user attribute "manager" screenshot, this user attribute is not a default user attribute but additionally added in each of the tenants. Now having this attribute in multiple tenants looks like the cause of my error.
To be sure I removed the user attribute "manager" from the user attribute in the Customer-1 tenant.

After the user attribute "manager"  was removed I could without any issue add the account to the Infrastructure and Tenant Administrator roles.

Note: Don't forget to add the user attribute "manager" back to the user attribute configuration of the tenant.

NSX Load balancer advanced HTTP redirect

One of the advanced features of NSX for vSphere is Load Balancing. The NSX for vSphere load balancer enables network traffic to follow multiple paths to a specific destination. It distributes incoming service requests according the selected load balancing algorithm among multiple servers in such a way that the load distribution is transparent to users. The NSX load balancer provides load balancing up to Layer 7.

When configuring a NSX load balancer you will need to map either an external, or public, IP address to a set of internal servers for load balancing. The load balancer accepts TCP, HTTP, or HTTPS requests on the external IP address and decides which internal server to use.

You create an application profile to define the behaviour of a particular type of network traffic. You also create a server pool consisting of backend server members and associate a service monitor with the pool to manage and share the backend servers flexibly and efficiently. The service monitor, monitors health of the server pool members according to a set of defined health check parameters.

During the creation of the application profile you can set up HTTP re-direct, by simply specifying the URL to which you want HTTP traffic to be redirected. For example, you can direct traffic from http://website.com to https://website.com

This kind of redirect works in most standard cases, but redirecting a login page which uses cookies will probably not work. For this you will need to make use of the Application Rule feature, which basically let's you create a rule for the traffic that you specific in this rule. A rule is actually a script, based on the HA-proxy scripting set.

VMware documentation around Application Rules is not extensive, you can find a few examples in the NSX Administration Guide. But if you need more details, most documentation will refer to the HA-proxy documentation. Although the NSX load balancer is based up on HA-proxy the ability to use HA-proxy scripting as a Application Rule is somewhat limited.

Lately I found a customer use case where HTTP redirection was required and where the basic HTTP auto redirection feature proofed not to be a valid solution. The use case for me is when a customer has both vROps (vRealize Operations Manager) and LogInsight. When you integrate these two solutions you get the option to search for syslog events in LogInsight that correlate to the object, error or warning presented in vROps.

Like in the example above when you click "Search for VM logs in vRealize Log Insight" the link will take you to the LogInsight web interface complete with the required filters to only show you the syslog events belonging to that particular object. To do so the link actually contains a search string or query. And good to know, the link is a HTTP link not a HTTPS link!

If you run vROps and LogInsight in an Enterprise scale, you will probably have multiple instances of each solution behind a load balancer, and if this load balancer is an NSX load balancer and you use the basic auto redirection feature for the VIP(s) of LogInsight. Then clicking the link will only take you to the default home screen of the LogInsight web interface, why?
Well the basic HTTP redirection does not take in to account the entire URL, but only FQDN that you have set up on the VIP. So here goes the nice integration between the two solutions....

The solution to keep the integration working between vROps and LogInsight while using NSX load balancers is fairly easy, you "only" need to make use of an Application Role and don't use the auto HTTP redirecting feature.
If you already had a look at the VMware Application Rule scripts or the HA-proxy scripting, you might find it a bit challenging to start with. But in fact not all scripts have to be complex, for the problem discussed in this blog post the script isn't complex or long.

# Redirect all HTTP requests to same URI but HTTPS redirect scheme 
Redirect scheme https if !{ ssl_fc }

As you can see, the solution only takes one single line (and maybe an additional line for description).
This solution will do a HTTP to HTTPS redirection which will respect the entire URL.