NSX syslog caveat

When you run a VMware based SDDC solution (in a lab or at a customer site). Which includes VMware NSX for network virtualization, it will probably also include VMware vRealize Operations (vROps) for monitoring and capacity planning. On top of that there is a pretty good change that VMware LogInsight is also included for log management and analytics.

Recently I ran into a syslog issue, using the combination of the three solutions mentioned above. When going through the final checks before handing over to the customer I saw that the Syslog server setting of NSX manager was incorrect, it showed the FQDN of one of the LogInsight nodes instead of the Load Balancer FQDN that is in front of the LogInsight nodes.
This Load Balancer was placed in front of the LogInsight nodes at a later stage during the project, so at the time of configuring syslog server in NSX manager UI one of the LogInsight nodes was used.
Therefore I thought at first it was a configuration change that had been forgotten, so I updated the Syslog server field in NSX manager UI and checked if LogInsight was still receiving syslog information from NSX manager. All looked just fine, it was still receiving syslog information and the NSX manager UI displayed the correct FQDN in the Syslog server field after refreshing the browser.

But when the customer validated this Syslog server field the next day, it was again pointing to one of the LogInsight nodes instead of the Load Balancer FQDN!
To be sure I again updated the Syslog server field and after rebooted the NSX manager appliance, after the reboot the Syslog server field still had the Load Balancer FQDN. A final check was to verify the field at Closing Of Business that day. At the end of the day, I checked the Syslog server field and it had again reverted back to the FQDN of one of the LogInsight nodes.

After doing some searching I found someone that had a similar issue with Syslog settings on ESXi hosts and this information pointed me into the direction of the vROps Management Pack for NSX-v. Apparently with the Management Pack installed, vROps pushes monitoring configurations instead of only just receiving monitoring data.
At the time vROps was using version 2.0 of the Management Pack which is fully compatible with the used NSX version 6.1.5.
Now I'm no vROps expert, but apparently there is an checkbox that you can check when installing the vROps Management Pack "Enable LogInsight Integration". The information I found also described a "solution" to the problem I was having, basically you can solve it by deselecting the LogInsight Integration checkbox. This supposed to solve the issue, but also disables the vROps and LogInsight integration for NSX. And the checkbox solution will only work if you update to version 3.x of the vROps Management Pack for NSX-v. Besides this, this is not what you want if you ask me, there is a reason why you are using vROps and LogInsight inside your environment right?!

The way that we (I got help from a LogInsight expert) got this fixed, is the following. Update the vROps Management Pack for NSX-v to version 3.x, it's probably not needed for this fix but it fixes a whole lot of other issues and future proofs vROps monitoring NSX-v by being able to distinct local from universal logical switches (please see release notes for full details on the Management Pack page).
After the upgrade, make sure that the LogInsight Integration checkbox is enabled.
Then go to the vROps web UI Home and select Administration.

After go to Inventory Explorer.

When you're in the Inventory Explorer you will have 2 panes, the left pane will have kind of a index and the right pane will have a lot of items. To get to the correct items you need to change you probably want to use the filter box on the top right within the right pane to filter only LogInsight related items.

For me there were two items I needed to change, this will probably be the same for other environments. The first Item that needs to be changed is named "Log Insight Server+FQDN", highlight it and then select the pencil icon to edit it.

Within the item you need to update the "HOST" field, currently it will reflect the wrong value / FQDN. Just update it so it reflects the correct FQDN and press OK.

The other item I needed to change is named "Log Insight Server Authentication+FQDN", again highlight it and select the pencil icon to edit it.

And again within the item you need to update the "HOST" field, currently it will reflect the wrong value / FQDN. Just update it so it reflects the correct FQDN and press OK.

That's it, it you give it some time (anywhere between one to two hours) and you will see that the Syslog server field within NSX manager is updated and now shows the correct FQDN.