One Click Upgrade with NSX (GI) on Nutanix

What’s One Click Upgrade?

One of the most pain-point in any infrastructure is the periodic firmware updates to the underlying infrastructure. In addition to being a painful process, this is a very time consuming activity. One of the Coolest feature of Nutanix systems has been 1 Click Upgrade where the entire update process for NOS, NCC, Hypervisor and firmware can be automated. 1 Click update is a non-disruptive process where there is no downtime required and is a completely seamless automatic rolling update process. It also runs set of pre-checks and post-checks to ensure the health status of the cluster and that’s Cool!

Note: One Click upgrade for firmware doesn’t work across all hardware OEMs and in some cases is work in progress.

What’s NSX Guest Introspection

Guest introspection is a service that is deployed from NSX Manager to offload security functions to a dedicated security appliance on each hypervisor host in a VMware environment thereby removing the need for an AV agent within the guest operating system. Using the Guest Introspection driver baked into VMware Tools and a third-party service virtual machine, such as McAfee MOVE, all virtual machines are protected by real-time inspection as soon as they are powered on. Guest introspection functionality was previously achieved using vShield Manager with vShield Endpoint as part of the vCloud Networking and Security suite.

What’s McAfee MOVE?

McAfee Management for Optimized Virtual Environments (MOVE) is an anti-virus solution that removes the need for an individual agent install on every guest virtual machine, providing performance benefits and administrative savings at the same time as full anti-virus and malware protection.

So where is the problem?

Our Nutanix Infrastructure is used for Virtual Desktop hosting and we run on top of vSphere. As you would know, post reboot of server for the NDFS volumes to come online we need to have the CVM boot first but then with Guest Introspection VMs there is a dead-lock scenario considering NSX mandates Guest Introspection VMs to come online first (Guest Introspection VMs are hosted in NDFS volumes)

As a workaround, we have added external NFS volume and have migrated the Guest Introspection and MOVE VMs on to them to ensure we don’t have Operational challenges during reboot and maintenance windows.

I am sure this is not a one-off case and most of you using Agentless AV on a hyper-converged platform will hit this scenario and wanted to understand how you overcome these scenarios to make the solution truly hyper-converged without dependency of external NFS volumes!

How this impact One-Click upgrade scenario’s?

How do I avoid NFS volume and avoid conflict between NSX GI and CVM?

  • Is there a way / mechanism to host these VMs (McAfee MOVE / Guest Introspection) on the solution like increased SATADOM or an on-board card?

One Click upgrade has 3 critical parts

  • Upgrade NCC and NOS from Nutanix Stand-point since this doesn’t not require host reboots. (There are no issues in this front)
  • Upgrade of Hypervisor
  • Upgrade of BIOS and Hardware

Upgrade involves 3 stages and summarizing the pain points against them as well

  • vMotion all VMs out of the Host
    • vMotion of VMs hosted on NDFS works like a charm (Virtual Desktops / Servers)
    • McAfee MOVE and Guest Introspection doesn’t migrate as they are locally hosted. This needs to be either shut-down or migrated for proceeding to next step.
  • Upgrade Firmware / BIOS / Hypervisor
    • Evacuation / Shutdown of the VMs from the host must be successful for us to proceed with hypervisor / firmware upgrade. Most often today this doesn’t happen because of the GI and MOVE VMs.
    • Is there a way to automate / Orchestrate the entire process externally via Orchestrator engine if there are no native options for automating shutdown of GI & MOVE VMs?
    • Wherever applicable and supported Hosts firmware is updated seamlessly but then this also depends on evacuation of VMs!

How are you handling these situations in your environment and do share your thoughts!

Leave a Reply