Organizations running one or more Azure Kubernetes Service (AKS) clusters may encounter the need to re-assign the monitoring workspace(s) associated with the AKS deployment(s). Azure Monitor for Containers gives you performance visibility by collecting memory and processor metrics from AKS controllers, nodes, and containers. There are two primary integrations in the container monitoring solution: AKS Insights applied to the AKS services and VM Insights applied to the Virtual Machine Scale Set (VMSS). When AKS Insights is enabled on the AKS instance during deployment from the Azure Portal, VM Insights is automatically enabled on the VMSS nodes.

Background

• When AKS is first installed, many organizations follow the best practice to create a Container monitoring instance (AKS Insights) during AKS deployment, which is associated with a specific Azure Log Analytics workspace.
• Additionally, when creating a Container monitoring instance during initial AKS deployment, each node in the Azure Virtual Machine Scale Set (VMSS) that constitutes the AKS cluster is monitoring by a VM Insights instance.
• Organizations with many AKS clusters may experience Container Insights and VM Insights instance ‘sprawl’, since default AKS operations may result in dedicated Insights instances (and Log Analytics workspaces) for each AKS instance.
• For those organizations with too many or redundant workspaces–as well as those seeking to consolidate monitoring into a centralized Azure Log Analytics workspace to support an Azure Sentinel deployment–there is a need to move both the Container Insights and the VM Insights monitoring pieces to new, more suitable and/or centralized workspaces.

Reference Procedures

Moving AKS container and VMSS monitoring to a new Azure Log Analytics workspace can be accomplished with confidence and without application downtime.

There are four (4) high-level steps that are described in this article:

• Step #1: Disable AKS Insights with Azure CLI
• Step #2: Uninstall VMSS VM Insights, upgrade VMSS instances in the Azure Portal
• Step #3: Enable AKS Insights in the Azure Portal
• Step #4: Run PowerShell script to re-enable VMSS VM Insights, then upgrade VMSS instances in the Azure Portal a second time

Step 1: Disable AKS Insights with Azure CLI

Reference Procedure:    How to Stop Monitoring Your Azure Kubernetes Service cluster – Azure Monitor | Microsoft Docs

Remove the solution using Azure CLI. The command is “az aks disable-addons” as follows:

az aks disable-addons -a monitoring -n MyExistingManagedCluster -g MyExistingManagedClusterRG

Run the Azure CLI commands in Azure Cloud Shell or a local Azure PowerShell instance. Look for the “omsagent” “enabled” “false” values shown in Figure 1:

Figure 1 – Begin by disabling the AKS “monitoring” addon with Azure CLI (Command Line Interface).

Soon after you disable the AKS Insights addon, you will see that the Insights monitoring blade of the AKS container in the Azure Portal is ready to re-enable container monitoring to a new workspace as seen in Figure 2. However, don’t enable the new workspace just yet. Turning off AKS Insights does not automatically turn off VM Insights associated with the nodes in the AKS cluster. We’ll take care of that next.

Figure 2 – When you see this view, it means you successfully disabled any existing AKS Insights integration—now you need to disable the VM Insights integration.

Step 2: Uninstall VMSS VM Insights in Azure Portal

Reference Procedure:    Disable monitoring in Azure Monitor for VMs – Azure Monitor | Microsoft Docs

Observe that before you disable VMSS monitoring, your VMSS pool is monitored as a Linux server cluster in the ContainerInsights solution of Azure Log Analytics:

Figure 3 – The VMSS cluster is monitored at the server node level as well as the AKS Container level.

Azure VMSS Extensions push the Log Analytics configuration into the VMSS nodes. Remove the Log Analytics configuration by uninstalling the involved extensions.

Navigate to the Extensions blade of the VMSS associated with your AKS cluster and uninstall the two extensions shown in Figure 4 (OMSAgentForLinux and DependencyAgentLinux).

Figure 4 – Uninstall these two extensions from the Virtual machine scale set after disabling the container Insights addon.

After uninstalling the extensions, each VMSS node needs to be upgraded. If your VMSS doesn’t have automatic upgrading enabled, navigate to the Instances blade of the Virtual machine scale set (VMSS) and selectively update each node as shown in Figure 5.

Figure 5 – VMSS instances (server nodes) in the KMS cluster need to be individually updated after extensions are deleted or added.

Step 3: Enable AKS Insights in the Azure Portal

Reference Procedure:    Enable Azure Monitor for containers – Azure Monitor | Microsoft Docs

After each VMSS node has completed updating to the latest model, log data will stop streaming from the VMSS server nodes to the originally specified Azure Log Analytics workspace.

Now you are ready to re-enable AKS Insights, this time pointing to the new workspace. Navigate to Kubernetes Service (Containers) -> Monitoring -> Insights seen previously in Figure 2. Select the desired Azure Log Analytics workspace and push the Enable button.

Re-enable of AKS Insights does not re-enable VMSS Insights. To do that, proceed to the next and final step.

Step 4: Enable VMSS VM Insights using PowerShell

Reference Procedure:    PowerShell Gallery | Install-VMInsights.ps1 1.0

Here is a relevant PowerShell command line from the reference procedure:

.\Install-VMInsights.ps1 -WorkspaceRegion eastus -WorkspaceId <WorkspaceId> -WorkspaceKey <WorkspaceKey> -SubscriptionId <SubscriptionId> -ResourceGroup <ResourceGroup> -ReInstall
(Include switch –ReInstall to reinstall extensions even if already installed, for example to update to a different workspace.)

Download the PowerShell script and run in a local Azure PowerShell instance, or upload the script to Azure Cloud Shell and run from there.

After running the PowerShell script you will find extensions MMAExtension (formerly OMSAgentforLinux) and DAExtension (formerly DependencyAgentLinux) are installed in the VMSS (see Figure 6).

Figure 6 – Updated extensions that enable AKS cluster node monitoring, re-installed to the VMSS.

Once again, if your VMSS doesn’t have automatic upgrading enabled, navigate to the Instances blade of the Virtual machine scale set (VMSS) and selectively upgrade each node as previously shown in Figure 5.

After upgrading the VMSS nodes, VM Insights data becomes available in the new workspace, like the service map of the AKS cluster nodes seen in Figure 7.

Figure 7 – VM Insights data from the nodes of the AKS cluster are combined to render a valuable service map.

Considerations after moving container monitoring to new workspaces:

• As a best practice, delete idled or redundant Log Analytics workspaces, that is, those workspaces that no longer have active resources reporting into them.
• To keep access to historical data available, consider retaining retired workspaces for 30 days, there is no additional charge for that.

Tags: #Kubernetes #AKS #MVPBuzz #AzureSentinel #AzureMonitor #AzureArc #sysctr #VMInsights