All customers using Microsoft Sentinel and Defender for Cloud-based security solutions have been seeing this notice in the Azure portal for some time:

“The Log Analytics agents (MMA, OMS) used to collect logs from virtual machines and servers will no longer be supported from August 31, 2024. Plan to migrate to Azure Monitor Agent before this date.”

What this means: The Microsoft Monitoring Agent (MMA) technology—used to connect servers to cloud-based Azure management solutions since 2015—is in the process of migrating to a framework based on the newer Azure Management Agent (AMA). Customers using the MMA to connect servers and devices to Azure Monitor, Microsoft Sentinel, and Microsoft Defender for Cloud must migrate to the AMA before August 31, 2024 when the MMA will be retired.

In addition to the monitored Windows and Linux servers on which the MMA is installed, firewalls and network devices may depend on MMA-based Linux syslog forwarders to ingest data into Microsoft Sentinel. Almost every customer that deployed Microsoft Sentinel from 2019 to the present is impacted.

This article discusses the ‘what’ and the ‘why’, then covers the ‘how’ in some detail to help customers start their migration as soon as practical. We will walk through a step-by-step protocol to replace the MMA with the AMA in a best practice Microsoft Sentinel deployment.

Don’t defer AMA deployment and migration preparation

New adopters of Azure monitoring and security solutions like Microsoft Sentinel and Defender for Cloud should deploy directly to the AMA.

For existing MMA-based users of Microsoft Monitor and Microsoft Sentinel, the time to begin piloting and migrating to the new Azure Monitor Agent (AMA) has arrived. The MMA-based platform has been very stable and reliable, and there have been legitimate reasons to wait before devoting effort to the inventible migration, see this gap analysis of AMA support for Microsoft Sentinel. But here is the wake-up call: A common observation among users beginning the migration task is the need to start getting familiar with the AMA architecture, which merits more study and preparation than might be expected.

What changes between MMA and AMA?

MMA has its roots in SCOM (System Center Operations Manager) and the legacy Azure Operations Management Suite (OMS) solution. In contrast, the AMA is a ‘from scratch’ cloud-based model using hyperscale features available in the Azure space. Figure 1 illustrates the significantly changed architecture from the MMA to the AMA models.

Figure 1 – MMA (left) and AMA (right) solution policy and data models side by side demonstrate a new way to do familiar things.

Architecture changes

In the MMA model, the Azure Log Analytics (ALA) Workspace had Solutions installed, and when linked to an Azure Automation (AA) account, the solutions installed in the AA account as well. These MMA and AA-based solutions were pushed to all the MMA agents connected to the workspace. ALA-based solutions (now called Legacy solutions), consist of (1) one or more data types like logs to collect, and (2) monitoring artifacts like dashboards associated with the data type.

The AMA modality does not depend on the “ALA solutions” lash-up of the MMA to collect and process data. The functions that ALA solutions accomplished are moved from the ‘monolithic’ ALA/AA+MMA model to the new AMA granular model that uses a collection of new artifacts named Data Collection Rules (DCRs), Data Collection Endpoints (DCEs), and Data Collection Rule Associations (DCRAs) that distribute the ‘solutioning’ down to the level of every managed computer.

Consider this example on the differences when you want to collect Windows IIS web server logs (Figures 2 and 3):

• Using the MMA, the collection of web service logs on all Windows servers with IIS installed is enabled by ticking the box at Azure Log Analytics -> Legacy Agents Management “Collect W3C format IIS log files”. This can be an expensive setting in a large environment and for that reason, created scaling issues with customers.
• With the AMA, you associate a DCE with a DCR that collects IIS Logs. Then you create a DCRA that associates the DCR with one or more specific web servers that you want to collect web logs from. With the flexibility to only ingest web server logs from selected high-risk or business-critical servers, the organization saves money and gains insight into valuable data.

   

  

  Figure 2 – Collecting web server logs using the MMA is turned on for all servers in the Log Analytics workspace.
  

  

  Figure 3 – Collecting web server logs using the AMA is turned on using a DCR in Azure Monitor and then creating one or more DCRAs for specific (or all) servers.
  

From the cloud architect and administrator perspective, complexity in the monitoring space is added by distributing and making granular the data collection pipeline. However, the changes must be regarded as healthy since they do make Azure as a monitoring platform even ‘cloudier’. The ability to assemble a curated selection of cloud microservices, configure them exactly to your needs, and pay exactly the costs associated with that solution are defining economic benefits of cloud computing.

Performance and configuration changes

Without requiring backward compatibility with SCOM, the AMA is by design more efficient at cloud communication, and the AMA features an ‘events per second’ event forwarding advantage of 25%-30% compared to the MMA.

A huge benefit of the AMA compared to the MMA is that the AMA allows for ‘pre-ingestion filters’ called Transforms that can be applied to any data type. This means you can filter-out log types of low interest or value that you don’t need to upload to Azure. You can also target custom or expensive logging to just those servers that are business-critical and/or have high security value. This feature alone can save a good deal on ingestion costs by avoiding the cost of over-collecting.

Azure Automation Configuration Management Tools

The update management, change tracking, and inventory solutions (installed in AA, that depend on the MMA) migrate to ‘stand-alone’ Azure microservices that don’t require the AA. The exact same type of data collection is done using the AMA that was done with the MMA, you just need to know the new locations in the portal to go to specify the data you want to collect.

Consider the following example of configuring change detection on Windows servers, in this case Windows Registry branches for surveillance by Change Tracking. Figure 4 shows this setting–in Azure Automation workspace configuration—that is applied globally to all connected computers in the legacy MMA model. Figure 5 exposes the same configuration data but now in the context of a Data Collection Rule (DCR), which can be granularly associated with specific (or all) computers in the new AMA model.

Figure 4 – Selecting which Windows Registry branches to monitor for changes using the MMA model (Azure Automation-based solutions)

Figure 5 – Selecting which Windows Registry branches to monitor for changes using the AMA model (DCR-based solution)

For customers that have come to depend on the update management, change detection, and inventory solutions in Azure Automation, it’s good news the same reliable data and management decision products are available in the new Azure native tools. A characteristic of the new Azure tools is they don’t require the ALA/AA overhead and linkage of their predecessors. Figure 6 is the change tracking display in Azure Automation created using the MMA model. Figure 7 exposes the same change tracking data but now in the context of the Azure Change Detection and Tracking Center which is based on the AMA model.

Figure 6 – Change tracking on servers in Azure Automation using data collected by the MMA.

Figure 7 – Change tracking on servers using data collected by the AMA and one or more DCRs.

Even when using AA for Hybrid Worker Role jobs on Azure VMs and Azure Arc-enabled servers, no ALA <-> AA ‘linkage’ is required. Rather, the Hybrid Worker Role, update management, change tracking, and inventory functions are accomplished by causing individual VM extensions to be installed by ARM using Azure Policy.

Administrator Experience changes

The administrative user experience (UX) for admins in the AMA era is quite different from the MMA. Consider this list of differences in the administrator UX:

• The MMA is installed as a Windows or Linux application, visible in Windows Control Panel, while the AMA is installed by the Azure Resource Manager (ARM) control plane using Azure VM and Azure Arc-enabled server VM extensions.
• The MMA writes application logs to Windows Event Viewer, AMA logging is not exposed in Event Viewer. [Tip: See this article for AMA agent troubleshooting.]
• In Windows, the MMA is seen running in Services. The AMA is run in the background by ARM using VM extensions (the AMA is not seen in Control Panel -> Services). Also, the Microsoft Management Agent applet in Control Panel is not present for the AMA. [Tip: Run

  tasklist /SVC /FI "IMAGENAME eq MonAgentCore.exe" 

  to see if the AMA agent process is running.]

• The AMA cannot be dual-homed to both SCOM and ALA—the AMA only communicates with Azure (not SCOM as well)—so SCOM customers will continue to use an MMA-type agent. (Both the MMA and the AMA can co-exist on the same computer.)
• There is more use of Azure Policy with the AMA, since the best practice is to create Data Collection Rule Associations (DCRAs) for each data type using Azure Policy.
• There is no longer a need to ‘link’ an Azure Automation (AA) account to the Azure Log Analytics workspace.
• The user-assigned Managed Identity (MI) resource becomes an important part of cloud management, replacing system-assigned service principals for Azure Policy assignment.

Fundamentally, the entire DCR, DCE, and DCRA experience is new for all admins. For this reason, every Azure Monitor (and Microsoft Sentinel) customer needs to start adding familiarity with the new toolkit.

Begin your migration journey in the lab

Regardless of your organization size, it is highly recommended to pilot and test your security solutions using the AMA before making any rollout plans. Definitely don’t experiment with migration scenarios in your production environment. Make sure you understand exactly the implications of each AMA-era artifact before making changes to production.

AMA lab or dev resource group recommendations

Here are suggestions for ways to learn and understand the AMA without impacting production:

• Use a lab subscription and lab resources. (Best)
• In a single subscription (no lab subscription available), create a development (dev) resource group for the purpose of testing and guiding the production deployment.
  • Create a dev instance of Sentinel in the dedicated resource group.
  • Deploy all AMA artifacts only to the dev resource group.
  • Assign policies involving DCR/DCE/DCRA only to the resource group.
  • Locate test/dev Linux and Windows servers in the dev resource group and/or dedicated dev/test resource groups.
• Do temporarily connect each type of production firewall and device to AMA-driven syslog forwarder pools in the lab or dev instance.

Developing the ‘AMA plan’

In developing your final solution in the lab or dev resource group, you will discover:

• How many MIs and DCEs you will need
• What DCRs will align with each DCE
• How to name and where to site MI/DCR/DCE resources
• What Azure policies will you need to assign to create the desired DCRAs
• Security RBAC roles and assignments needed by MIs

These production artifacts can be cloned from the lab subscription or copied from the dev resource group to their production locations, allowing for a low risk and rapid cut-over from the MMA to the AMA in production.

To avoid: Co-mingling of experimental or test Azure policy and DCR content with production assets. Only deploy in production AMA-era artifacts that are part of your AMA plan.

Detailed information about the AMA-related resources

Figure 8 illustrates some familiar Azure resources like Azure Policy and VM Extensions, as well as the new AMA-related artifacts (the DCR, DCE, and DCRA) that together create a highly scalable management pipeline:

Figure 8 – This diagram illustrates the relationship between AMA-related artifacts, as well as documents the management resource pipeline in the AMA era.

User-assigned Managed Identity (MI)

The first thing you should consider when planning for enterprise AMA migration involves the Azure resource known as the user-assigned Managed Identity (MI). An Azure Managed Identity in the context of a VM is a cloud-native service principal that is either system-assigned or user-assigned. A system-assigned MI is tied to the lifecycle of the resource it was created for, can only be used for management tasks with that resource, and is deleted when the resource is created, much like the Computer account in Windows Active Directory.

User-assigned MI avoids the sprawl of system-assigned MIs

A user-assigned MI can be used for multiple tasks on multiple systems and can have an extended lifetime. Both types of MI are in fact service principals of a special type which can only be used with Azure resources. Each Azure VM creates a system-assigned MI, and many assignments of Azure Policy create a service principal in the Entra ID directory for every policy assignment.

Azure administrators may also be familiar with the creation of system-assigned service principals (SPs) when assigning an Azure policy to a resource group or subscription that includes remediation actions. An unsatisfactory condition occurs over time using system-assigned SPs. When that service principal is assigned Azure role permissions via Azure Policy, the security permissions on delegated Azure subscription(s) and/or resources groups are not cleanly deleted when the source Azure policy assignment and service principal (SP) is deleted (see Identity not found in Figure 9). Therefore, many organizations find they are experiencing “MI sprawl” from hundreds or thousands of system-assigned MIs and/or “SP shrapnel” from orphaned Azure RBAC delegations on Azure resources.

Figure 9 – “Service principal shrapnel”: Orphaned RBAC role assignments from inactive Azure Policy assignments (“Identity not found”)

User-assigned MI opportunities

Moving to the user-assigned MI model for VM Identity and for Azure Policy remediation tasks can happen co-incident to your migration from the MMA to the AMA. All you need to do to start benefitting from the streamlined administration of user-assigned MIs are these steps:

1 – Pre-create user-assigned Managed Identities (like jjnet-PolicyAssignmentMI-eastus in this example).

a – Grant the MI the minimum necessary Azure role assignments to perform the complete scope of intended actions for the MI, such as running Azure Policy remediation actions to create DCEs, DCRs, and DCRAs.

b – Figure 10 illustrates all the role assignments necessary when all the Azure policies used to configure DCRs for Microsoft Sentinel are assigned only to a specific resource group. These three (3) roles assigned as indicated are the minimum privileged roles and all the other role assignments seen in Figure 10 are secondary:

– Azure Connected Machine Resource Administrator (required at Resource Group, preferred at Subscription)
– Contributor (required at Subscription)
– User Access Administrator (required at Resource Group, preferred at Subscription)

c – Notice that some permissions are required at the subscription level even when the assignment is only to the resource group level.

d – Smaller organizations and lab/dev deployments can leverage a single MI for all scenarios.

e – Large and distributed organizations can follow this guidance on best practices for creation of multiple MIs.

Figure 10 – The Managed Identity blade in the Azure portal for a user-assigned MI that will be enforcing Azure policy assignments.

2 – Assign VMs to use the MI through Azure Policy (Figure 11).

Figure 11 – Configuring an Azure VM to use a pre-defined user-assigned Managed Identity (MI).

3 – Configure Azure Policy remediation actions to use the user-assigned MI when assigning the policy or initiative (example in Figure 12).

Figure 12 – Selected a pre-created user-assigned Managed Identity as the service principal to perform Azure policy remediation actions.

Azure Policy

The recommended best practice approach is to use Azure policies and initiatives to error-proof the AMA management pipeline. Azure policies can create DCEs, create DCRs, and associate DCRs with computers (DCRAs). You can manage your AMA-based management pipeline through the well-understood Azure Policy compliance framework. Figure 13 illustrates how a single Azure policy initiative can assign a data collection policy to thousands of Windows and Linux Azure VMs at once using pre-defined user-assigned MI and DCR.

Figure 13 – Configuring an Azure Policy assignment to use pre-created user-assigned Managed Identity and Data Collection Rule (DCR).

While not authoritative, the list of Azure Policy and Initiative resources seen in Figure 14 is sufficient to connect all Windows and Linux servers in Azure (Azure VMs) and outside Azure (Azure Arc-enabled servers) to all best practice Microsoft Sentinel data sources.

Figure 14 – Azure Policies assigned to the resource group containing Windows and Linux servers to be connected to Microsoft Sentinel.

Notice the policy with the green star: Configure Windows Machines to be associated with DCE. If all your computers requiring DCEs exist in the same Azure region, you may only need one pre-configured DCE. Otherwise, you will need to create a DCE in each region where you have Azure VMs and/or Azure Arc-enabled servers and create additional Azure policies to push out the geographically correct DCEs to your computers.

VM Extensions

Azure administrators with server assets are familiar with VM Extensions, they are a common management tool for both Azure VMs and Azure Arc-enabled servers. However, the number and variety of extensions required in the AMA model is larger and more complex than was true using the MMA model.

Extensions using the MMA

Figure 15 is a screenshot of the Extensions menu of an Azure Arc-enabled domain controller computer using the MMA. WindowsAgent.AzureSecurityCenter is the Qualys vulnerability scanner deployed by Defender for Cloud, MDE.Windows indicates protection by Defender for Endpoint deployed by Defender for Cloud, finally the MMA and Dependency Agent (DA) indicate the computer is monitored by VM Insights.

Figure 15 – Typical extensions seen on a computer connected to Microsoft Sentinel and Defender for Cloud using the MMA.

Extensions using the AMA

Recall that value-add solutions servicing computers using the MMA were installed in the Log Analytics workspace and optionally in a linked Azure Automation account. For this reason, a smaller number of VM extensions were necessary. Value-add solutions and tools delivered via the AMA require a DCR and DCRA to service a computer. DCRAs may create VM extensions with specific purposes.

Figure 16 is a screenshot of the Extensions menu of an Azure Arc-enabled domain controller computer using the AMA. New extensions from the MMA-era server seen in Figure 15 are highlighted. MicrosoftDnsAgent replaces the DNS solution that was installed in the Log Analytics workspace to forward DNS logs to Microsoft Sentinel. ChangeTracking-Windows and WindowsPatchExtension replace the solutions installed in AA and previously linked to the ALA workspace.

Figure 16 – VM extensions installed on an Azure Arc-enabled server which is a Windows domain controller and managed with the AMA.

Data Collection Endpoints (DCE)

Data Collection Endpoints (DCEs) are AMA-specific resources created within specific regions. A DCE in a given region can only be associated with machines in the same region. However, you can have more than one DCE within the same region according to your needs. DCEs are required in the following two circumstances:

• Azure Monitor Agent when network isolation is required
  
• Logs ingestion API
  

Which DCRs require DCEs?

Only some Microsoft Sentinel default best-practice DCRs require a DCE, and they fall into the Logs ingestion API category. Some specific DCRs send data to an ALA workspace using either a REST API call or client libraries and thus require a DCE. Most DCRs don’t require a DCE however these relevant data sources do:

• IIS Web logs
• Windows Firewall logs
• Defender for Cloud workload protection for SQL Server

A handy means to verify which DCRs require DCEs is exposed in the Microsoft Sentinel Optimization Workbook. Figure 17 is a screenshot of the Data Collection Rules & Ingestion Transformations tab from this workbook. If any DCRA associated with a computer requires a DCE, you must create a DCE and assign it to the computer. Since all Windows computers are in scope for Windows firewall, and some will be in scope for IIS web logs and Defender for SQL Server, every Windows computer is going to be assigned a DCE.

Figure 17 – The Data Collection Rules & Ingestion Transformations tab from the Microsoft Sentinel Optimization Workbook clearly shows which DCRs require a DCE.

DCE Configuration vs. DCE Data Transfer functions

If you examine Figure 8 carefully (the AMA management pipeline) you will observe there are two purposes for a DCE:

• If a computer is assigned to any DCE, that computer will check the DCE for configuration when DCRAs are evaluated.
• If a DCRA requires a DCE, data collection will occur at the DCE. (DCRAs that don’t require a DCE communicate directly via the AMA after checking the DCE for configuration.)

This information will explain why DCRAs that don’t require a DCE can still appear to point to a DCE as seen in Figure 18. Bottom line: You will need to create DCEs in each region you have Windows computers.

Figure 18 – Even if a DCRA does not require a DCE, any DCE assigned to a computer will cause all DCRAs on that computer to consult the DCE for configuration before transmitting data.

Data Collection Rule (DCR)

The DCR is the star of the AMA-era show. DCRs replace all previous models of ingesting logs to Azure Log Analytics workspaces. They underpin all modern Azure-based server management technologies. DCRs determine how to collect and process telemetry sent to Azure for monitoring and connection to Microsoft Sentinel.

Curate your DCR Library

You will be developing a library of DCRs to assign computers to data types and destinations. Here are some tips on DCR management:

• Create granular DCRs, that is, DCRs that collect a single data type, unless otherwise specified. To avoid duplication of collection, only combine multiple data types when there is an overriding reason. Otherwise, keeping one data type per DCR makes it easy to troubleshoot and avoid conflicts.
• DCRs that involve a specific DCE or region will require duplication or cloning each set of assignments to other regions/resource groups since MIs and DCEs are region-specific.
• If you have a single Log Analytics workspace and computers in multiple regions, you create a single data collection rule (DCR) in the region of the Log Analytics workspace. There is no 1:1 correspondence between the number of regions containing data sources and the number of DCRs–you create a single DCR that sends data from multiple machines in different regions to the same workspace by specifying the target table and transformation in the DCR.
• You will be duplicating and customizing the policy “Configure Windows (or Linux) Machines to be associated with DCR x” to push out DCRAs for any log type that Microsoft does not provide a built-in policy for.
• Try and document this AMA management pipeline for each data type needed:

   

  • User-Assigned Managed Identity enforces -> Azure Policy which might create -> VM Extensions as well as assign -> Data Collection Rules which create -> Data Collection Rule Associations that may connect to Data Collection Endpoints
    

Table 1 from the demonstration environment covered in this article can get you started on keeping these artifacts lined up:

Table 1 – DCR to Azure Policy Assignment map: The beginnings of your DCR Library. (Assignments with blue background are ‘custom’ definitions, all others are ‘built-in’.)

Be able to read DCRs

Another (and not difficult) skill to cultivate is being able to crack open the JSON resource record of a DCR and read what the DCR is all about. For example, the default name of the DCR created by Azure policy initiative Enable Azure Monitor for VMs with Azure Monitoring Agent(AMA) starts with the string “MSVMI-PerfandDa“. If you don’t understand what that means, you could open the Resource JSON of the DCR as seen in Figure 19. It’s not tough to decipher that MSMVI refers to Microsoft VMInsights and that Perf and Da refer to Performance counters and installation of the Dependency Agent extension.

Figure 19 – A DCR in the Azure portal: Looking up the Resource JSON to understand what the DCR does.

Data Collection Rule Association (DCRA)

The DCRA is ‘where the rubber meets the road’. It’s the actual connection of a DCR to a specific computer and is what causes data collection to commence. DCRAs can be created manually or programmatically (such as via the API) to single computers. They can also be deployed to many computers in one action by assigning Azure policy. Regardless of how the DCRA is created, the computer appearing in the list of resources added to the DCR as previously seen in Figure 18 is authoritative.

Review: High-level sequence of events to migrate to AMA

The starting point for this plan is a Microsoft Sentinel instance configured as follows:

• Windows and Linux servers in Azure and Azure Arc-enabled settings (MMA model)
• Firewalls delivering syslog and/or CEF logs (MMA model)
• Defender for Servers P2 enabled and connected
• Defender for SQL on Machines enabled and connected
• Defender for Endpoint extension assigned to all computers
• Linux servers forwarding syslog, some configured for CEF forwarding
• Windows servers forwarding System, Application, and Security logs
• Windows servers forwarding Windows Firewall logs
• Windows DNS servers forwarding DNS log
• Windows servers running IIS forwarding web logs
• Windows and Linux servers participating in Azure Automation-linked solutions: Update management, Change detection, and Inventory
• Windows and Linux servers configured for VMInsights including network/process mapping

Roll out the migration from MMA to AMA in these broad steps:

1. Develop user-assigned MI strategy to include regional and organizational considerations.
2. Develop DCE plan to support indicated regions.
3. Develop DCR plan to collect necessary data types.
4. Identify Azure policies and initiatives that deploy MIs, DCEs, DCRs (curate your DCR library)
5. Stage MIs, DCEs, and DCRs in production but don’t assign yet.
6. Make sure Microsoft Sentinel connectors for AMA-based data sources are installed.
7. First migrate the Linux VMs to AMA that are serving as syslog forwarders for firewall logs (or replace them with AMA-based forwarders).
8. Assign DCR and DCE policies to production server resources (creating DCRAs).
9. Verify Microsoft Sentinel is properly ingesting and parsing data from the AMA-based resources.
10. When parity with data from MMA model is verified, cause MMA to be uninstalled from computers.
11. Delete MMA-era connectors from Microsoft Sentinel.

Time’s a wasting!

Hopefully this article has opened some eyes as to what tasks are really involved in successfully completing the MMA to AMA migration in a best-practice Microsoft Sentinel and Defender for Cloud scenario. The larger and more complex your environment, the more planning, preparation, and internal communication will be necessary to develop, test, document, and transfer to production an AMA-based management pipeline. We all have until August 31, 2024!

I will be presenting a 90-minute deep-dive session at Live! 360 Events November 12-17, 2023 Royal Pacific Resort at Universal Orlando, Florida:

https://live360events.com/Events/Orlando-2023/Sessions/Wednesday/CRW06-Regulatory-and-Security-Compliance.aspx

We will cover MMA to AMA migration for Microsoft Sentinel customers in detail. Read up about this conference, there are a vast number of sessions across the six (6) co-located Live! 360 events.

 If you are a stakeholder in a hybrid IT estate, you owe it to yourself to learn about Azure Arc, get a copy of my book: (co-authored by Microsoft’s Steve Buchanan)

Azure Arc-Enabled Kubernetes and Servers: Extending Hyperscale Cloud Management to Your Datacenter

https://www.amazon.com/Azure-Arc-Enabled-Kubernetes-Servers-Hyperscale/dp/1484277678

 Tags: #MVPBuzz #MicrosoftSentinel #AzureMonitor #HybridCloud #CloudSecurity #DefenderForCloud #SQLServer #AzureArc #AzureHybrid #DefenderforSQL