A common and useful concept in management applications is to distinguish between (1) built-in, non-changing monitoring settings and (2) custom, dynamic exceptions to the standard settings. This article details methods to use existing Azure features to achieve this operational necessity.
Objective: Keep standard rulesets and environment-specific exceptions separate
Once you are using Azure Monitor and Azure Sentinel rules at scale, you realize you need to implement some standards of usage and naming (think taxonomy and lexicon governance) that guide rule management. One challenge to consider is keeping reusable standard rulesets separate from the custom exceptions to standard rules. For managed services providers, this maps to (1) cross-environment and (2) customer-specific information being segregated and portable.
In the Microsoft System Center model, standard settings are contained in sealed management packs (.MP files) that apply to all environments. Overrides to standard rules and monitors are saved in unsealed management packs (.XML files) that generally apply only in their specific environments. This protocol keeps the two types of controls (standard and custom) separate by design.
While Azure has no native concept of management packs, Azure Monitor and Azure Logic apps provide means of keeping these two kinds of data separate. All Azure monitoring artifacts, ‘standard’ and ‘custom’ can be exported and imported as Azure Policy (.JSON files). The basics are there: separation of control types as well as control portability.
Standardizing on modern cloud-based monitoring tools like Azure Monitor and Azure Logic Apps (and Azure Function Apps) is possible with a strategy to manage alerts and overrides at scale. Standard monitoring policies can reside in secure and universal GitHub or Azure DevOps repositories and be shared across environments, while custom policies are supported and can be authored as needed using a protocol you specify in your Azure governance controls.
Azure Log Analytics is a log provider to Azure Monitor, along with other Azure services like Application Insights. For the purposes of this article, alert rules, action groups, and action rules in these services function identically. On the other hand, Azure Sentinel leverages only Azure Logic Apps for alert actions, so there is a different method to define monitoring overrides to Azure Sentinel that is also explained in this article.
Brute force: Override an Azure Monitor rule by specifying exceptions in the search query
Consider an Azure Monitor alert rule that detects when a new Azure resource group is created in a subscription. “Resource group sprawl” is a real thing, and Azure admins need to be aware of when a new resource group is created so they can vet the creator (caller) and purpose. This is a valid governance control for any organization. The following search query works as a condition to detect resource group creation:
AzureActivity |
where OperationNameValue == "Microsoft.Resources/subscriptions/resourceGroups/write" |
where Properties contains "Created"
An alert from this monitor would look like the email seen in Figure 1.
Figure 1 – An Azure Monitor alert lists the actual search query that produced the alert.
Let’s say you don’t want to get alerted when you personally create a resource group. That is, you want to override the rule (and not alert you) when your name is the caller. Modifying the alert rule like this (to include ‘your.name@company.com’ in this example) works, but is not the most elegant method:
AzureActivity
| where OperationNameValue == "Microsoft.Resources/subscriptions/resourceGroups/write"
| where Properties contains "Created"
| where Caller != "your.name@company.com"
Reasons you want to avoid this method include that the ‘override’ criteria (in this case your name) becomes part of the search query, so the criteria gets stamped into the alert summary, possibly causing confusion when someone reads the alert. Also this method doesn’t scale well in a large organization, since a lot of ‘overrides’ in the search query can quickly obscure or even impede the intended purpose of the alert rule.
Better: Override an Azure Monitor rule by creating a suppression Action rule
Azure Monitor Action rules allow for granular control of notifications, automated actions, suppression, or diagnostics for fired alert instances. Azure Monitor Action rules are separate from Azure Monitor Action groups—You access both Action groups and Action rules at the Azure Portal at Azure Monitor -> Alerts -> Manage actions button. Figure 2 shows where to access the Action rules area of the portal.
Figure 2 – Azure Monitor rules management: Action groups and Action rules.
Current users of Azure Monitor are familiar with Action groups. If you ever got an email or other alert action out of an Azure alert rule, you used Action groups. Action groups contain lists of automatic actions to take such as email someone, create a ITSM ticket, or take action like launch a runbook or recovery script. There are three Azure monitoring artifacts in play to achieve the objective:
1 – The Azure Monitor Alert rule that runs a search query, and when an alerting condition is discovered, calls selected Action groups
2 – One or more Azure Monitor Action group(s) specified by a rule that fires, performing notifications and taking remediation actions
3 – An optional Azure Monitor Action rule, configured to suppress activation of the normal Action group when certain (override) criteria are met
Create an Action suppression rule to override an Alert rule’s Action group(s)
Returning to the example use case (to not alert when a specific user, such as yourself, creates a new Azure resource group), consider that you need some information to create the ‘override’ Action rule. Specifically, you need to know the Alert Rule ID of the Azure Monitor Alert rule you want to override, and a text string from the Alert context (payload) that indicates the presence of the overriding condition. Figure 3 shows how you enter this information when adding filters to the suppression rule.
Figure 3 – Associate an Azure Monitor Alert rule with an Alert context (payload) to trigger suppression of an Azure Action group.
Finding the Alert rule ID depends on the type of alert you are overriding, for example, a log alert, a metric alert, or in this case an activity log alert. You can use the Azure PowerShell command Get-AzActivityLogAlert to find the value of the Id attribute as seen in Figure 4. An Azure Monitor Alert rule ID consists of a long canonical string that includes your Azure subscription, Azure resource group, and Azure Monitor rule name.
Figure 4 – Using Azure PowerShell to learn the Id of the Azure Alert rule to override.
After saving your Azure Monitor Action suppression rule, you can see it listed as Enabled in the Action rule list (see bottom of Figure 2). All future alerts arising from the specified Azure Monitor alert rule will have the Action rule filter applied, and if there is a match, the actions specified in the Azure Monitor Action group are suppressed. The Azure Monitor Alert still fires and appears in the portal, but no actions occur other than creating the alert.
You will notice you don’t receive an email or other normal alerting action. To verify your action override is working properly, locate and select the alert that fired in the Azure Monitor alerts view, and push the History button as shown in Figure 5. Observe the suppression action occurred immediately after the alert fired.
Figure 5 – The Azure Monitor Alert History verifies the desired suppression workflow occurred.
Exporting Azure Monitor Action Rules as Azure Policy objects
The Action rules you create as overrides can be exported for separate backup, storage, and transfer by extracting the portion(s) of the Azure Resource Group definition using the Export template feature of the Azure portal. A community solution to do this conveniently consists of installing the Chocolatey package manager on your Azure PowerShell admin computer, then using Chocolatey to install the ARMClient Azure API command line tool by David Ebbo. Figure 6 shows the complete definition of the action rule override, and here is the code that extracts the Azure Monitor Action rules from a specific Azure resource group (courtesy @David2075, updated).
$subid='<Your Azure Subscription>'
$rsgname = '<Your Resource Group>'
armclient get /subscriptions/$subid/resourcegroups/$rsgname/providers/Microsoft.AlertsManagement/actionRules/?api-version=2019-05-05-preview
Figure 6 – The complete Azure Monitor action suppression rule, captured as an ARM template.
TIP: Extract the caller name from an Azure Activity event
You may want to extract just the caller’s name (such as “your.name@company.com”) from an Azure Activity event for additional processing. This query will return a text string containing the caller’s full name (UPN). Modify or remove the OperationName as appropriate:
AzureActivity
| where OperationName == "Update resource group"
| extend UPNUserPart = extract("([a-z.]*)@", 1, Caller)
| distinct UPNUserPart, Caller
Applying Overrides to Azure Sentinel Incident Rules
One of the most exciting cloud services in 2020 is Microsoft’s new cloud-native SIEM (Security Information and Event Management) tool, Azure Sentinel. Much like Azure Monitor Alert Rules, Azure Sentinel Analytics Rules take the form of timed search queries that create alerts when query conditions are met. For example, Figure 7 shows a custom Azure Sentinel analytics rule that alerts when client computers request DNS lookups of domain names that are known by Microsoft to be hostile.
Figure 7 – An Azure Sentinel analytics rule works just like an Azure Monitor alert rule.
This is a very valuable alert for the security professional because it can represent the first indication of an imminent hostile attack or malware incident. Because this alert rule maps event attributes (like IPAddresses) to incident entities (like IPCustomEntity), all the sophisticated investigation and machine learning (ML) features of Sentinel can be brought to bear.
However, in practice, it may emerge that domains needed by your users that are listed in the malware databases are in fact no longer hostile. You can imagine that a Security Operations Center (SOC) might develop a list of such domains and want to override Sentinel Automated response to this rule to avoid false positives. A method to achieve this is by conditional branching inside the Azure Logic App that is invoked by the Azure Sentinel rule when an incident is created.
Create Workflow within a Logic App to Override Sentinel Automated response
The Azure Sentinel console features a Playbooks space where you can create and manage a collection of Azure Logic Apps that achieve automated and/or on-demand workflow on Sentinel incidents (learn more here). For each Azure Sentinel timed query analytics rule, there is an Automated response page where all the playbooks you have defined are listed. Selecting a playbook (seen in Figure 8) will cause the Logic App to fire automatically when the incident is created.
Figure 8 – Azure Sentinel automated responses are Azure Logic Apps known as playbooks.
The most basic Azure Sentinel playbook performs a simple Send an email action when an Azure Sentinel alert is triggered as shown in the Logic Apps Designer at Figure 9. Notice that you specify everything about how the email looks by pasting Sentinel alert dynamic content like Entities and Alert display name on the designer canvas.
Figure 9 – A basic Azure Sentinel playbook is a Logic App that sends emails for every Sentinel alert.
The resulting email alert when the DNS Client Lookup to a Hostile Domain occurs can be seen in Figure 10.
Figure 10 – The email alerting product when the playbook in Figure 9 is called by the analytics rule in Figure 7.
Clone a standard Sentinel Playbook to create a Playbook with custom override workflow
When the need arises to create an override to the standard workflow in an Azure Sentinel automated response, open the existing playbook at Azure Sentinel workspaces -> Azure Sentinel -> Playbooks and push the Clone button. Try and create a playbook that has usefulness beyond just the current alert, that is, consider constructing the override such that it can be reused across a class of similar incidents.
1 – To override this analytics rule to not fire alerts when specific ‘white listed’ DNS domains are involved, name the new rule Sentinel-Incident-With-IP-Whitelist. (In this alert the IP address of the hostile DNS domain is mapped to the IP entity. The playbook will work to suppress sending alert emails for all types of Sentinel incidents with the IP entity mapped, so this playbook can be used with many possible alert rules.)
2 – Click on the arrow between the first and second step in the Logic Apps Designer (refer to Figure 9) and select to Add an action. Choose the Alert – Get IPs component and select the Entities dynamic content for the Entities list.
3 – Select Add an action again and select the Control -> For each function, then select the IPs dynamic content for the output from previous steps.
4 – Select Add an action again and select the Control -> Condition function, then selecting the IPs Address dynamic content, the contains expression, and enter the IP address to be whitelisted.
5 – In the If true component, don’t enter anything. In the If false component, select to Send an email using the same settings as the original Logic App playbook.
6 – Delete the orphaned Send an email component left over from cloning the original Logic App.
7 – Push the Save button and navigate to the Analytics page. Select the Active rule you created the new playbook for and push the Edit button.
8 – Advance to the Automated response page, select the new custom playbook, and un-select the previous standard playbook. Finally, push Next and then Save.
Figure 11 shows the completed Logic App (playbook). Once this playbook is specified as the automated response to any Azure Sentinel incident, if the IP entity field contains the whitelisted IP address, no further actions are taken. The Azure Sentinel Incident still appears in the portal, but no actions occur other than creating the incident.
Figure 11 – A logic app (playbook) that performs for-each looping and conditional checking.
You will notice you don’t receive an email or other normal alerting action when the white-listed domain is encountered. To verify your action override is working properly, navigate to Azure Sentinel -> Playbooks and open the playbook of interest. On the Overview page, at the bottom is the Runs history. Select the run you want to verify and push the Run Details button. Observe in Figure 12 that the Send an email step was skipped because the checked condition (if the IP in the alert is whitelisted) was true.
Figure 12 – Logic app run details document the proper function of the Azure Sentinel rule override.
Exporting Azure Sentinel Logic App Playbooks as Azure resource templates
The Logic Apps you create that include custom overrides can be exported for separate backup, storage and transfer using the Export template feature of the Logic App. Figure 13 shows the complete definition of the Sentinel playbook action override ready to download as a ZIP file. The ZIP will contain a Parameters.JSON file and a Template.JSON file.
Figure 13 – Exporting a custom Logic App JSON template that contains reusable workflow components.
Summary: A high-level governance plan for Azure Monitor and Azure Sentinel rule overrides
This article has stepped through examples of how you might establish governance for overrides in your Azure subscription(s). Here are the key concepts:
- Azure Monitor alert rules are reusable across environments and may reside in a code repository, they don’t incorporate custom overrides.
- Azure Monitor action group overrides exist as Azure Monitor action rules, these can be exported separately from the alert rules they apply to.
- A small number of Azure Sentinel playbooks with standard components contain repetitive workflow that apply to most or all Sentinel incidents.
- When custom overrides to Azure Sentinel playbooks are required, clone the source logic app and add For-each and If-then conditional branching steps.
- Both Azure Monitor rule and Azure Logic App artifacts can be cleanly exported from an Azure subscription for governance purposes.
Tags: #MVPBuzz #AzureMonitor #AzureSentinel #LogicApps #securityManagement #SIEM #Governance #sysctr