When your apps all ran on-premises, servers only received inbound Internet traffic if you specifically put them behind a firewall and enabled that kind of connection. The opposite is true of most PaaS services like web servers and cloud storage which operate by default over the Internet, usually accessed over HTTPS URLs, like https://myfileshare.file.core.windows.net for a file share storage account.
To provide secure access to PaaS services accessed over the Internet, application-specific precautions like restricting incoming source IP addresses to a web server, or using a shared access signature to read a storage account are often employed. As cloud computing evolved, the facility to connect PaaS services to virtual networks emerged. Doing so takes the services off the public Internet, and vastly streamlines the task of restricting unauthorized network access.
If your organization is wisely using the Regulatory compliance features of Microsoft Defender for Cloud, you may have noticed recommendations about enabling Private endpoints on your Azure PaaS services like web servers and storage accounts (Figure 1). If you have not used private endpoints before, this might seem a daunting security control to implement. This article will guide you through migrating Azure web apps, databases, and other Azure PaaS services to a highly secure design that significantly reduces your risk exposure to Internet cyberthreats.
Figure 1 – Microsoft Defender for Cloud recommendations to enable private endpoints.
Azure Network Security
If you want to do a complete job of implementing security best practices for your Azure applications, you might start by reviewing Azure security best practices and patterns. That’s a great plan, but you won’t find information about deploying private endpoints in the sections on securing PaaS services like Azure App Service and Azure Storage. That’s perhaps because the disciplines to implement private endpoints reside mainly in the networking space—these are not security controls that most application owners can implement within their domains.
Organizational readiness for Private endpoints
You need to drill into the Azure best practices for network security documentation, in particular the section: Secure your critical Azure service resources to only your virtual networks:
“Use Azure Private Link to access Azure PaaS Services (for example, Azure Storage and SQL Database) over a private endpoint in your virtual network. Private Endpoints allow you to secure your critical Azure service resources to only your virtual networks. Traffic from your virtual network to the Azure service always remains on the Microsoft Azure backbone network. Exposing your virtual network to the public internet is no longer necessary to consume Azure PaaS Services.”
Central IT sponsorship, and collaboration with all PaaS and IaaS application owners in the estate, are necessary to create the environment to properly migrate to private endpoints—it’s not a trivial undertaking. It can mean a revisit to Azure networking architecture, and even a re-architecture of that networking in order to properly use private endpoints. Key takeaway: Private endpoints are all about Azure network architecture and Azure network security design.
Private Endpoints cost more
Another consideration for moving to private endpoints is budgetary. Public-facing PaaS services generally incur no networking charges to access, and they work with the lowest cost/entry-level SKUs. Moving Azure PaaS services to private networks will incur new costs:
- Private endpoints themselves consume fixed and variable-based costs (Example rates start at $7.20 per month per 24×7 private endpoint, $0.50 per month per private DNS zone).
- Egress and transit charges for data that now has to follow a private network path from the regional Azure cloud to endpoints in other Azure regions and users anywhere.
- Some Azure PaaS services only support private endpoints in their more costly offerings (SKUs beyond entry-level). For example, Function apps must be deployed to ‘premium plans’ rather than less expensive ‘consumption plans’ in order to natively support virtual networking.
Bottom line here is that you are re-creating–in the public cloud–the networking conditions of an on-prem network/private cloud–over which you have complete control and complete exclusivity. It’s more costly because you are replacing physical network components, that would cost thousands of dollars, with equivalent virtual networking structures that span the hybrid estate. 100% private network access is not a design feature of a lowest cost, commodity public cloud solution. There is no free lunch when going private.
Private endpoints are like NICs for PaaS services
Consider Figure 2, which is a screenshot of the diagram view of an Azure ‘spoke’ virtual network with private endpoints deployed.
Figure 2 – Private endpoints of PaaS services appear as network interfaces in the VNET map.
In the virtual network diagram seen in Figure 2, there is an Azure VM on the left side that has only a private network interface. In the center and on the right are the private endpoints of two Azure services, a Web app and an AKS container registry. The PaaS services with private endpoints cannot be accessed over the Internet. However, the Azure VM can access the PaaS services using the IP addresses of their private endpoints.
Consider App Gateway for Web apps
If your organization hosts HTTP/HTTPS-based Azure apps like Web sites and Function apps that need to be accessed over the Internet, you can’t go ‘fully private’. But there is a secure and compelling solution available that allows for the back-end servers and databases to have private endpoints, and still make the app available over the Internet. This is the Azure App Gateway, a web traffic load balancer and HTTP(S) full reverse proxy. It’s a best practice where additional security of an Azure app is required—by providing dedicated tenancy, rather than shared tenant App Service Plans. Optionally app gateway can perform reverse-proxy Secure Socket Layer (SSL) encryption and decryption, or add the Web Application Firewall (WAF) feature to inspect web traffic (including geo-blocking) and detect attacks at the HTTP layer.
Figure 3 is an Azure reference architecture for secure management of Azure apps that shows how App gateway (lower, center) is the endpoint for traffic inbound from the Internet (lower left).
Figure 3 – Architecture to securely manage web applications using Azure App Gateway. (Diagram: Microsoft)
The diagram in Figure 3 also helps illustrate other key concepts covered in this article:
- In the upper left, employees gain access via VPN or ExpressRoute to the app environment (upper, center).
- Public access to resources in the app environment would occur via the App Gateway WAF.
Connecting to Private endpoints
You must have a routed private connection to the virtual network where the private endpoints are to access the PaaS services. You can achieve that private connection three ways:
- Via an Azure VM with a network interface on the same virtual network (as shown in Figure 2). Connect securely to the VM via RDP or SSH using Azure Bastion.
- Using an Azure virtual network gateway and point-to-site (P2S) client VPN.
- Using an Azure virtual network gateway and a site-to-site (S2S) Azure IPSEC tunnel or ExpressRoute gateway.
If your applications and users don’t have one or more of those connection methods deployed, you are not ready to implement private endpoints. For example, if you don’t have a hub-and-spoke or similarly planned Azure network topology, you might need to redeploy Azure resources into new virtual networks and subnets that conform to design best practices. If you need connection from your on-premises offices and networks to the private endpoints, it’s time for investment in a robust and redundant connection using a VPN or ExpressRoute with a Telco/ISP provider. If your users don’t have a way to VPN into your Azure estate, now may be the time to deploy an Azure-connected client VPN solution.
Tip: If you plan to use network security groups (NSGs) for security segmentation of subnets containing private endpoints within a virtual network, make sure you locate your private endpoints in regions that support this feature.
Private DNS for Private endpoints
By default, Azure PaaS services have a DNS configuration for an Internet endpoint with a public IP address. This configuration must be overridden to connect using your private endpoint. For example, an Azure SQL server might have a public FQDN of https://azsql1.database.windows.net which resolves via public DNS to a public IP in the Microsoft cloud. Deploying a private endpoint to that SQL server will result in the SQL server only being accessible over a private IP such as 10.5.0.5. Your challenge: present the private IP to all the services, apps, computers, and users that need to access the storage account.
DNS is a critical component to make the application work correctly by successfully resolving the private endpoint IP address. Microsoft has engineered a novel solution, which is automatic creation of public DNS zone CNAME records for Azure resources with private endpoints. Simply using the default Azure DNS services from a DNS client in a virtual network subnet (as configured in Figure 4) automatically includes any private DNS zones you deployed.
Figure 4 – Specifying the default Azure-provided DNS servers for a subnet automatically includes private DNS zones.
Private DNS zones
Figure 5 shows the Azure portal page when you are creating a private endpoint for an Azure PaaS service, in this example a key vault. Notice that you must select the virtual network and subnet where the private endpoint’s “virtual NIC” will be created, and you must also specify the Azure subscription and location for the private endpoint’s configuration resource. You can’t modify the private DNS zone name, because private DNS zone names are required to match a certain name depending on the resource type (like privatelink.vaultcore.azure.net for key vaults or privatelink.database.azure.net for SQL servers).
Figure 5 – Creating a private endpoint for an Azure PaaS service automatically creates a Private DNS zone of the correct name.
Virtual Network Linking
After you create a private DNS zone in Azure, you’ll need to link a virtual network to it. Once linked, computers hosted in that virtual network can access the private DNS zone. Every private DNS zone has a collection of virtual network link child resources, Figure 6 is an example. Each one of these resources represents a connection to a virtual network. A virtual network can be linked to private DNS zone as a registration or as a resolution virtual network. You can create only one link between a private DNS zone and a virtual network.
Figure 6 – Each private DNS zone is linked to one or more virtual networks, this enables name resolution to private endpoints from Azure VMs in that virtual network.
CNAME records of Private endpoints
When you register a private endpoint for an Azure PaaS service, a DNS ‘CNAME’ record is created in the appropriate DNS zone. The CNAME (alias) records redirect name resolution to a private DNS zone name which corresponds to the private IP address of the endpoint. The private DNS zone name scheme adds ‘privatelink‘ to the FQDN of the PaaS service, i.e., public DNS A record ‘azkv1.vaultcore.windows.net’ is redirected to public DNS CNAME record ‘azkv1.privatelink.vaultcore.windows.net’ which returns the private IP address. The process is completely transparent and non-invasive to existing applications.
DNS client name resolution options
You can use the following options to configure a client’s DNS settings to connect to private endpoints:
Option 1 – Use a HOSTS file (only recommended for testing). You can use the HOSTS file on a computer to override any DNS.
Option 2 – Use a private DNS zone (recommended). You can use private DNS zones to override the DNS resolution for a private endpoint. A private DNS zone can be linked to your virtual network to resolve specific domains.
Option 3 – Use your DNS forwarder (optional). You can use your DNS forwarder, such as an Active Directory Domain Controller or DNS server, to override the DNS resolution for a private link resource. Create a DNS forwarding rule to use a private DNS zone on your DNS server hosted in a virtual network.
DNS configuration scenarios
DNS client setting Option 1 from the list above (using a HOSTS file for testing) is self-explanatory. How to work with client setting options 2 and 3 will now be explained.
Since the FQDN of an Azure PaaS service resolves automatically to a public IP address, you may need to change your DNS configuration to enable name resolution to the private IP address of the PaaS service’s private endpoint. There are three well-defined scenarios to guide you, depending on how your organization is using Azure virtual networking.
Scenario 1: Virtual network workloads without custom DNS server (Client name resolution Option 2, uses Azure defaults, works ‘automagically’)
Scenario 2: On-premises workloads using a DNS forwarder (Client name resolution Option 3, in-Azure only)
Scenario 3: Virtual network and on-premises workloads using a DNS forwarder (Client name resolution Option 3, Hybrid networking model)
The first and simplest of the three scenarios is illustrated in Figure 7. This model works when all the DNS clients reside in Azure and use the built-in/default Azure DNS server services.
Figure 7 – Private DNS name resolution is key to working with private endpoints. (Diagram: Microsoft)
On-premises and hybrid scenarios
You can use the scenario 1 solution when your DNS client can access the virtual public IP address 168.63.129.16 (a communication channel to Azure platform resources). That IP address cannot be routed to from non-Azure networks. If you use the built-in Azure DNS servers as shown in Figure 4, you are using this channel. If your organization needs to support DNS clients outside Azure resolving the CNAME records to the private endpoints of PaaS services, you need to enable a DNS Proxy capability that forwards DNS client lookups for selected DNS zones from non-Azure networks to an endpoint in Azure where virtual public IP address 168.63.129.16 can be reached.
Scenarios 2 and 3 include various models to incorporate the DNS Proxy capability into your network design. Azure Firewall includes a built-in DNS Proxy feature you can simply enable and point your DNS forwarder(s) to the private IP of the Azure Firewall. Alternately, you can deploy an IaaS-based DNS forwarder in Azure, then configure your on-premises DNS solution to forward DNS traffic to Azure DNS via a conditional forwarder that references the IaaS-based DNS forwarder deployed in Azure.
Consider the two Private DNS zones seen in Figure 8. This Azure subscription has at least one Kubernetes container registry (DNS suffix *.asurecr.io) and at least one Azure app (DNS suffix *.azurewebsites.net) with private endpoints.
Figure 8 – Private DNS zones for an Azure subscription with container registry and web app private endpoints.
- Inside each zone are “A” records associating each private endpoint by name with its private IP address. For example, private endpoint name ‘privatelink-webapp-dev’ in private DNS zone ‘privatelink.azurewebsites.net’.
- Azure DNS has CNAME records for each private endpoint in the zone for that Azure service type. For example, CNAME ‘privatelink-webapp-dev.privatelink.azurewebsites.net’ for public DNS name ‘privatelink-webapp-dev.azurewebsites.net’.
- Non-Azure DNS forwarders are set to forward traffic for the public DNS zone to the in-Azure DNS Proxy. For example, public DNS zone ‘azurewebsites.net’ is forwarded to DNS Proxy so that the IP address of CNAME ‘privatelink-webapp-dev.privatelink.azurewebsites.net’ is returned to the client when it asks for ‘privatelink-webapp-dev.azurewebsites.net’
Private Link Center
As you ‘go private’ with more Azure private endpoints, more PaaS network resources surface in the Private Link Center -> Private endpoints view seen in Figure 9. Notice the wide variety of Target sub-resource types, each of which is uniquely identified by a private endpoint name, private IP address, and associated virtual network subnet. When performing the network planning exercises recommended earlier in this article, consider your end state might resemble this.
Figure 9 – The Private Link Center in the Azure portal provides a consolidated view of all private endpoints—each resource is assigned a private IP in a VNET and subnet.
Tags: #MVPBuzz #HybridCloud #AzureNetworking #CloudSecurity #DefenderForCloud #AzureSecurity #PrivateEndpoint