Azure private link / endpoints allow you to connect resources to your private virtual network and with that - when removing public access - shield resources from being accessed or even attacked from the internet. For most of enterprise mission critical systems I help designing and implementing in the cloud, this kind of locked down environment is a hard requirement.
Private link as a way of restricting access to resources only for a defined range of virtual networks is an additional offering for service endpoints which I used so far in projects till spring 2020. This post by Sameera Perera shows the basic differences between these 2 offerings.
When bringing up a new environment I learned that even some resources like Azure Container Registry have a better support for private linking then for service endpoints. Hence I started looking into this other offering to check whether I can achieve a similar or even better behavior.
TL;DR
linking multiple private DNS zones to a virtual network is possible if none or not more than 1 DNS zone has auto registration enabled
(as observed) most resource types do not autoregister into the private DNS zone linked to a virtual network anyway -> manual creation of private DNS recordsets is required
the private DNS zone name is also the resource name and so (if required) the same private DNS zone name can only be created once in a resource group
target setup
The application is deployed in multiple regions (more than 2) across the globe to allow for a certain degree of autonomous operation or even take over operation if a region is down.
global resources
Resources that are globally deployed or replicated hold state or configuration data that is relevant throughout all regions:
- Front Door
- API Management
- Cosmos DB (with multi master write)
- Container Registry
regional resources
Resources that are deployed in individual regions, hold region specific data, process regional data and should be only accessible from within the region:
- Application Gateway to handle ingress from Front Door
- API Management (Gateway)
- AKS cluster
- Storage
- SQL Server
- ServiceBus
- KeyVault
considerations
Front Door / Application Gateway
... are just used to control global ingress into regions and have no attachment to this private link / DNS scenario.
API Management
... can be deployed globally and is linked into frontend virtual networks in each region with a dedicated IP address. API Management currently has no affiliation with private link and hence also no way to sensibly bring regional gateway name resolution into private DNS zones. As API gateways are only addressed internally from containers I feed IP adress / FQDN pairs into K8S hostaliases.
AKS
... setup is based on this post courtesy of Dennis Zielke.
challenges
multiple private DNS zones linked to a virtual network
Creating a single resource and private linking it to a virtual network is pretty straight forward and has docs dedicated to each of these resources e.g. for Cosmos DB. To achieve DNS name resolution - without standing up an own custom DNS server or fiddling around with hosts. files (which btw would not work e.g. for API Management) - private DNS zones can be used. But: each resource type needs a dedicated private DNS zone to be created and maintained.
In my scenario I created required private DNS zones with in the ARM template used to create the network configuration, immediately linking these zones to the frontend and backend virtual networks.
...
"privateZoneNames": {
"type": "array",
"defaultValue": [
"privatelink.database.windows.net",
"privatelink.vaultcore.azure.net",
"privatelink.blob.core.windows.net",
"privatelink.servicebus.windows.net"
]
},
...
// private DNS zones
{
"type": "Microsoft.Network/privateDnsZones",
"apiVersion": "2018-09-01",
"name": "[parameters('privateZoneNames')[copyIndex()]]",
"location": "global",
"copy": {
"name": "zonecopy",
"count": "[length(parameters('privateZoneNames'))]"
}
},
{
"type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameBackend'),'-link')]",
"location": "global",
"dependsOn": [
"[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]",
"[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameBackend'))]"
],
"properties": {
"registrationEnabled": false,
"virtualNetwork": {
"id": "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameBackend'))]"
}
},
"copy": {
"name": "zonebackendlinkcopy",
"count": "[length(parameters('privateZoneNames'))]"
}
},
{
"type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameFrontend'),'-link')]",
"location": "global",
"dependsOn": [
"[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]",
"[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameFrontend'))]"
],
"properties": {
"registrationEnabled": false,
"virtualNetwork": {
"id": "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetNameFrontend'))]"
}
},
"copy": {
"name": "zonefrontendlinkcopy",
"count": "[length(parameters('privateZoneNames'))]"
}
},
...
Only the DNS zones for the 4 regional resource types are created here for the overall network. Creation of DNS zones for the global resources is covered later.
When creating private endpoints for multiple resources and with that linking multiple private DNS zones to the same virtual network, auto registration cannot be enabled on more than one private DNS zone.
Auto registration for me seems to make sense for automatically registering VMs in other type of scenarios - not so much when handling Azure PaaS resources.
However when auto registration is disabled, you have to create DNS recordsets manually or to script the creation - not saying when it is enabled that resources - at least based on my observations - would necessarily register automatically.
Creating these private DNS recordsets can be achieved by looping through the FQDN entries registered for the created endpoint. These entries look like this e.g. for Azure SQL:
CustomDnsConfigs : [
{
"Fqdn": "my-sqlsvr.database.windows.net",
"IpAddresses": [
"10.2.6.42"
]
}
]
This entry needs to be created in a private DNS zone privatelink.database.windows.net but without the public domain name database.windows.net - hence this domain name suffix needs to be removed before:
$resourceGroup = "resourcegroupfordnszone"
$globalDnsSuffx = ".database.windows.net"
$dnsZoneName = "privatelink.database.windows.net"
...
# get custom DNS entries for endpoint just created
(Get-AzPrivateEndpoint -Name $privateEndpoint.Name).CustomDnsConfigs | % {
$recordSetName = $_.Fqdn -replace $globalDnsSuffx, ""
# remove existing record
if (Get-AzPrivateDnsRecordSet -ResourceGroupName $resourceGroup -ZoneName $dnsZoneName `
-Name $recordSetName -RecordType A -ErrorAction SilentlyContinue) {
Remove-AzPrivateDnsRecordSet -ResourceGroupName $resourceGroup -ZoneName $dnsZoneName `
-Name $recordSetName -RecordType A
}
# create new record
New-AzPrivateDnsRecordSet -ResourceGroupName $resourceGroup -ZoneName $dnsZoneName `
-Name $recordSetName `
-PrivateDnsRecord (New-AzPrivateDnsRecordConfig -IPv4Address $_.IpAddresses[0]) `
-RecordType A -Ttl 3600
}
```
### handling DNS entries for global resources with mutliple IP addresses
Creating the DNS entries for the regional resources was possible because the resource names (SQL, KeyVault, Storage, ...) had the region somewhere in the resource name anyway (e.g. fancy-sql-westus, fancy-sql-eastus) and with that providing unique name to IP address mappings.
For global resources there is one name (e.g. fancy-cosmos-global, fancy-cr-global) with an IP address / a set of IP addresses for each private link endpoint created in the regional virtual networks.
For CosmosDB it would result in a list like this:
```text
fancy-cosmos-global {10.1.6.17,10.2.6.15}
fancy-cosmos-global.eastus.data {10.1.6.16,10.2.6.14}
fancy-cosmos-global.westus.data {10.1.6.15,10.2.6.13}
```
No way for a consuming resources in `backend` or `frontend` network trying to resolve `fancy-cosmos-global.documents.azure.com` or in fact `fancy-cosmos-global.privatelink.documents.azure.com`.
> In my first attempts I failed with the assumption or expectation that somehow the source network would be considered here in the private DNS resolution.
To work around this, one would only need to create a dedicated private DNS zone for each region and feed the entries/IP addresses relevant for this region, but ... as private DNS zone name is also the resource name within a resource group, you can only have one private DNS zone (for a given resource type) in one resource group (until that stage I only had one resource group holding all the network resources including private DNS zones).
Hence I created additional resource groups for private links, endpoints and DNS zones specifically for a region but keep the common resource group for all the private DNS zones where this is not required.
For that I use a similar ARM template to create private DNS zones and link those to the virtual networks:
```json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vnetNameFrontend": {
"type": "string",
"defaultValue": "frontend",
"metadata": {
"description": "VNet frontend name"
}
},
"vnetNameBackend": {
"type": "string",
"defaultValue": "backend",
"metadata": {
"description": "VNet backend name"
}
},
"networkResourceGroup": {
"type": "string",
"defaultValue": "network",
"metadata": {
"description": "network resource group name"
}
},
"privateZoneNames": {
"type": "array",
"defaultValue": [
"privatelink.documents.azure.com",
"privatelink.azurecr.io"
]
}
},
"variables": {
},
"resources": [
// private DNS zones
{
"type": "Microsoft.Network/privateDnsZones",
"apiVersion": "2018-09-01",
"name": "[parameters('privateZoneNames')[copyIndex()]]",
"location": "global",
"copy": {
"name": "zonecopy",
"count": "[length(parameters('privateZoneNames'))]"
}
},
{
"type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameBackend'),'-link')]",
"location": "global",
"dependsOn": [
"[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]"
],
"properties": {
"registrationEnabled": false,
"virtualNetwork": {
"id": "[resourceId(parameters('networkResourceGroup'), 'Microsoft.Network/virtualNetworks', parameters('vnetNameBackend'))]"
}
},
"copy": {
"name": "zoneclusterlinkcopy",
"count": "[length(parameters('privateZoneNames'))]"
}
},
{
"type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
"apiVersion": "2018-09-01",
"name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', replace(parameters('privateZoneNames')[copyIndex()],'privatelink.',''),'-',parameters('vnetNameFrontend'),'-link')]",
"location": "global",
"dependsOn": [
"[resourceId('Microsoft.Network/privateDnsZones', parameters('privateZoneNames')[copyIndex()])]"
],
"properties": {
"registrationEnabled": false,
"virtualNetwork": {
"id": "[resourceId(parameters('networkResourceGroup'), 'Microsoft.Network/virtualNetworks', parameters('vnetNameFrontend'))]"
}
},
"copy": {
"name": "zonebackendlinkcopy",
"count": "[length(parameters('privateZoneNames'))]"
}
}
]
}
```
### putting it all together
I placed the create endpoint & maintain DNS entries in a common function (PowerShell module) as it was called for several resources.
```PowerShell
function Update-PrivateLink {
param (
[string]$locationCode,
[string]$resourceName,
[string]$resourceId,
[string]$groupId
)
# mapping from https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns
# list of groupIds / sub resources https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-overview#private-link-resource
# place DNS entries for global resources in location specific resource group private DNS zone to allow dedicated assignment to local VNETs
$dnsZoneResourceGroupName = "network-common"
$linkPrefix = $resourceName
switch ($groupId) {
# global resources have regional resource groups for private DNS zones -----------
"registry" {
$globalDnsSuffx = ".azurecr.io"
$dnsZoneName = "privatelink.azurecr.io"
$dnsZoneResourceGroupName = "network-" + $locationCode
$linkPrefix = $resourceName + "-" + $locationCode
}
"Sql" {
$globalDnsSuffx = ".documents.azure.com"
$dnsZoneName = "privatelink.documents.azure.com"
$dnsZoneResourceGroupName = "network-" + $locationCode
$linkPrefix = $resourceName + "-" + $locationCode
}
# regional resources are handled with the common network resource group -----------
"blob" {
$globalDnsSuffx = ".blob.core.windows.net"
$dnsZoneName = "privatelink.blob.core.windows.net"
}
"namespace" {
$globalDnsSuffx = ".servicebus.windows.net"
$dnsZoneName = "privatelink.servicebus.windows.net"
}
"sqlServer" {
$globalDnsSuffx = ".database.windows.net"
$dnsZoneName = "privatelink.database.windows.net"
}
"vault" {
$globalDnsSuffx = "(.vaultcore.azure.net|.vault.azure.net)"
$dnsZoneName = "privatelink.vaultcore.azure.net"
}
Default {
throw $("no DNS name mapping defined for groupId:" + $groupId)
}
}
subnetId = "{subnet-id-from-another-magic-function}"
$subnet = Get-AzVirtualNetworkSubnetConfig `
-ResourceId $subnetId `
-ErrorAction Stop
Write-Host "update private endpoint+DNS for" $resourceName $groupId "to" $subnet.Id
$privateLink = New-AzPrivateLinkServiceConnection -Name $($linkPrefix + "-link") `
-PrivateLinkServiceId $resourceId `
-GroupId $groupId
$privateEndpoint = New-AzPrivateEndpoint -ResourceGroupName $dnsZoneResourceGroupName `
-Name $($linkPrefix + "-endpoint") `
-Location $location.name `
-Subnet $subnet `
-PrivateLinkServiceConnection $privateLink `
-Force
(Get-AzPrivateEndpoint -Name $privateEndpoint.Name).CustomDnsConfigs | % {
$recordSetName = $_.Fqdn -replace $globalDnsSuffx, ""
if (Get-AzPrivateDnsRecordSet -ResourceGroupName $dnsZoneResourceGroupName -ZoneName $dnsZoneName `
-Name $recordSetName -RecordType A -ErrorAction SilentlyContinue) {
Remove-AzPrivateDnsRecordSet -ResourceGroupName $dnsZoneResourceGroupName -ZoneName $dnsZoneName `
-Name $recordSetName -RecordType A
}
New-AzPrivateDnsRecordSet -ResourceGroupName $dnsZoneResourceGroupName -ZoneName $dnsZoneName `
-Name $recordSetName `
-PrivateDnsRecord (New-AzPrivateDnsRecordConfig -IPv4Address $_.IpAddresses[0]) `
-RecordType A -Ttl 3600
}
}
```
This function is called after creation of each of the resource types.
The source of `resourceId` to be passed to the function varies by resource type. It can be `.Id` or `.ResourceId`.
`groupId` passed in is based on this [list](https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-overview#private-link-resource) and refers to the private link service `-GroupId`.
### Storage
```PowerShell
$storage = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageName -ErrorAction SilentlyContinue
if ($storage) {
Update-PrivateLink -locationCode $locationCode `
-resourceName $storageName -resourceId $storage.Id `
-groupId "blob"
}
```
### SQL
```PowerShell
$sqlServer = Get-AzSqlServer -ResourceGroupName $resourceGroupName -ServerName $sqlServerName -ErrorAction SilentlyContinue
if ($sqlServer) {
Update-PrivateLink -locationCode $locationCode `
-resourceName $sqlServerName -resourceId $sqlServer.ResourceId `
-groupId "sqlServer"
# can only be set after private endpoint is created
Set-AzSqlServer -ResourceGroupName $resourceGroupName -ServerName $sqlServerName -PublicNetworkAccess "Disabled"
}
```
> SQL server resource public network access needs to be enabled when creating the resource and no private endpoints yet defined. It can be switched to disabled after the endpoint creation.
### ServiceBus
```PowerShell
$sbNamespace = Get-AzServiceBusNamespace -Name $serviceBusNamespaceName -ResourceGroupName $resourceGroupName
if ($sbNamespace) {
Update-PrivateLink -locationCode $locationCode `
-resourceName $($sbNamespace.Name+"-sb") -resourceId $sbNamespace.Id `
-groupId "namespace"
}
```
### KeyVault
```PowerShell
$kv = Get-AzKeyVault -VaultName $keyVaultName -ResourceGroupName $resourceGroupName -ErrorAction SilentlyContinue
if ($kv) {
Update-PrivateLink -locationCode $locationCode `
-resourceName $keyVaultName -resourceId $kv.ResourceId `
-groupId "vault"
}
```
### CosmosDB
For CosmosDB I iterate through an array with all the relevant global locations (which I used before to setup multi master write regions) so that endpoint and DNS entries are created for and in each region.
```PowerShell
$cosmosDb = Get-AzCosmosDBAccount -ResourceGroupName $resourceGroupName -Name $accountName
if ($cosmosDb) {
foreach ($instanceLocationCode in $instanceLocationCodes) {
Update-PrivateLink -locationCode $instanceLocationCode `
-resourceName $accountName -resourceId $cosmosDb.Id `
-groupId "Sql"
}
}
```
### Container Registry
Same goes for the 2nd global resource.
```PowerShell
$acr = Get-AzContainerRegistry -ResourceGroupName $resourceGroupName -Name $acrName
if ($acr) {
foreach ($instanceLocationCode in $instanceLocationCodes) {
Update-PrivateLink -locationCode $instanceLocationCode `
-resourceName $acrName -resourceId $acr.Id `
-groupId "registry"
}
}
```
> limitation: ACR build does not work out-of-the-box when placed behind a firewall or as in this case in a closed down network - except when using ACR private build agents; as I did not want to have 2 types of build agents to maintain I choose regular Azure DevOps build agents (several Docker containers running on the jump VMs) to build Docker images
## conclusion
Service endpoints are easier to setup and handle. Private link requires more planning and a higher sophistication in infrastructure automation but with that allows really fine grain control on network access paths.
Currently I have a _mix_ of imperative and declarative infrastructure code which I basically do not prefer. This is tributed to the flexible way we want to spin up regions and instances of the infrastructure. Maybe in some other iteration we refactor and reintegrate in the one way or the other.
Let me know whether this makes sense to you and whether it helped you out in your work.
## credits
Thanks to [my good buddy Matthias](https://dev.to/matttrakker) for reviewing.
----
## March 2023 addendum - Blob Triggers in container hosted Azure Functions
Having this environment in place for a while we experienced a strange issue:**Azure Functions** hosted inside containers in above environment, which access storage account over private link, did **not get triggered by blob triggers**. Another esteemed team mate of mine - Fabian - figured out that in such a case a private link to only the blob endpoint is not sufficient but also a private link+DNS to the blob queue endpoint is required [see here](https://github.com/fabistb/azure-function-blob-trigger-private-endpoint).
Top comments (11)
added March 2023 addendum - Blob Triggers in container hosted Azure Functions
Hi Kai,
Great post! Thanks for sharing!
Did you create multiple DNS Private Zones for the same resource for each regional VNET?
As per my understanding Microsoft recommends creating one single Private DNS Zone as suggested here. You can link that Private DNS Zone with multiple VNETs as described here
This way you can manage all your Private Zones in one resource group. Unless you faced any limitations with that approach?
Look forward to your thoughts :)
Hello Paco,
right, for most of the resources - which have a unique name within a region and hence a unique IP address - I created one private DNS zone and linked that to multiple VNETs. For the 2 in our environment "global" resources like CosmosDb and Container Registry, which are linked into all regional VNETs and with that have different IP addresses for the same "global" name, this did not work. I did add same name with multiple IP addresses into the one private DNS zone for e.g. CosmosDB, but with that resolution and accessing the service was not possible - our VNETs are not cross-connected and there was no way controlling that always the correct "local/regional" IP address was resolved. Hence the approach that for these global resources I have regional private DNS zones with the proper local/regional IP address.
Does this make sense? Maybe I overlooked something here but I was really checking docs and also haunting MS support :-)
-Kai
It does make sense. Thanks for clarifying Kai!
woooww, you blog blasted my brain! ;) thanks kai for sharing this real life insights and learnings in enterprise-scale challenges with azure. one questions came to my mind on reading: Does access/management to the secured resources via portal.azure.com still work in this scenario with service/private links ootb across the network boundaries? Or do you have even no more need to use portal.azure.com (eg. use the portal-storage explorer to check blobs in storage..) for administration in this scenario?
Thanks JJ!
Indeed implementing access management - if you refer to IAM - is one of the next things I will add to this setup - so right now I cannot tell. Right now devs do not have access to this environment anyway and admins would use jump VMs - for Portal, PowerShell and CLI.
Also we use a script which an admin can use to link his/her own VM to the enviroment. Storage Explorer, Data Explorer, etc. I do not want to be available in publicy accessible Azure Portal.
added another post how I added IAM to the solution : dev.to/kaiwalter/getting-started-w...
for me these are totally independent - with IAM you restrict people being able to create or modify Azure resources and with private link you restrict access to the resources data (SQL, CosmosDB,Storage,ACR) or functionality (AKS,ACR,ServiceBus)
to be able to read the contents you would need to have a peering from where you come (Tunel or Express Route) and let your (I assume on-prem environment) also know about the Private DNS Zones in Azure. Here is a nice doc from msft how this can be handled: docs.microsoft.com/en-us/azure/pri...
added limitation of ACR build in private link setup
How do you handle private-link names for keyvault if having a single privatelink.vaultcore.azure.net dns-zone? They would be clashing.
What do you mean by "clashing"?
If I would have to produce the same DNS record but with different IP addresses (like for Cosmos DB multi-regional) I would have private DNS resource spreaded over multiple (regional) resource groups and only assign relevant private DNS resource to the appropriate VNET.