TL;DR
When operating Azure API Management in an internal virtual network, already integrated with Azure Application Gateway, an upcoming feature AllowApplicationGatewayPrivateLink
allows you to connect this configuration to another virtual network using Private Link and Private Endpoint.
Motivation
In a post I made beginning of February 2022 I linked a virtual network with limited address space - like in a corporate / ExpressRoute / SD-WAN connected scenario - to a Container Apps environment:
That setup works as long as ingress only goes from resources in Hub network to Spoke network. But if Container Apps needs to call (as in my target scenario) Azure API Management - calling from Spoke network into Hub network - that solution is not sufficient.
Looking for options I had this post which Marcel.L pointed me to back then, where he forwards calls to API Management in another virtual network using a Private Endpoint/Link and Virtual Machine Scale Set combination.
However, as pointed out in my earlier post, my endgame should be to replace Azure Service Fabric IaaS container hosting with a higher level abstraction, PaaS like Container Apps, hence I did not necessarily want to add just another IaaS in the process - even one with a lower complexity.
Searching for alternatives I checked on private linking capabilities of Azure API Management itself. However this cannot be used and mixed when it is already operated in external or internal virtual network mode - see preview limitations. Hence no option for me.
"As with container compute I need my API Management with 100% ingress and egress within a virtual network"
Researching on private IP address options on Azure Application Gateway I stumbled over a private link feature in Azure CLI which - lacking tangible documentation - I exploited here and converted to Bicep
Solution Elements
To achieve this configuration
following solution elements additionally to my earlier post are required:
- a Private Link Configuration on the Application Gateway; still to subnet within Hub virtual network
- a Private Endpoint in Spoke virtual network
- a Private DNS Zone linked to Spoke virtual network so that Container Apps resolve to Private Endpoint IP address
complete configuration for all snippets shown in this post can be found in this repo / tag; I'll state the
filenames
so that snippets can be found more easily
Prerequisites
- Azure CLI
- Bicep
- Linux shell / bash / ...
Stage 1 - get preview feature working (as of Feb'2022)
When first trying to deploy Application Gateway with specifying privateLinkConfiguration
, I was rewarded with a nice error code SubscriptionNotRegisteredForFeature
and a message like Subscription /subscriptions/... is not registered for feature PrivateLinkConfigurations required to carry out the requested operation
. Finding no suitable documentation I tried to find the feature switch myself with
az feature list --namespace Microsoft.Network -o table | grep Priv
I then activated what seemed to make sense for my case:
az feature register --name AllowApplicationGatewayPrivateLink --namespace Microsoft.Network
az feature register --name AllowAppGwPublicAndPrivateIpOnSamePort --namespace Microsoft.Network
az provider register -n Microsoft.Network
DISCLAIMER: The above steps I figured out being unaware that this is (as of Feb'22) a private preview.
The official process: For inquiries about private preview features, please send your subscription id, region, and customer name to appgwpreview@microsoft.com for enrollment.
This will entitle you officially to the preview and you will get proper documentation and sample code.
Stage 2 - configuration Application Gateway
To add private link, an entry in privateLinkConfigurations
section is required. It needs to be put in the same virtual network but a different subnet as the gateway itself (once Application Gateway is deployed to a subnet, it only allows for other Application Gateways to share this subnet, but no other resources).
That privateLinkConfiguration
is then to be referenced on the frontendIPConfiguration
. In my sample I share it with the public IP, but I guess, this also can be separated to allow various forwarding rules depending from where ingress is coming from.
appgw-priv.bicep
...
var fipName = 'appgw-frontend'
...
frontendIPConfigurations: [
{
name: fipName
properties: {
publicIPAddress: {
id: pip.id
}
privateIPAllocationMethod:'Dynamic'
privateLinkConfiguration:{
id: resourceId('Microsoft.Network/applicationGateways/privateLinkConfigurations', appGwName, 'private')
}
}
}
]
privateLinkConfigurations: [
{
name: 'private'
properties: {
ipConfigurations: [
{
name: 'private-ip'
properties: {
primary: true
privateIPAllocationMethod:'Dynamic'
subnet: {
id: subnetJumpHubId
}
}
}
]
}
}
]
...
for simplification in my sample I am using HTTP on port 8080; in production this will be replaced by a proper HTTPS, FQDNs and certificates
Based on private link configuration in Hub network now a private endpoint can be added in Spoke network:
appgw-priv.bicep
resource pep 'Microsoft.Network/privateEndpoints@2021-05-01' = {
name: 'pep-priv-gateway'
location: location
properties: {
subnet: {
id: subnetJumpSpokeId
}
privateLinkServiceConnections: [
{
properties: {
privateLinkServiceId: appgw.id
groupIds: [
fipName
]
}
name: 'pep-priv-gateway'
}
]
}
}
fipName
Important to highlight: the name of the groupId
added to the Private Endpoint has to match the name of the frontendIPConfiguration
of the Application Gateway.
Stage 3 - private DNS zone
Spoke network needs a private DNS zone so that traffic towards the Private Endpoint is resolved correctly:
appgw-priv-dns.bicep
param vnetSpokeId string
param apiName string
param pepIp string
var privateDNSZoneName = 'internal-api.net'
resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {
name: privateDNSZoneName
location: 'global'
}
resource privateDnsZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
parent: privateDnsZone
name: '${privateDnsZone.name}-link'
location: 'global'
properties: {
registrationEnabled: false
virtualNetwork: {
id: vnetSpokeId
}
}
}
resource privateDnsZoneEntry 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
name: apiName
parent: privateDnsZone
properties: {
aRecords: [
{
ipv4Address: pepIp
}
]
ttl: 3600
}
}
I use my own DNS zone internal-api.net
here as - unlike other Azure private link capable resources - Application Gateway does (and I guess will) not have its own private endpoint DNS zones.
Stage 4 - deploy API Management, Application Gateway and DNS zone
I had to split deployment of API Management & Application Gateway, return and extract private IP address information from private endpoint and then continue with private DNS deployment. I was not able (or just too lazy) to wire this up within one Bicep file.
deploy-stage-2.sh
...
az deployment group create --resource-group $RESOURCE_GROUP \
--template-file apim.bicep \
--parameters apimName=$APIMNAME \
appInsightsName=$APPINSIGHTNAME \
logAnalyticsWorkspaceName=$LOGANALYTICSNAME \
fapp1Fqdn=$fapp1Fqdn \
fapp2Fqdn=$fapp2Fqdn
VNET_SPOKE_ID=`az network vnet list --resource-group ${RESOURCE_GROUP} --query "[?contains(name,'spoke')].id" -o tsv`
PEP_NIC_ID=`az network private-endpoint list -g $RESOURCE_GROUP --query "[?name=='pep-priv-gateway'].networkInterfaces[0].id" -o tsv`
PEP_IP=`az network nic show --ids $PEP_NIC_ID --query ipConfigurations[0].privateIpAddress -o tsv`
az deployment group create --resource-group $RESOURCE_GROUP \
--template-file appgw-priv-dns.bicep \
--parameters "{\"pepIp\": {\"value\": \"$PEP_IP\"},\"vnetSpokeId\": {\"value\": \"$VNET_SPOKE_ID\"},\"apiName\": {\"value\": \"$APIMNAME\"}}"
along with API Management instance I deploy an API to test calls to Function Apps hosted in Container Apps environment;
appgw-priv.bicep
shown above is a module ofapim.bicep
, hence deployed in the first block
Stage 5 - testing
To check, that I have not introduced regressions I first test access to Function Apps over public IP of Application Gateway:
test-api.sh
...
APIMID=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query id -o tsv`
APIMURL=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query gatewayUrl -o tsv`
GWURL=`az network public-ip show -n ${APIMNAME}-priv-gateway-pip -g $RESOURCE_GROUP --query dnsSettings.fqdn -o tsv`
GWPORT=`az network application-gateway show -n ${APIMNAME}-gateway -g $RESOURCE_GROUP --query frontendPorts[0].port -o tsv`
SUBKEY=`az rest --method post --uri ${APIMID}/subscriptions/test-subscription/listSecrets?api-version=2021-08-01 --query primaryKey -o tsv`
curl -s ${GWURL}:${GWPORT}/test/fapp1?subscription-key=$SUBKEY
printf '\n'
curl -s ${GWURL}:${GWPORT}/test/fapp2?subscription-key=$SUBKEY
printf '\n'
To make more thorough tests of the Functions Apps in combination with calls to API Management I have to hop on a jump VM in Spoke network:
test-fapps.sh
...
IP=$(az vm list-ip-addresses -g $RESOURCE_GROUP --query "[?contains(virtualMachine.name, 'hub')].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv)
declare -a apps=("fapp1" "fapp2")
for app in "${apps[@]}"
do
echo "$app"
fqdn=$(az containerapp show -n $app -g $RESOURCE_GROUP --query configuration.ingress.fqdn -o tsv --only-show-errors)
ssh ca@$IP curl -s https://$fqdn/api/health
echo " <<-- check APIM internal status"
ssh ca@$IP curl -s https://$fqdn/api/apim-status
echo " <<-- check $app APIM health"
ssh ca@$IP curl -s https://$fqdn/api/apim-internal-status
echo " <<-- check $app APIM internal status"
done
where
-
curl -s https://$fqdn/api/health
checks Function App only with a call to a HTTP trigger -
curl -s https://$fqdn/api/apim-status
andcurl -s https://$fqdn/api/apim-internal-status
call HTTP triggers which themselves forward calls to API Management instance (using theinternal-api.net
domain)
[FunctionName("Apim-Status")]
public static async Task<IActionResult> ApimStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-status")] HttpRequest req)
=> new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/status-0123456789abcdef"));
[FunctionName("Apim-Internal-Status")]
public static async Task<IActionResult> ApimInternalStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-internal-status")] HttpRequest req)
=> new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/internal-status-0123456789abcdef"));
Public IP vs SKU vs Private Link Configuration
For my scenario a public IP would not be required. Removing it is not supported for Standard_v2
SKU:
Application Gateway /subscriptions/.../ca-kw-priv-gateway does not support Application Gateway without Public IP for the selected SKU tier Standard_v2. Supported SKU tiers are Standard,WAF.
On the other hand PrivateLinkConfigurations
is only supported for Standard_v2
:
Application Gateway /subscriptions/.../ca-kw-priv-gateway does not support PrivateLinkConfigurations for the selected SKU tier Standard. Supported SKU tiers are Standard_v2,WAF_v2.
So for the moment I'd keep the public IP and will check again at a later stage of the preview.
Conclusion
With this solution a major roadblock for me is out of the way and I do not need to operate my own compute resources for network traffic forwarding. It still requires a some polishing and hardening before I can use it even close to our production resources.
A positive effect for me is that with reconfiguration of the existing Application Gateway and a very low footprint of IP addresses consumed in my corporate virtual network, I can even operate the existing environment with Service Fabric in coexistence with the new Container Apps environment, which reduces migration risks tremendously.
Top comments (9)
@pwd9000 this might be interesting for you; maybe and if you have an Application Gateway at hand an evolution for what you did with private link/endpoint, load balancer and VMSS; I hope mentioning you here in this post is fine for you - otherwise let me know and I rephrase/remove
@kaiwalter Your post is awesome! I will try this out soon! Thanks for sharing it with the community and also thanks for the mention!
Thank you for sharing these insights. With the help of your post it was quite easy to setup a private link configuration with terraform while utilizing a frontend ip config containing the gateway's private IP.
You are very welcome. I am glad that my post is of some use!
added reference to preview limitations See preview limitations
added Public IP vs SKU vs Private Link Configuration
added private preview disclaimer
Taking into consideration this scenario, can I use the APIM for external and internal requests with the private endpoint configuration in APIM?
So if you would setup APIM only with private endpoint and then connect AppGw to it, from my understanding you should be able to connect to APIM externally and internally over AppGw. However I did not yet validate this kind of configuration.