Test Azure Functions scaling on Container Apps with Azure Load Testing

motivation

I am currently evaluating, whether and how I can transfer this Azure Functions containers on Service Fabric workload to Azure Container Apps.

Primary reason to use Service Fabric back then was to be able to host Azure Functions within our own virtual networks - inbound and outbound - a feature that is now also available with regular Azure Functions (Premium) / App Service hosting. However this type of hosting has other limitations - e.g. the amount of outbound IP addresses required - which does not fit into our scenario.

Container Apps seem to become a valid hosting option for our scenario and our requirements once BYO/bring-your-own virtual network feature is available.

"you could do Kubernetes" ... yeah, but there would be no operational savings & gain for us, compared to Service Fabric; I want the real deal: I just don't care about the cluster/compute resource

While waiting I want to explore some of the other capabilities which might come in handy later - starting with auto scaling.

setup

With the steps described in the accompanying repository these resources are created - using Pulumi Native this set of resources is created:

the step - change ca-pulumi-dotnet/Program.cs to FunctionAppStack.cs - before deploying is important in case to test any of the other Container Apps workload like Dapr; to get the exact working version it makes sense to checkout the tagged version

• a Container Registry to store the images of the 2 Function Apps
• a Container App Environment to host 2 Container Apps : fapp1 = HTTP ingress onto Service Bus, fapp2 = Service Bus queue ingress
• a Service Bus Namespace to process queued ingress
• a Load Testing resource to execute the actual load test
• an Application Insights instance to evaluate some of the scaling behavior

What is created with Pulumi here can also be achieved with Azure CLI, ARM or Bicep - or a combination. I just wanted the whole stack including Docker image creation in one concise script.

special ingredients

Being new to Pulumi IaC these things needed some more exploring and work on my end:

• hooking the private Container Registry credentials into the Container App deployment
• hooking Application Insights in the Function Apps
• creating the Load Test service including an assignment of Load Test Owner role to whomever is executing the deployment

considering Functions host Id

A few days after original creation of this post I remembered that when hosting Function Apps in containers "on your own" / outside App Services and you want to have proper singleton and scaling behavior, you need to

• actively set hostId to the same value on all container instances for the same Function App with environment variable AzureFunctionsWebHost__hostId
• connect to a storage account where Functions runtime keeps all synchronization information with environment variable AzureWebJobsStorage

    new EnvironmentVarArgs
    {
        Name = "AzureWebJobsStorage",
        SecretRef = "storageconnection",
    },
    new EnvironmentVarArgs
    {
        Name = "AzureFunctionsWebHost__hostId",
        Value = Guid.NewGuid().ToString().Replace("-", ""),
    }

I conducted the test again after this addition, however it had no noticeable influence on scaling and test results.

load test creation and execution

I created a simple JMeter test plan and added it to the repository as ca-pulumi-dotnet/loadtest.jmx which then allows inserting loadtesturlfapp1 Pulumi output value as parameter ingress_url when creating the test:

Once created the test plan can be executed. It is configured to submit 10000 requests and then stop:

an error Non HTTP response code: java.net.UnknownHostException indicates, that the URL passed into the ingress_url parameter is not correct

test results

A simple Application Insights query cannot show the effective scaling of the Container App Environment but it reveals, that both Function Apps (both in pink color) dwell ~ 15-30 seconds with their minimum single instances and then kick into full gear and start scaling to handle the increased load:

requests
| where timestamp between ( todatetime("2022-01-30T12:45:00") .. todatetime("2022-01-30T12:55:00"))
| summarize count() by bin(timestamp,15s), cloud_RoleInstance
| render columnchart 

From 12:49:00 PM one can see, that HTTP Ingress is finished and the fapp2 instances process the queued ingress in a balanced manner.

Additionally I crafted this Azure Monitor query to see whether the same amount of instances was started and stopped:

ContainerAppConsoleLogs_CL 
| where ContainerImage_s contains "fapp"
| where TimeGenerated between( todatetime("2022-01-30T12:45:00") .. todatetime("2022-01-30T13:15:00") )
| where Log_s contains "Application is shutting down..." or Log_s startswith "Application started."
| parse Log_s with * "Application" eventName "." *
| summarize count() by eventName, bin(TimeGenerated, 5m)
| render columnchart 

Application started is logged twice, I only take the entry where the message starts in the beginning without leading blanks.

conclusion

To me the simplicity of

• hosting
• scaling
• load testing

in this sample is overwhelming. I am sure there will be some quirks waiting for our team along the way. I will keep you posted here.

similar posts

• Moving Azure Functions from AKS to Container Apps