DEV Community

loading...
Cover image for The case of the Mysterious Azure Storage spikes

The case of the Mysterious Azure Storage spikes

peterlindholm profile image Peter Lindholm ・2 min read

Recently I noticed a significant increase in Azure costs on a client project. For this particular client cloud storage is used for virtual machine disks, a few assets and hosting of a set of application. Cloud storage costs is something I've almost thought of as free. Naively.

Over a short period of time storage costs increased more than 300%. Weird. I checked and compared usage over a couple of billing periods. No spikes in user traffic that could explain this.

Fortunately Azure Cost Analysis lets you dive pretty deep on all individually incurred costs. The majority of costs was incurred by 'lrs write operations'. Digging into the 'Total Requests' metric of the storage account I was seeing approx. 1.4M requests per hour.

Drilling further into ingress/egress metrics revealed almost no data moving across the network πŸ€”

Next step was grouping the different kinds of requests making up those crazy numbers. Only a single operation stood out with millions of operations occurring during a regular day: ChangeNotify.

What changed?

So, troubleshooting 101...what has changed? Going through the latest commit log revealed upgrades to a few Azure Functions projects (sharing storage). The projects was updated to Azure Function v3 runtime and moving from .NET Core 2.1 to .NET Core 3.1. But no code changes.

This gave a few Google'able hints to go for and surely I was not the only one seeing this.

Apparently the culprit seemed to be how IConfiguration was built and used inside individual functions. In the oldest of the functions, before DI in Azure Functions was a thing, I had this approach to accessing the configuration data:

var config = new ConfigurationBuilder()
                .SetBasePath(context.FunctionAppDirectory)
                .AddJsonFile("local.settings.json", 
                    optional: true, 
                    reloadOnChange: true)
                .AddEnvironmentVariables()
                .Build();
Enter fullscreen mode Exit fullscreen mode

In particular this line below was causing the issue:

.AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)
Enter fullscreen mode Exit fullscreen mode

'local.settings.json' is as the name implies meant for local usage/testing scenarios. It is never committed to source control and thus never published to Azure. But even though it is marked as optional, something in the Azure Functions runtime goes crazy because of this "missing" file.

I cleaned up the old functions and allowed for them to use constructor injection of whatever configuration was needed in the functions. This was I completely removed the above code block initialising the config.

Did a few quick tests and deployed to production. Either way the functions would be better off that before. But immediately I saw the transactions count dropping.

Screenshot-2021-02-20-at-15.18.10

Guess when my cleanup was deployed? πŸ˜€

Lessons Learned

I have no idea why the increased storage operations was triggered after upgrading to a newer runtime. But my best bet is that the Azure Functions team have had a regression πŸ€·β€β™‚οΈ

This issue was a welcome opportunity for me to learn more about the Cost Management part of an Azure Subscription. Going forward monitoring of cost analysis grouped by resources will be Yet Anotherℒ️ activity to include in my day-to-day operations.

Discussion (3)

pic
Editor guide
Collapse
jdedwards3 profile image
James Edwards

I am also seeing this with a nodejs function app and have contacted azure support. There appears to be many tickets open in regard to this.

I am not accessing the configuration builder since it is nodejs and not .net, any thoughts?

Collapse
peterlindholm profile image
Peter Lindholm Author

Thank you for your comment. Sorry, I'm late to reply I somehow missed the notification.

Is this new behaviour for you? I was wondering if your issue is also related to a package upgrade?

Also, are you able to identify which operation(s) that are causing this. Like in my case where it was ChangeNotify.

  1. Go to your storage account in Azure Portal
  2. Find "Insights" in the sidebar
  3. See the graph for "Transactions by storage type"

Does anything seem out of proportion? Are you seeing a lot of failures?

If you have a lot of failures try switching to the "Failures" tab to identify which type of transactions are failing.

Let me know if I can help you track this down.

Collapse
jdedwards3 profile image
James Edwards

Thanks Peter!

storage

Here's the last 14 days from my storage account. The peaks for the file transactions appear to be related to function app deploys.

I'm using github actions to deploy the functions

So far I have spoken to two representatives from azure support (one functions and one storage) and they have provided essentially zero assistance in resolving this issue. I have provided them with this thread and others mentioning the same thing.

I still don't have any clue what the cause is, but I am running other storage accounts with way more data transfer and data stored for fractions of the cost that the one associated to azure functions is incurring.