Prerequisite: The Budget Alarm

Before hitting the hay and laying your head down for a good night’s sleep, you set an alarm. Before putting the cake mixture into the oven you set a timer. It is easy to notice the marathon runners all lined up in a similar pre-gun shot position awaiting the bang that marks the start of the race, but if you pay close attention to them you will notice that they are all touching their wrists, ready to set the stopwatch that tracks their race time.

Tracking metrics and setting alarms to tell us the time it took us to run 42.195 km or to wake us up after a night’s slumber is essential to us. It is no different when managing our cloud environments. Before we think about optimizing, creating savings plans, or cutting costs we have to make sure we don’t go over our budget in the first place. If we do go over, we have to be notified and be ready to take action quickly. And if that action is automatic then even better.

If you have never had the horrible experience of getting a shockingly high bill from your cloud provider I’m happy for you. But in case you are wondering what it feels like, just imagine that time you were charged for the damn subscription you thought you had canceled but magnified by an order of magnitude.

Imagine literally taking a 30-minute nap and waking up to a bill of just under 72.000$, granted that each situation is different and not everything thing is the users fault, cloud providers can be in the wrong too.

One thing is for sure though, it is in our best interest to be the first ones to know when budgets go over, and ideally, automation is in place to make sure to avoid financial stress by taking timely automated action.

In this post, we will explore how we can set up budget alerts the traditional way and then explore how we can extend the sophistication of our budget alert response by leveraging the Tailwarden budget alert and integrating it with a useful incident response tool like PagerDuty.

Since doing this is easier said than done, let’s dig a little deeper.

Account set up

When you set up either a personal or professional cloud account in any of the major cloud providers, probably one of the first things you are going to want to do is set up a billing alarm starting at a certain amount (for example 1000$) and then other subsequent alarms at base unit increments (2000$, 3000$, …) You will be able to find the budget alarm configuration section in the Billing preferences part of our cloud providers console. In this blog post, we will be using AWS but the concepts covered are applicable to any cloud provider.

Where to set the budget notification in AWS?

Head over to the Billing Console (check in with your organizations AWS account admin in case you can’t access the billing console). We then go to Budgets and start creating our first Budget by choosing the budget type. In this case, to keep things easy we will choose the provided templated version which is more than enough.

We then have three options, we will choose the Monthly cost budget which notifies us if we exceed our forecasted dollar amount for the month.

Further down on the configuration page we will add the name for our budget alarm, enter our budget amount in dollars for the month, and then choose the email recipient/s who will get notified if the budget threshold is breached. As we can see in the call-out below we can be notified on different occasions:

• When your actual spend reaches 85%
• When your actual spend reaches 100%
• If your forecasted spend reaches 100%

After filling in all the relevant information we can then hit “create budget”. Now just sit back, relax and enjoy the Budget health dashboard’s pleasant under-budget metric and the alerts thresholds that are in the green.

Even though this is definitely a great first step. Imagine if the person or people that are notified via email are on holiday, or out sick or if they don’t check their email that much. Crazier things have happened. What better way could behave of making sure we know exactly when we are going over budget?

Choose the right channel for you

You don’t want to want to depend on emails to know if you are within your budget or not. As an organization and team, it’s worth determining which is the most useful notification channel to use. Even though there are other alternatives let’s kill two birds with one stone by using the billing notification function on Tailwarden and getting notified if our accounts go over budget (regardless of what cloud provider they are in) via Slack and PagerDuty.

Slack integration

The first thing to do is to head over to your profile setting and click the integrations tab. Below you click the connect button and a slack permissions page will appear.

In this case, we are allowing the Tailwarden slack app permission to access the Tailwarden Slack workspace. Press allow and we should moment after see the successful integration message.

Hurray! we are ready to start configuring our notifications.

In the Cloud accounts section, we can choose the account to which we would like to attach our billing alarm by clicking on the slack icon in the bottom right-hand corner.

A pop-up will appear in which you can either simply choose a Slack channel to which you would like to send your most up-to-date cost prediction estimate at a time of day of your choosing. Or you can additionally toggle the budget alarm feature to unfold the pop-out.

You will then be able to add the dollar amount threshold at which you would like to be notified if your account goes over.

Underneath the budget section, you have an area where you can add a URL that will receive the notification POST request.

This POST request URL endpoint can then go on to trigger any type of automation of your choosing. You can trigger a function to take an action on your behalf, can trigger an automation through a tool like Zapier or trigger an incident on PagerDuty. Let’s look into this use case below.

PagerDuty integration

PagerDuty is a stellar service for organizing and coordinating incident response efforts. If your organization is using the service you probably have a team or series of teams that share an on-call rotation. These on-call rotations though might, unfortunately, wake you up in the early hours of the morning. They prove to be very effective at alerting and enabling quick responses to incidents as they arise. That’s why we have made it easy to send a Tailwarden billing alert POST request whenever an alert is created so you can integrate it into your PagerDuty alerts. So you never miss an important billing notification.

💡 Note that you are going to require an active PagerDuty account to follow the steps below.

Firstly we are going to create a new service in the Service Directory section in the PagerDuty console.

Add a name and a description, in our case we are giving the service a name that represents which AWS account it (Production) is and its function (Billing alerts).

On the second slide, we can choose the Escalation policy configurations.

An escalation policy is the sequence of steps a notification will go through if unacknowledged.

Even though each team is different an example of an escalation policy might be:

Notify the on-call individual, if 30 minutes pass and the notification is unacknowledged send the notification to the team lead, and then if still unanswered to the team director and so on.

Here we will choose intelligent alert grouping to avoid multiple related alerts.

And stage number 4 is where we choose our integration type. In this case, we will choose the PagerDuty Events API V2. This is the API that we are going to send the Tailwarden billing alarm.

By creating the service with this integration type we don’t need to do anything on the PagerDuty side, now we can take the Integration URL (Alert Events) that has been provided to us on the lower right-hand side of the screen.

And we paste it into the POST URL section of the notification pop-out in Tailwarden. By hitting save the integration will be complete and we will be covered.

We can also check on our PagerDuty dashboard to make sure no notifications have come in.

Since you don’t want to be looking at the PagerDuty dashboard 24/7. The best way to keep track of the pager alerts is for you and your team to download the PagerDuty app to get the notifications delivered straight to your phone.

Conclusion

We are all working and building on the cloud because there are incredible advantages and services that we have access to that engineers from generations past couldn’t have even imagined having in their wildest dreams. It’s never been easier to test out the newest technology, break servers and grow businesses in record time. Since most of us have high aspirations and big plans for what we want to do in the cloud. Let’s start off on the right foot and set budget alarms so to avoid getting too excited and burning our financial reserves too soon. There will always be time to optimize and iterate. But as a first step make sure that you don’t go one dollar over budget without knowing about it.

Regardless if you are a Developer, DevOps, or Cloud engineer. Dealing with the cloud can be tough at times, especially on your own. If you are using Tailwarden or Komiser and want to share your thoughts doubts and insights with other cloud practitioners feel free to join our Tailwarden Discord server. Where you will find tips, community calls, and much more.