Because spinning up VMs in the cloud is so easy, it’s equally easy for your monthly bill to scale up as well. What may have started as a few hundred dollar-a-month charge for a few VMs can quickly ratchet up into tens of thousands of dollars.
In larger businesses, I’ve seen this get so bad that management freaks out and moves everything back on-premises, even when the growth was legitimate and not the result of waste or mismanagement.
There is also a tendency for devs and ops folks to treat the financial aspects of their infrastructure and applications as “not my job”, which is an idea that makes my brain hurt from all the angry it creates.
(ノಠ益ಠ)ノ彡┻━┻
(ノಠ益ಠ)ノ彡┻━┻
(ノಠ益ಠ)ノ彡┻━┻
Three tables flipped, and I still can’t write about the “not my job” angle. It will have to wait for another blog post. For now, let’s just assume that you, dear reader, are a minimally-functional adult who understands the concept of owning the things you build.
Right-sizing
As you’re building your infrastructure, it’s always a good idea to bake in instrumentation to understand utilization. Most people view this as a must-have for troubleshooting and monitoring, but there’s also a cost angle.
Assuming you already have infrastructure, the first step in lowering your AWS EC2 cost is identifying what you actually need.
You can either roll your own solution for right-sizing your infrastructure using the performance and inventory data you should already be collecting with AWS (or your cloud provider of choice) or you could use a service like CloudHealth Tech or CloudCheckr that are purpose-built for cost optimization.
The SaaS solutions do a lot of stuff related to cost optimization, but one of the main things they provide is utilization reporting and recommendations.
Maybe you thought you needed 50 m4.2xlarges for your app. Maybe what you really need are 25 t2.mediums (a $14k a month difference). I have seen gaps that large revealed when running utilization reports for the first time.
You may be thinking “someone who understands their app would never let that happen”, and that is true most of the time. The problem is:
- Very few people actually understand the apps they run.
- Very few people consider the cost of running those apps as being something they own.
- Related to #1, many people don’t have the luxury of running apps that they have built themselves. They are handed someone else’s shitty app and told to keep the lights on.
Getting a good grip on actual utilization tends to also help with conversations like “maybe we should spend the time to make our app stateless so we can auto-scale servers” or “maybe it’s time to convert this app to PaaS.”
_ Note : If you do end up switching a bunch of servers to t2-tier, make sure you monitor CPU credits. This tier has a CPU-usage quota and if you go over that, your server is throttled until the credits rebuild. I’ve seen people flip over to t2s thinking “it’s got the same specs as my m4/5” and then wonder why their high-CPU-usage app is crawling a few hours later._
Buy a baseline
An argument that is normally used to justify hybrid cloud is “we have a baseline on-prem, so we’ll just use the public cloud for burst elasticity”, which is reasonable. However, you can do the same thing in-cloud.
Once you get a grip on your baseline needs, reserved instances (RIs) become an option. It is entirely realistic to cut costs in half with RIs.
RIs take away some flexibility in return for lower instance prices and are similar to financial instruments you see elsewhere in the world, like cell phone plans.
“You can pay month-to-month with no contract, or you can save X% by signing up for a minimum term.”
Unlike a cell phone plan, there is an RI market where you can resell your RIs if you need to get out of them, and there are also lots of options around converting them different instance sizes. Converting RIs to different instance families is also possible, but a bit trickier.
_ Note: There are limits around selling your RIs and the marketplace doesn’t have a high volume in many regions, so selling opportunities may be limited. So don’t purchase RIs with 100% certainty that you’ll be able to resell them._
RI discounts are primarily tied to term (the length of the contract) and percentage paid up front (none, partial, or full). Three-year full-upfront RIs are going to be the cheapest option.
A trick I’ve learned in the past year is that many resellers like SHI and CDW offer financing for AWS RIs. So instead of paying for 1 or 3 years fully upfront, you can amortize that “upfront” cost over a term with your reseller. Obviously, you’re paying interest as part of the financing, but if the decision was between a 10%-off no-upfront RI purchase and a 60%-off full-upfront purchase, paying a few points in interest is a no-brainier.
Fun fact: RIs can also be purchased for RDS instances and this is almost always an easy win because of how “permanent” your database servers likely are in comparison to the rest of your infrastructure.
Spot instances are your friend
Assuming you’ve purchased a baseline of RIs, any additional instances above that will be either at the “on-demand’ or “spot instance” price.
With spot instances, you are effectively bidding on AWS’ excess capacity, so pricing is highly dynamic based on demand. Because ‘spots’ are excess, they will be pulled back into AWS’ inventory when that capacity is needed by other users paying the on-demand or RI rate.
Spot instances are awesome when they’re feasible for what you’re doing (test environments, fault-tolerant apps, EMR processing, like Spark). Where you might get an RI for 50% off, I’ve seen spot instance discounts as high as 80-90%.
Some folks have done an amazing job of analyzing historical spot prices for the types of instances they need and have purchasing algorithms that help them both track price trends and buy & run their instances when it’s cheapest to do so.
_ Note: A word of caution here – spot price purchasing is something you want to keep an eye on. Because it’s demand-driven, spot pricing can sometimes (though it’s not often) spike above the price of on-demand pricing. So you’ll want to make sure you’ve accounted for that in any automation you build._
You’re not stuck
A common assumption that people make with their EC2 environments is that the cost “is-what-it-is” until the apps running there can be sunset or migrated to PaaS. Fortunately, that is rarely true. By leveraging some financial tools in your config, it’s highly likely you’ll be able to bring the cost of your EC2 environment down.
If you’re working in enterprise, this stuff will make the biz folks love you, and they’re likely who control your salary.
If you’re running a startup, this stuff can make the difference between you being able to hire more people, or even stay in business at all.
None of it’s hard, it just needs to be accounted for.
Top comments (3)
One silly bit that really should be removed
Let's just assume you never wrote that, because honestly, it won't convince anyone you deem a non-minimally-functional adult to take on your point of view, and anyone that is that in my view will think it's a bit strong.
Devs that own financial costs have one thing in common. Poor management and higher-up team members.
Finance deals with costings. It's their Job to say no if something is unreasonable. It's the CTO's job to ultimately approve or reject large infra costs, and redirect team energy. It's technical architects with an infrastructure focus to choose what infra plans to forward to CTO for approval. It is absolutely a terrible idea to ask someone who is a bog-standard, or even mid-level developer, or sysadmin to guess at infra, or take on yet more responsibilities, others should handle. It doesn't make a painter a poor painter to say they don't decide upon the structural engineering or budget.
I think I agree with you in the context of larger, traditional enterprise development where a dev's entire job might be to maintain a very narrow silo, like "these 5 classes related to the ERP".
In the context of end-to-end ownership, like one would see in a DevOps practice, or smaller, 2-pizza teams within an enterprise (actually, in any environment where decision-making is distributed) I do not agree.
I don't think it's unfair to ask devs to consider factors other than "just the code". Business cases, cost, and ethics come to mind.
Generally, I think throwing cost over the wall to finance is the same as throwing code over the wall to ops. Many of the decisions made at the code level determine cost, especially in a consumption model, like cloud. Finance isn't going to know what's reasonable, because they don't have the context. They only know what % of their budget something is eating.
If the team is stratified enough that there are architects, they likely carry that burden as part of their designs and reviews. Management will own cost at the macro level, but the micro-level decisions that devs make feed into the macro.
My expectation is not that a junior dev is 100% responsible for the cost of the infrastructure it takes to run their code, merely that they understand their ability to impact that cost and take that into account in their decisions.
We mostly agree. But again
It's not "throwing it over the wall" to buy a VCR, or entrust your friend who is all things AV to advise you or purchase a VCR for you (going 80's). Building your own VCR, getting right into it takes something you cannot get back. Time. So only build your own (metaphorical) VCR if you have to. It's not good advice to learn how to build one, or how much the parts cost.
Yes it would be great if you could ask coders to spec out your infra, you'd be short-term saving a bundle on experts. Long term, you'd be asking the painter about interior decor. They are separate skills.