DEV Community

How we lost $800/mo with Amazon ECS Fargate

Raphael Jambalos on August 18, 2019

It is well-known that containerizing an application can help reduce server costs. But if not designed properly, it can increase other costs such as...
Collapse
 
hcalsos profile image
Hans Christian Alsos

Interesting report, but in my view this sounds more like a problem with your vpc setup and nat-gateway. Have you considered using vpc endpoints for ecr access?
This would allow you to read/write to ecr without going though your nat-gateway, and by doing so, reduce your cost related to the nat-gateway.

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

Ohh, that's a great insight. I think that would be the most appropriate solution to this problem. I'll try that on my setup. Thanks Hans! :D

Collapse
 
alican profile image
Ali

I am looking to add VPC endpoints to avoid crazy NAT gateway bandwidth charges.

My understanding is that 'gateway' type of endpoints are free but 'Interface' costs money. ($0.01 per hour + $0.01 per GB at the time of writing)

S3 endpoints can be a 'gateway' type but ECR endpoints need to be 'interface' type.

so... I am not clear on these two things:

(a) Am I right to assume that, since ECR image storage is actually provided by S3, I would just need to have an S3 type of VPC endpoint to avoid these huge NAT gateway bandwidth charges?

OR

(b) Do I need to have ECR and S3 endpoints together and my cost saving is going to be paying $0.01 per GB instead of $0.045 per GB (nat gateway price)?

Collapse
 
chayanikaa profile image
Chayanika Khatua

Hi! I was inspired to research this based on your question, the effect of the Gateway endpoint alone vs ECR endpoints: dev.to/chayanikaa/cost-optimisatio...

Collapse
 
davisengeler profile image
davis • Edited

I've been worried about exactly this after building something with Fargate recently. I noticed the 'restart loop' behavior if you push up a crashing version, and it scared me into looking closer at how they bill for that image transfer. Happy to come across your post and discussion in the comments, but hate it caused such a big bill for you all. That's scary.

I'm a bit confused in my case, though, because my image in ECR is like 1.5GB and I'm not seeing any sort of transfer charges for that data. I have Fargate charges already, but nothing related to the transfer of the image (it's been a week or two with hundreds of cold starts). Is it possible those charges are much more delayed than Fargate's?

If I'm not actually being charged for it, how is that happening? I actually have 0 NAT Gateways on my account (that I can tell), yet I'm able to use my image from ECR in Fargate tasks.

Should I expect a big surprise bill coming soon? Based on the phrasing of their docs, I would have expected a cost of roughly 10¢ per cold start in my case since the first one I do blows past the ECR free tier for transfer out.

Collapse
 
davisengeler profile image
davis

Found this in the ECR pricing documentation:

Data transferred between Amazon Elastic Container Registry and Amazon EC2 within a single region is free of charge (i.e., $0.00 per GB).

I guess since EC2 is underlying Fargate and I'm using the same region, it's free in this case.

Question: What was the need for your NAT Gateway in your configuration? I don't have one and ECR <–> Fargate seem to be communicating fine.

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

Do you have VPC endpoints set up for your network? That's probably why your fargate instances can fetch ECR repositories without much charges.

If you don't have NAT gateway set up (and no VPC endpoint), you're probably using your Fargate containers in your public subnet where its using the Internet Gateway to fetch images from ECR. I'm not sure if Internet Gateways have a per-GB charge, I think it doesnt. If this is the case, you probably don't have to worry about this problem

Thread Thread
 
davisengeler profile image
davis • Edited

Thanks for the insights!

Collapse
 
manipulator01 profile image
Jack • Edited

Great read! thanks for sharing your story.
I wish there was an easy way to monitor the cost.

I also found it very hard to calculate fargate cost.

Does anyone know if AWS provides any cost calculator?.
I found this website fargate.org/ which does an OK job... but it's not complex enough for my needs

Collapse
 
chayanikaa profile image
Chayanika Khatua • Edited

Excellent article! I was wondering what solution you ended up implementing for this, all the interface endpoints or just the Gateway one?

I looked into this a bit with NAT Gateway metrics and different compositions: dev.to/chayanikaa/cost-optimisatio...

Collapse
 
peteonthings profile image
Peter P • Edited

Ah, thank you for this article. I am glad to know I am not alone.

We got hit by this with our ScheduledTasks which run in Fargate. We were testing development out of ScheduledTasks in a new Fargate cluster and we set a scheduled task to run every minute. So every minute we were downloading the image through our NatGateway!! Ack! Our excess bill was in the $2k range :((

Indeed, it seems the solution is VPC endpoints. Its crazy that AWS doesn't make this mandatory. Why would anyone want to go out to the public internet for their container on ECR?

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

AWS is so sneaky with this hidden charge. I feel crazy for missing it for months. I agree that VPC endpoints are the permanent fix for this. But I think a team should have someone reviewing their AWS bill every month to look into every charge so this kind of mistake can be caught early.

Collapse
 
rophilogene profile image
Romaric P.

Hi Raphael, very interesting feedback. Your experience sounds so familiar to me... That's one of the reasons why I have built Qovery. To have all the advantages of containers and AWS, without the disadvantages.

Collapse
 
jappyjan profile image
jappyjan

How does scaling work worth qovery? It’s not mentioned on their site...

Collapse
 
ramduos profile image
Ram

Hi Raphael,i came across the simar issue by using aws fargate and got billed very high couple of months back. I have raised the issue with AWS and got the refund.

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

Ohh! I'm glad I am not alone on this problem. I'll reach out to AWS as per your advice and maybe we'd get a refund too.

Collapse
 
rametta profile image
Jason

Very informative, I'm setting up fargate stuff now so this is great to know. luckily my docker images are only ~10mb so should be fine

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

Thank you Jason! It's always best practice to keep the Docker image small. Unfortunately for our team, our app has a lot of dependencies.

Collapse
 
mourik profile image
MOURIK

Interesting, thanks a lot !

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

Thank you Mourik!

Collapse
 
pharindoko profile image
Florian Fuß

Had the same issue. 300 $ until I found the issue.
Always use vpc endpoints for ecr endpoints when using fargate to avoid this issue.

Collapse
 
raphael_jambalos profile image
Raphael Jambalos

that's true!! i hate that AWS makes this like a hidden charge!! this is nowhere in their documentation yet it costs a fortune!