It is well-known that containerizing an application can help reduce server costs. But if not designed properly, it can increase other costs such as...
For further actions, you may consider blocking this person and/or reporting abuse
Interesting report, but in my view this sounds more like a problem with your vpc setup and nat-gateway. Have you considered using vpc endpoints for ecr access?
This would allow you to read/write to ecr without going though your nat-gateway, and by doing so, reduce your cost related to the nat-gateway.
Ohh, that's a great insight. I think that would be the most appropriate solution to this problem. I'll try that on my setup. Thanks Hans! :D
I am looking to add VPC endpoints to avoid crazy NAT gateway bandwidth charges.
My understanding is that 'gateway' type of endpoints are free but 'Interface' costs money. ($0.01 per hour + $0.01 per GB at the time of writing)
S3 endpoints can be a 'gateway' type but ECR endpoints need to be 'interface' type.
so... I am not clear on these two things:
(a) Am I right to assume that, since ECR image storage is actually provided by S3, I would just need to have an S3 type of VPC endpoint to avoid these huge NAT gateway bandwidth charges?
OR
(b) Do I need to have ECR and S3 endpoints together and my cost saving is going to be paying $0.01 per GB instead of $0.045 per GB (nat gateway price)?
Hi! I was inspired to research this based on your question, the effect of the Gateway endpoint alone vs ECR endpoints: dev.to/chayanikaa/cost-optimisatio...
I've been worried about exactly this after building something with Fargate recently. I noticed the 'restart loop' behavior if you push up a crashing version, and it scared me into looking closer at how they bill for that image transfer. Happy to come across your post and discussion in the comments, but hate it caused such a big bill for you all. That's scary.
I'm a bit confused in my case, though, because my image in ECR is like 1.5GB and I'm not seeing any sort of transfer charges for that data. I have Fargate charges already, but nothing related to the transfer of the image (it's been a week or two with hundreds of cold starts). Is it possible those charges are much more delayed than Fargate's?
If I'm not actually being charged for it, how is that happening? I actually have 0 NAT Gateways on my account (that I can tell), yet I'm able to use my image from ECR in Fargate tasks.
Should I expect a big surprise bill coming soon? Based on the phrasing of their docs, I would have expected a cost of roughly 10¢ per cold start in my case since the first one I do blows past the ECR free tier for transfer out.
Found this in the ECR pricing documentation:
I guess since EC2 is underlying Fargate and I'm using the same region, it's free in this case.
Question: What was the need for your NAT Gateway in your configuration? I don't have one and ECR <–> Fargate seem to be communicating fine.
Do you have VPC endpoints set up for your network? That's probably why your fargate instances can fetch ECR repositories without much charges.
If you don't have NAT gateway set up (and no VPC endpoint), you're probably using your Fargate containers in your public subnet where its using the Internet Gateway to fetch images from ECR. I'm not sure if Internet Gateways have a per-GB charge, I think it doesnt. If this is the case, you probably don't have to worry about this problem
Thanks for the insights!
Great read! thanks for sharing your story.
I wish there was an easy way to monitor the cost.
I also found it very hard to calculate fargate cost.
Does anyone know if AWS provides any cost calculator?.
I found this website fargate.org/ which does an OK job... but it's not complex enough for my needs
Excellent article! I was wondering what solution you ended up implementing for this, all the interface endpoints or just the Gateway one?
I looked into this a bit with NAT Gateway metrics and different compositions: dev.to/chayanikaa/cost-optimisatio...
Ah, thank you for this article. I am glad to know I am not alone.
We got hit by this with our ScheduledTasks which run in Fargate. We were testing development out of ScheduledTasks in a new Fargate cluster and we set a scheduled task to run every minute. So every minute we were downloading the image through our NatGateway!! Ack! Our excess bill was in the $2k range :((
Indeed, it seems the solution is VPC endpoints. Its crazy that AWS doesn't make this mandatory. Why would anyone want to go out to the public internet for their container on ECR?
AWS is so sneaky with this hidden charge. I feel crazy for missing it for months. I agree that VPC endpoints are the permanent fix for this. But I think a team should have someone reviewing their AWS bill every month to look into every charge so this kind of mistake can be caught early.
Hi Raphael, very interesting feedback. Your experience sounds so familiar to me... That's one of the reasons why I have built Qovery. To have all the advantages of containers and AWS, without the disadvantages.
How does scaling work worth qovery? It’s not mentioned on their site...
Hi Raphael,i came across the simar issue by using aws fargate and got billed very high couple of months back. I have raised the issue with AWS and got the refund.
Ohh! I'm glad I am not alone on this problem. I'll reach out to AWS as per your advice and maybe we'd get a refund too.
Very informative, I'm setting up fargate stuff now so this is great to know. luckily my docker images are only ~10mb so should be fine
Thank you Jason! It's always best practice to keep the Docker image small. Unfortunately for our team, our app has a lot of dependencies.
Interesting, thanks a lot !
Thank you Mourik!
Had the same issue. 300 $ until I found the issue.
Always use vpc endpoints for ecr endpoints when using fargate to avoid this issue.
that's true!! i hate that AWS makes this like a hidden charge!! this is nowhere in their documentation yet it costs a fortune!