DEV Community

Cover image for Check your settings, save mad loot

Check your settings, save mad loot

Justin Dorfman
👋 Hi! I started to get heavily involved with the FOSS community in 2012. I contributed to Bootstrap, Font Awesome, jQuery, GNU Bash, and many others. I also host a weekly podcast called "Sustain".
Updated on ・2 min read

Photo by Christine Roy


I co-founded and have been leading the BootstrapCDN project since 2012. I have made many mistakes, here is one I hope others can learn from. “Set and forget” web services can be a blessing and a curse. No matter what service you are using so, log in and audit the bill.

Since 2013 we have been using S3 as our object store origin. You see, the beauty of S3 is how reliable/secure it is. That’s the blessing. The curse to that is you rarely have to login.

Late last November, I had to login to check a setting and had some time to kill so decided to check the cost explorer. I always knew our S3 costs where high but never dove into it because it was never an issue. So I decided I see what was happening and digging through the S3 logs, noticed strange query strings (e.g. ?v=1&v=2&v=3...&v=5409) which was treated as a separate object that couldn't be cached (on the CDN) since it was constantly changing, "Cache Busting" so to speak. I logged into the MaxCDN Control Panel, and disabled “Treat as separate cacheable item”.

This is what 1 simple change did:

Things I learned

  • At the time I started the investigation, at our scale, a 0.43% non-cache hit ratio equaled 297mm+ requests going back to the origin i.e. S3.
  • Making changes to a service that serves ~72 billion requests a month is scary but as long as you have a team to help monitor abnormalities, don’t be afraid.
  • No matter how well things are working, login and audit!

Before change (October 2017):

After change (January 2018):

In closing

Writing this post wasn’t easy. Admitting to something so stupid opens yourself up to ridicule, but I am 99.57% positive there are others making similar mistakes. No matter what service you are using, login and audit. In the meantime, I'll be finding out why 211mm+ requests are cache misses.🤔

Discussion (4)

ben profile image
Ben Halpern

Damn the size of the numbers mentioned in this post amazes me.

jdorfman profile image
Justin Dorfman Author

It wasn't overnight:

nebojsac profile image
Nick Cinger

Hah, awesome.
AWS is such a money sink until you start realizing where you're losing the most. It takes a while to a get your bearings, but until then you're already down a few K.

Thanks for sharing real numbers, always interesting to read something like this!

r3lik profile image
Mike Czarny

I got charged ~$100 on AWS for a Docker project that I forgot about. All it did was serve cat gifs.