Check your settings, save mad loot

Photo by Christine Roy

TL;DR

I co-founded and have been leading the BootstrapCDN project since 2012. I have made many mistakes, here is one I hope others can learn from. “Set and forget” web services can be a blessing and a curse. No matter what service you are using so, log in and audit the bill.

Since 2013 we have been using S3 as our object store origin. You see, the beauty of S3 is how reliable/secure it is. That’s the blessing. The curse to that is you rarely have to login.

Late last November, I had to login to check a setting and had some time to kill so decided to check the cost explorer. I always knew our S3 costs where high but never dove into it because it was never an issue. So I decided I see what was happening and digging through the S3 logs, noticed strange query strings (e.g. ?v=1&v=2&v=3...&v=5409) which was treated as a separate object that couldn't be cached (on the CDN) since it was constantly changing, "Cache Busting" so to speak. I logged into the MaxCDN Control Panel, and disabled “Treat as separate cacheable item”.

This is what 1 simple change did:

Things I learned

• At the time I started the investigation, at our scale, a 0.43% non-cache hit ratio equaled 297mm+ requests going back to the origin i.e. S3.
• Making changes to a service that serves ~72 billion requests a month is scary but as long as you have a team to help monitor abnormalities, don’t be afraid.
• No matter how well things are working, login and audit!

Before change (October 2017):

After change (January 2018):

In closing

Writing this post wasn’t easy. Admitting to something so stupid opens yourself up to ridicule, but I am 99.57% positive there are others making similar mistakes. No matter what service you are using, login and audit. In the meantime, I'll be finding out why 211mm+ requests are cache misses.🤔