Last month at DEV we made the switch from Memcache to Redis. This post talks about the why, the how, and also has a few gotchas to watch out for so you can get the most out of whatever cache solution you use.

Why We Switched

Memcache is a bit of a black box when it comes to caching. It is very hard to tell what is in it and how it is being used. When I first joined DEV I was greeted with a 75GB Memcache instance which we were paying a lot of money for. A 75GB cache seemed a bit big given our usage so I immediately started trying to figure out how I could view its contents. After a couple of hours of Googling, I figured out that unless you know the cached key you can't really see what is in Memcache. Not having visibility into your datastore is an SRE nightmare and it makes you feel pretty helpless and incapable of doing your job.

In addition to poor visibility, Memcache doesn't really play nice with Rails. In order to use it, you have to use the Dalli gem which I recently learned overrides a bunch of core cache methods in Rails to get Memcache to work. It's never great when you need to include an additional gem to get your cache solution to work, it just adds more complexity to the mix.

Finally, Memcache supports a very limited number of data structures. This is not a huge deal at first when all you want are simple key-value pairs like strings and numbers. However, as you grow, you may want more options for storing data and that's where Redis shines. It offers many different data structures that you can use to store data more efficiently.

For all the above reasons we chose to replace Memcache with Redis for our application cache. In addition to everything I mentioned above, Redis also has nice features such as asynchronous deletion and Lua scripts to handle complicated logic. Plus, it allows us to use a faster background worker service such as Sidekiq which we are in the process of migrating to from DelayedJob.

Once we decided that we wanted to make the switch the next question was how do we go about doing it.

Strategy For Moving Keys

In order to move to Redis, we decided to do it slowly by rolling keys over one or a couple at a time. The point of this was to ensure that a large cold cache didn't slow things down. It also allowed us to look at everything we were caching and evaluate if we really needed to cache it.

The way we did this was by creating a new Rails cache client using Redis and calling it RedisRailsCache. This new client behaved just like Rails.cache but instead of pointing to Memcache, it pointed to Redis.

RedisRailsCache = ActiveSupport::Cache::RedisCacheStore.new(url: redis_url, expires_in: DEFAULT_EXPIRATION)

Rolling over individual keys went smoothly and we really didn't have any surprises or slow down along the way. Then came the dilemma of rolling over the Rails fragment caches. Initially, the plan was to do it the same way we did the Rails.cache keys, by replacing the fragment calls one by one with a client of some sort that pointed to Redis.

One way we attempted to do this was by setting up a dual Rails cache as our Rails cache_store. I was pretty darn proud of the DualRailsStore that I had created. However, when we pushed it out to production we learned that the Memcache dalli gem was NOT storing things the way the Rails cache store expected which caused a lot of things to break. We quickly reverted the PR and went back to the drawing board.

All or Nothing

After looking at all of our options and evaluating their risk I decided to throw out a crazy idea in Slack:

Molly: "Because of the dalli_store gem, the way we cache things in Memcache is completely different from how Rails cache's things which is why trying to do them in parallel is such a PITA. SO I have another crazy idea: We rely a lot on Fastly for caching, so I'm wondering how bad do we think it would be to just flip to Redis? Could we schedule an hour or 2 of "maintenance"(basically expected degraded performance time) and just do it?"

This was my first time really throwing out a risky idea like this at DEV and I thought for sure people were going to slam it down and respond with "No way, you are crazy!" But instead, this is the response I got:

Ben: "That’s a possibility. I definitely have ideas about how to make it smoother."

My reaction IRL 👇 "Oh thank the lord they don't think I'm crazy!"

After that, I figured we could schedule it for a weekend or night when things slowed down, but to my surprise, Ben asked if I wanted to do it that afternoon. Of course, I replied "YEAH!" I would much rather rip the bandaid off then wait in anticipation.

Flipping the Switch

In order to flip to Redis, we made a few changes to the app to help lighten the heavy additional load we were expecting

• Added additional dynos(servers) to help with the load.
• Commented out the cache-busting for the home page and tag pages which can be served more stale
• Cut off traffic to “additional content boxes” and “social previews” controllers via Fastly since those were heavily cached and we could live without them for a little bit
• Removed “Additional articles” below posts and on the sidebar temporarily

Once all of that was done we added an ENV variable that we could use to flip over to Redis. Once things were in place we set the ENV variable and waited and watched. Ben and I both were expecting to see Redis traffic spike and load to slam our servers. We braced for the worst. But the worst never came.

At one point we thought possibly the ENV variable wasn't working so I jumped into Redis to see if fragment keys were coming in. Sure enough, they were! Redis traffic jumped a little but then the number of keys slowly and steadily climbed as people moved around the site. There were no giant spikes or dramatic slowdowns. Honestly, none of our alarms even went off. Ben and I watched and waited for almost 2 hours before we could convince ourselves that things were probably going to be fine and sure enough they were.

I had our Redis storage cranked to 25 GB because after having a 75GB Memcache I had no idea what to expect. Turns out there was a lot of cruft in Memcache because in Redis right now the size of our cached data sits around 3 GB.

Cacheing Gotcha's

During this whole process, I did A LOT of looking at and evaluating cache's across DEV's platform and made a lot of changes even before making the switch to Redis. Here are a couple of common problems I saw that you should watch out for when using any sort of cache store like Redis or Memcache.

Creating an Unnecessary External Request

Redis and Memcache are both super fast which is why they are great to work with. However, you are still making an external request from your server when you are talking to them and that external request takes time.

A few times I saw chunks of HTML that were being cached, but those chunks weren't making any external requests when they rendered. In other words, rather than serving the HTML from the server we were instead making an external request to get it from Redis. Redis may be fast, but it can't beat the speed of internal memory. If a chunk of code is not making any external requests, don't add one by unnecessarily caching it. You will only slow your app down in the process. Here is one example of a place we removed caching completely.

Not Expiring Keys

Especially when you are starting out with an application it can be really easy to shove stuff into an external cache without an expiration. This easily can get out of hand and lead to things like a 75GB cache full of junk you don't actually need.

When you set up your cache store client in your code make sure you set a default expiration. Every time a key goes in, it needs to have an expiration set. You can try to rely on devs to ensure they are set, but in my experience, we are all pretty unreliable. The easiest thing to do is have the code set it by default whenever an expiration is not explicitly set. That way nothing slips through the cracks.

Low Cache ROI

Adding a cache adds complexity and overhead. You now have to make sure that the key is expired when necessary otherwise you end up with outdated information being shown to users. You also now have another variable to take into consideration for testing and development. Do you cache things in those environments? Do you stub the cache? etc. Because of that added complexity, you want to make sure you are actually getting benefits from your cache when you use it.

The purpose of a cache is to make things fast by taking data that normally takes a long time to retrieve, remembering it, and serving it without all the original overhead of retrieving it from its original source. However, sometimes the cache isn't worth it.

One scenario is if you are caching an already very fast request. For example, if you are caching a very fast User.find() command you need to consider, is the small increase in speed from a cache worth having to ensure anytime that user changes you have to bust that cache?

Another scenario where a cache might not be ideal is if you are not requesting the data a lot. Let's say you cache a page view for a user so if the user reloads the page everything is lightning fast. That's awesome if the user is going to sit there and reload the page over and over. But if they look at the page and move on, then the cache is not doing you any good.

Finding these scenarios can be tricky, but one way to do it is by looking at your Hit rate. Most cache databases will tell you your hit rate as a percentage like 75%. That means 75% of the time a key is requested, the cache has it. The higher the hit rate, the more use your cache is getting and the more it is speeding things up.

Too Many Unique Keys

This kind of relates to the ROI scenario I talked about above. One thing to watch out for is too many unique keys. Let me explain what I mean. Say you have this chunk of code for caching:

Rails.cache.fetch("user-follow-count-#{id}-#{updated_at.rfc3339}", expires_in: 1.hour) do 
  followers.count
end

That code caches the number of followers a user has and we have added the updated_at timestamp to help ensure that it doesn't get stale if a user is updated. However, there are a lot of ways a user can be updated and its follower count won't change. For example, we could update the name or email or many other things and each time we would be creating a new cache key here while the follower count remains unchanged.

A better way to do this would be to name our cache key with something more specific like the last_followed_at timestamp:

Rails.cache.fetch("user-follow-count-#{id}-#{last_followed_at.rfc3339}", expires_in: 1.hour) do 
  followers.count
end

This way the cache key will only be reset if the user is followed which is appropriate since that means the follower count will have changed. Be on the lookout for situations like these where you might be using an expiration method like updated_at that is too aggressive and lowers the effectiveness of your cache. Ideally, the cache key should only change when the value of the cache changes.

Happy Caching!

Hopefully, you found this post educational and our experience at DEV switching our cache store can help you decide if a switch is right for you. If you want more details or would like to see all the PRs that it took to make this transition checkout this Github issue. Let me know if you have any questions or any part of the post is not clear!