DEV Community

Cover image for Building Cache Layer Using Redis and Mongoose
Ahmed Magdy
Ahmed Magdy

Posted on

Building Cache Layer Using Redis and Mongoose

Introduction.

If you ever built an api you will find that you will be needing to cache some GET requests that repeat alot and a find (if you are using mongoose) or select (sql) queries can be expensive over time. We are going to introduce a solution to this problem in this article.

Solution.

We will be following a very simple strategy here, But before we start you need to be familiar with mongoose and node.js

Strategy

Imagine we are working with a Query to fetch all dev.to blogs and The model will be called Blogs

Blogs Model

const blogSchema = new mongoose.Schema({
    owner : {
        // user in the database 
        type: mongoose.Types.ObjectId,
        required: true,
        ref: "User"
    },
    title: {
        type : String,
        required: true
    },
    tags: {
        type : [mongoose.Types.ObjectId],
    },
    blog: {
        type : String
    }
});
Enter fullscreen mode Exit fullscreen mode

now the request to fetch all the blog

app.use("/api/blogs",(req,res,next)=>{
         const blogs = await Blogs.find({}); 
          res.send(blogs);
});
Enter fullscreen mode Exit fullscreen mode

now after we get the image of what we are working with lets get back to the strategy

  • send a query to the database to ask for a certain thing
  • if this query has been fetched before aka exists in cache (redis)?
  • if yes, Then return the cached result
  • if no, Cache it in redis and return the result

The trick here is that there is a function in mongoose that is automatically executed after every operation
The function is called exec.

so we need to overwrite this exec function to do the caching logic.

first step to overwrite
const exec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async function (){
    // our caching logic
    return await exec.apply(this, arguments);
}
Enter fullscreen mode Exit fullscreen mode

now we need to make a something that tells us what gets cached and what doesn't. Which is a chainable function.

making the chainable function
mongoose.Query.prototype.cache = function(time = 60 * 60){
    this.cacheMe = true; 
    // we will talk about cacheTime later;
    this.cacheTime = time;
    return this;
}
Enter fullscreen mode Exit fullscreen mode

So Now if i wrote

Blogs.find({}).cache(); // this is a valid code
Enter fullscreen mode Exit fullscreen mode

Now if you are not familiar with Redis GO GET FAMILIAR WITH IT. there are thousands of videos and tutorials and it won't take that much time.

We need some data structure or types for the cached results. After some thinking, I've found out this is the best structure and I will explain why.

Alt Text.

Blogs is the collection name;

let's say you are doing Blogs.find({"title" : "cache" , user : "some id that points to user" })

then Query will be { "title" : "cache" , "user" : "some id ... " , op : "find" // the method of the query } ;

result is the result we got from database;

This structure is called NestedHashes.

Why we are doing Nested Hashes like this

we need to say if Blogs got a new Update or Insert or Delete operation delete the cached result. Because the cached result is old and not updated by any of the new operations.

NOW back to code.

mongoose.Query.prototype.exec = async function(){
    const collectionName = this.mongooseCollection.name;

    if(this.cacheMe){   
      // You can't insert json straight to redis needs to be a string 

        const key = JSON.stringify({...this.getOptions(),
             collectionName : collectionName, op : this.op});
        const cachedResults = await redis.HGET(collectionName,key);

      // getOptions() returns the query and this.op is the method which in our case is "find" 

        if (cachedResults){
          // if you found cached results return it; 
            const result = JSON.parse(cachedResults);
            return result;
        }
     //else 
    // get results from Database then cache it
        const result = await exec.apply(this,arguments); 

        redis.HSET(collectionName, key, JSON.stringify(result) , "EX",this.cacheTime);
       //Blogs - > {op: "find" , ... the original query} -> result we got from database
        return result;
    }

    clearCachedData(collectionName, this.op);
    return exec.apply(this,arguments);
}
Enter fullscreen mode Exit fullscreen mode

Remember the part where I said we need to clear cached data in case of Update, Insert or Delete.

clear the cached data

async function clearCachedData(collectionName, op){
    const allowedCacheOps = ["find","findById","findOne"];
    // if operation is insert or delete or update for any collection that exists and has cached values 
    // delete its childern
    if (!allowedCacheOps.includes(op) && await redis.EXISTS(collectionName)){
        redis.DEL(collectionName);
    }
}
Enter fullscreen mode Exit fullscreen mode

Expected Results

Much faster find queries.

What to Cache

  • Don't Cache large data Imagine if you have a find query that return 20 MB or even 100 MB worth of data you will be slowing down your whole application.
  • Don't Cache Requests that don't get a lot of traffic and that is highly dependent on your application.
  • Don't Cache Important Data like users or transactions.

Final Notes

  • My redis setup.
  • cacheTime paramter is option I put a default of 1 hour but you can edit it as you wish, i suggest 1 or 2 days.

Top comments (4)

Collapse
 
zisan34 profile image
Fazlul Kabir

Thanks for the article. It was a great starting point for me. But I had to make some improvements to get accurate behaviour. Here's the gist: gist.github.com/zisan34/64f5029449...

Collapse
 
ahmedmagdy11 profile image
Ahmed Magdy

I did some improvements on it after I wrote the article. Your comment is much appreciated though.

Collapse
 
matiusnugroho profile image
Math

where to override exec, your instruction is not clear

Collapse
 
ahmedmagdy11 profile image
Ahmed Magdy

I override exec at the start of the application in a module and make load at start up.