DEV Community

Cover image for Testing Node.js + Mongoose with an in-memory database
Paula Santamaría
Paula Santamaría

Posted on • Updated on

Testing Node.js + Mongoose with an in-memory database

Last few weeks I've been working on creating Unit Tests for a Node.js and Mongoose application where most of the logic is handled by mongoose and MongoDB.

The first thing I tried was to create mocks to match every operation executed in mongoose and its different outcomes (at first it looked like the most logical thing to do). But half through the process I started to realize it was taking a lot of time, and what if the queries change? Will I have to change all my mocks as well?

After googling for a while I found this package on Github mongodb-memory-server which, simply put, allows us to start a mongod process that stores the data in memory. So I decided to give it a try.

In this article I'll tell you how to use an in-memory MongoDB process to test your mongoose logic without having to create any mocks.
If you want to go straight to the code, I created a Github repo that serves as example or boilerplate.

In-memory database pros & cons

I wasn't convinced about using an in-memory database instead of mocks at first so I did a bit of digging and come up with this list of pros an cons:

Pros:

  • No need for mocks: Your code is directly executed using the in-memory database, exactly the same as using your regular database.
  • Faster development: Given that I don't need to build a mock for every operation and outcome but only test the query, I found the development process to be faster and more straightforward.
  • More reliable tests: You're testing the actual code that will be executed on production, instead of some mock that might be incorrect, incomplete or outdated.
  • Tests are easier to build: I'm not an expert in unit testing and the fact that I only need to seed the database and execute the code that I need to test made the whole process a lot easier to me.

Cons:

  • The in-memory database probably needs seeding
  • More memory usage (dah)
  • Tests take longer to run (depending on your hardware).

In conclusion, the in memory database turned out to be perfect to test applications where the logic is mainly handled through database operations and where the memory and execution time are not an issue.

Let's start coding!

In this example we'll create a mongoose schema and a service that executes some operations with that schema.
We will later test the operations executed by the service.

This is how our project will look like once we finish:

Project folder structure

1. Setup & Install dependencies

Run npm init to setup your project, don't worry about the test script yet, will take care of it later.

And then execute the following commands to install all dependencies:

npm install --save mongoose
npm install --save-dev jest mongodb-memory-server
Enter fullscreen mode Exit fullscreen mode

Note: When installing mongodb-memory-server the mongod binaries will be downloaded an installed in node_modules/.cache. There are other options you can try like mongodb-memory-server-global which will download the binaries in %HOME/.cache so they'll be available to test other projects. Or mongodb-memory-server-core which will only download the binaries on server start if it can't find them.

Pick the option that best suits your needs.

More info in github.com/nodkz/mongodb-memory-server.

2. Write code to test

Now we'll build the model schema and the service that we'll test later.

2.a Product schema

// src/models/product.js

const mongoose = require('mongoose');

/**
 * Product model schema.
 */
const productSchema = new mongoose.Schema({
    name: { type: String, required: true },
    price: { type: Number, required: true },
    description: { type: String }
});

module.exports = mongoose.model('product', productSchema);
Enter fullscreen mode Exit fullscreen mode

2.b Product service

// src/services/product.js

const productModel = require('../models/product');

/**
 * Stores a new product into the database.
 * @param {Object} product product object to create.
 * @throws {Error} If the product is not provided.
 */
module.exports.create = async (product) => {
    if (!product)
        throw new Error('Missing product');

    await productModel.create(product);
}
Enter fullscreen mode Exit fullscreen mode

3. Configure jest

First, we'll add the test script to the package.json:

"scripts": {
    "test": "jest --runInBand ./test"
}
Enter fullscreen mode Exit fullscreen mode

Note: The --runInBand parameter will make sure all tests run serially. I do this to make sure there's only one mongod server running at once.

And finally add this to your package.json, since we are running a node application.

"jest": {
    "testEnvironment": "node"
}
Enter fullscreen mode Exit fullscreen mode

4. In-memory database handling

I wrote a module that executes some basic operations that I'll use to handle the in-memory database.

// tests/db-handler.js

const mongoose = require('mongoose');
const { MongoMemoryServer } = require('mongodb-memory-server');

const mongod = new MongoMemoryServer();

/**
 * Connect to the in-memory database.
 */
module.exports.connect = async () => {
    const uri = await mongod.getConnectionString();

    const mongooseOpts = {
        useNewUrlParser: true,
        autoReconnect: true,
        reconnectTries: Number.MAX_VALUE,
        reconnectInterval: 1000
    };

    await mongoose.connect(uri, mongooseOpts);
}

/**
 * Drop database, close the connection and stop mongod.
 */
module.exports.closeDatabase = async () => {
    await mongoose.connection.dropDatabase();
    await mongoose.connection.close();
    await mongod.stop();
}

/**
 * Remove all the data for all db collections.
 */
module.exports.clearDatabase = async () => {
    const collections = mongoose.connection.collections;

    for (const key in collections) {
        const collection = collections[key];
        await collection.deleteMany();
    }
}
Enter fullscreen mode Exit fullscreen mode

5. Write some tests

And finally we test our product service with the following code:

// tests/product.test.js

const mongoose = require('mongoose');

const dbHandler = require('./db-handler');
const productService = require('../src/services/product');
const productModel = require('../src/models/product');

/**
 * Connect to a new in-memory database before running any tests.
 */
beforeAll(async () => await dbHandler.connect());

/**
 * Clear all test data after every test.
 */
afterEach(async () => await dbHandler.clearDatabase());

/**
 * Remove and close the db and server.
 */
afterAll(async () => await dbHandler.closeDatabase());

/**
 * Product test suite.
 */
describe('product ', () => {

    /**
     * Tests that a valid product can be created through the productService without throwing any errors.
     */
    it('can be created correctly', async () => {
        expect(async () => await productService.create(productComplete))
            .not
            .toThrow();
    });
});

/**
 * Complete product example.
 */
const productComplete = {
    name: 'iPhone 11',
    price: 699,
    description: 'A new dual‑camera system captures more of what you see and love. '
};
Enter fullscreen mode Exit fullscreen mode

There are more test examples on the repo in case you want to check them out.

6. Try it out!

To try out our new tests just run npm test in the terminal 👩‍💻 and watch your tests come to life!

Top comments (39)

Collapse
 
yurigiovani profile image
yurigiovani • Edited

Nice way to implement tests.
I will try this way.
I've used mongo in docker container to perform my local tests.
Same in ci/cd.
Do you tried this way in ci/cd, like bitbucket pipelines?
Thanks for sharing.

Collapse
 
paulasantamaria profile image
Paula Santamaría

Yes, one of the first things I tried was executing this tests in a CI pipeline. I was worried to find any unexpected issues, but it worked perfectly!
For CI pipelines I'd recommend using mongodb-memory-server-core and a mongodb + node.js docker image. That way the npm install will not download mongod binaries but use the ones already installed instead.

Collapse
 
manuel114 profile image
manuel114

Hey, I've been struggling to get mongodb-memory-server to work on Circle Ci, any chance you could share your CircleCi config? I keep getting this error when running tests on the CI pipeline:

Error: Status Code is 403 (MongoDB's 404)
This means that the requested version-platform combination dosnt exist
Enter fullscreen mode Exit fullscreen mode

This is my db configuration:

const mongoose = require('mongoose');
const { MongoMemoryServer } = require('mongodb-memory-server');

const mongod = new MongoMemoryServer();

/**
 * Connect to the in-memory database.
 */
module.exports.connect = async () => {
    const uri = await mongod.getUri();

    const mongooseOpts = {
        useNewUrlParser: true,
        autoReconnect: true,
        reconnectTries: Number.MAX_VALUE,
        reconnectInterval: 1000,
    };

    await mongoose.connect(uri, mongooseOpts);
};
Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
paulasantamaria profile image
Paula Santamaría

🤔 looks like MongoMemoryServer is trying to download a binary for the default MongoDB version that is compatible with the OS running on your CircleCI server/container but can't find it.

I've never worked with CircleCI but I'd recommend you to check if the MongoDB version that MongoMemoryServer is trying to download is available for your OS & architecture.
Here's the code where the download URL for the binaries is generated, in case you want to check that out: MongoBinaryDownloadUrl.

Maybe you can set a specific version of MongoDB that exists for your platform, like this:

    mongod = await MongoMemoryServer.create({ binary: { version: '4.2.6' } });
    const uri = await mongod.getConnectionString();

Another way to go would be to activate debug mode so you can get more info on the error. Just set the debug option to "1" in the package.json config section. More info here

Thread Thread
 
kowshiksundararajan profile image
Kowshik Sundararajan

Thanks for the suggestion, Paula. I found that for my circleci build, specifying a version like 4.2.6 did not work but specifying latest worked.

Hope that helps!

Collapse
 
codenutt profile image
Jared

This is a very smart method of testing mongodb. much better than my code lol. Time for a refactor!

Collapse
 
paulasantamaria profile image
Paula Santamaría

Thank you! I've had great results using this method so far. Please share your experience if you try it!

Collapse
 
astonyao profile image
Aston Yao

awesome. thanks for sharing this.

I'd also check

const savedProduct = await productService.create(productComplete)
expect(savedProduct._id).toBeDefined()
expect(savedProduct.name).toBe(productComplete.name)
Collapse
 
thestoicdeveloper profile image
The Stoic Developer

Hi, nice article. I would just like to remind people a few things regarding mongodb-memory-server, please someone correct me if I'm wrong:

1) it's faster (~3x) for a battery of parallel tests, because when you use a real database you're usually constrained to one process (github.com/nodkz/mongodb-memory-se...)

2) it's slower (~4x) for one off tests, because it has to create the database engine every time. This is how I code, I always have one test running many times while TDDing (based on my own testing).

3) it is has somewhat the same speed in the other cases (based on my own testing). Please remember that mongo >= 3.2 runs with a memory cache by default.

docs.mongodb.com/manual/core/wired...

Collapse
 
tperrinweembi profile image
tperrin • Edited

Thanks for this awsome article. But since mongodb-memory-server version 7.0.0 we can't use anymore const mongoServer = new MongoMemoryServer(); but must use instead const mongo = await MongoMemoryServer.create(); . But I didn't succeed to adapt this to your code since that make me run await at file top level so I got the error "SyntaxError: await is only valid in async function". How would do you migrate your code to version 7?

Collapse
 
emmmarosewalker profile image
Emma Walker • Edited
import mongoose from "mongoose";
import { MongoMemoryServer } from "mongodb-memory-server";

let mongo: MongoMemoryServer;

/**
 * Connect to the in-memory database.
 */
export const connect = async () => {
    mongo = await MongoMemoryServer.create();
    const uri = mongo.getUri();

    const mongooseOpts = {
        useNewUrlParser: true,
        useUnifiedTopology: true,
    };

    await mongoose.connect(uri, mongooseOpts);
}

/**
 * Drop database, close the connection and stop mongod.
 */
export const closeDatabase = async () => {
    await mongoose.connection.dropDatabase();
    await mongoose.connection.close();
    await mongo.stop();
}

/**
 * Remove all the data for all db collections.
 */
export const clearDatabase = async () => {
    const collections = mongoose.connection.collections;

    for (const key in collections) {
        const collection = collections[key];
        await collection.deleteMany({});
    }
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
tperrinweembi profile image
tperrin • Edited

of course, simply move MongoMemoryServer.create into the connect method, I should have find this.... Thanks a lot!

Collapse
 
paulasantamaria profile image
Paula Santamaría

Thanks Emma!

Collapse
 
armoredelephant profile image
Keith Alleman

Excellent write up. I've been working on a MongoDB/GraphQL project to learn with, and have been getting familiar with Jest over the last couple of days. I'm going to try implementing your db-handler tonight!

Collapse
 
paulasantamaria profile image
Paula Santamaría

Thanks, Keith! How did that go?

Collapse
 
gmarokov profile image
Georgi Marokov • Edited

I get this error when using .deleteMany with TS:
Expected 1-3 arguments, but got 0.ts(2554)
index.d.ts(945, 16): An argument for 'filter' was not provided.

So I called it with empty obj: await collection.deleteMany({});

Collapse
 
parveen99 profile image
parveen99

Hey amazing article, I am doing my first assignment using node JS express and Mongo DB as a developer and doing Unit testing for the very first time as a newbie developer. This was the most effective article I found. Thank you, Paula. Also how should I run individual tests, npm test runs it on the whole right ? I

Collapse
 
paulasantamaria profile image
Paula Santamaría • Edited

In glad my article was helpful 😊.

If you follow this guide, npm test will execute all the tests in the "tests" folder (as specified in the package.json). You can however add some parameters (like --testNamePattern) when running npm test to execute only one test. I'd recommend reading my article Mastering NPM Scripts to learn how pass parameters to an npm script.

Also, if your using VSCode, checkout the Jest extension. It let's you run and debug tests individually.

Collapse
 
jimlynchcodes profile image
Jim Lynch

Nice article Paula!

I kind of feel like there could be more assertions in your final it block though than just, "expect ... to not throw".

To me it would be great if there was some was to look inside the mongo memory lib and say something like, "expect the fake mongoDb's collection to now have that additional document that I inserted".

Collapse
 
paulasantamaria profile image
Paula Santamaría

Thanks!
I completely agree. I actually added an extra test to check if the product exists after creating it on the repo, but didn't include it here because I wanted to keep the examples for the article simple.

Collapse
 
itbits2 profile image
ItBits2

I implemented this on Windows 10. Unfortunately it is not working. getConnectionString() never returns anything and test times out. This happened even if I increased jest timeout to 1 min. Am I missing something?

Collapse
 
itbits2 profile image
ItBits2
    const mongoose = require('mongoose');
    const { MongoMemoryServer } = require('mongodb-memory-server');
    const mongoUnit = require('mongo-unit');
    const mongod = new MongoMemoryServer();
    jest.setTimeout(60000);

    module.exports.connect = function(callback) {
        mongod.getUri().then((mongoUri) => {

            const mongooseOpts = {
              autoReconnect: true,
              reconnectTries: Number.MAX_VALUE,
              reconnectInterval: 1000,
            };

            mongoose.connect(mongoUri, mongooseOpts);

            mongoose.connection.on('error', (e) => {
              if (e.message.code === 'ETIMEDOUT') {
                console.log(e);
                mongoose.connect(mongoUri, mongooseOpts);
              }
              console.log(e);
            });

            mongoose.connection.once('open', () => {
              console.log(`MongoDB successfully connected to ${mongoUri}`);
              callback();
            });
          });
        }

    module.exports.closeDatabase = async () => {
        await mongoose.connection.dropDatabase();
        await mongoose.connection.close();
        await mongod.stop();
    }


    module.exports.clearDatabase = async () => {
        const collections = mongoose.connection.collections;

        for (const key in collections) {
            const collection = collections[key];
            await collection.deleteMany();
        }
    }
Collapse
 
brianle10680751 profile image
Brian Le

Hi, Great article, I have tried your configuration, and it works in some case.
However, if I use my customvalidate function in my schema , it not works. Is there any way to let the 'mongodb-memory-server' understand what we define in Schema - Model

Collapse
 
paulasantamaria profile image
Paula Santamaría

I haven't had any issues with mongoose validations in this setup. 'mongodb-memory-server' shouldn't prevent mongoose validations from working.
Would you please provide an example of the code that has the issue?

Collapse
 
sackingsand profile image
Yudha Putera Primantika • Edited

it's my first time using this, and I have no idea what going on the screenshot. I tried several test example and the all look like the picture here
dev-to-uploads.s3.amazonaws.com/i/...

Collapse
 
paulasantamaria profile image
Paula Santamaría

It's hard to know without seeing the code, but my best guess is that jest is not finding any tests to run. Try specifying where your test files are, like so: jest ./tests-folder.

Collapse
 
sackingsand profile image
Yudha Putera Primantika

That's just the thing, when it ran on my partner's pc it works just fine. Even after we continue building it there, the result on my laptop remains the same.

Collapse
 
axcosta profile image
André Xavier Costa

Many kudos for your nice and neat article! Picky comment is that last Github repo link is broken (which should lead to "more test examples" at the end).

Collapse
 
paulasantamaria profile image
Paula Santamaría

Thanks for letting me know! It's fixed now :)

Collapse
 
ranguna profile image
Luis Pais

I'm guessing this doesn't support multi document transactions ?
Since I think document-level locking is not supported on the memory engine.

Collapse
 
paulasantamaria profile image
Paula Santamaría

Hi Luis!
I haven't tried it myself yet, but apparently mongodb-memory-server does support multi-document transactions. However, for that to work, you must start the mongodb server as a replica set

Collapse
 
reallyone profile image
Kenneth Uyabeme

I love the db-handler module idea, makes for very clean code. Great article.

Collapse
 
paulasantamaria profile image
Paula Santamaría

Thank you, Kenneth! :)