This post was originally published on my blog.
Testing the code you write is an important step in the process of software engineering. It ensures that your software works as expected and reduces the risk of shipping bugs and vulnerabilities to production. Automated tests in particular play an important role when it comes to testing frequently and consistently. Continous integration makes them even more powerful.
In this blog post I'll show you an architecture for testing Node.js REST APIs that use a database in the background. There are some things you should consider in this scenario that we'll talk about. You'll see how to separate and organize your application's components in a way that you can test them independently. Therefore, we'll use two different approaches. On the one hand we setup a test environment in which we run ours tests against a test database. On the other hand we mock the database layer using mock functions so that we can run them in an environment in which we have no access to a database.
We'll start by writing unit tests for testing single components of our application. In the next step we combine those components and test them using integration tests. Last but not least, we setup a CI/CD pipeline using GitHub actions and run the tests on each push that is made.
Note that this is not a guide on how testing in general works. There are thousands of articles about frameworks like Jest, Mocha, Supertest and so on.
This is more a guide on how to prepare your Node application and test environment in a way that you can write tests effortlessly and efficiently with or without a database connection. There's also an example repo on GitHub. You should definitely check it out.
Disclamer: I know there really is no right or wrong when it comes to the architecture. The following is my prefered one.
Let's start with some of the tools we'll use. You should know most of them:
- Language: Typescript
- Server: Express
- Database: Postgres
- Testing: Jest & Supertest & Chai
- CI/CD: GitHub Actions
- Containerization: Docker
One benefit of this architecture is that you can use it with other databases than Postgres too, like MySQL for example. In this architecture we don't use any kind of ORM. Moreover, you can replace Jest with Mocha if that is your desired testing framework.
Application Architecture
The architecture of our application looks roughly like this:
node-api
├── api
│ ├── components
│ │ ├── user
| | │ ├── tests // Tests for each component
| | | │ ├── http.spec.ts
| | | │ ├── mock.spec.ts
| | │ | └── repo.spec.ts
| | │ ├── controller.ts
| | │ ├── dto.ts
| | │ ├── repository.ts
| | │ └── routes.ts
│ └── server.ts
├── factories // Factories to setup tests
| ├── helper.ts
| ├── abs.factory.ts
| ├── http.factory.ts
| └── repo.factory.ts
└── app.ts
Note: The example repo contains some more code.
Each component consists of the following four files:
- controller.ts: HTTP Handler
- dto.ts: Data Transfer Object (more)
- repository.ts: Database Layer
- routes.ts: HTTP Routing
The tests directory includes the tests of the according component. If you want to read more about this architecture, checkout this article of mine.
Config
We'll start by creating an .env.test file that contains the secret environment variables for testing. The npm Postgres package uses them automatically when establishing a new database connection. All we have to do is to make sure that they are loaded using dotenv.
NODE_PORT=0
NODE_ENV=test
PGHOST=localhost
PGUSER=root
PGPASSWORD=mypassword
PGDATABASE=nodejs_test
PGPORT=5432
Setting NODE_PORT=0 lets Node choose the first randomly available port that it finds. This can be useful if you run multiple instances of a HTTP server during testing. You can also set a fixed value other than 0 here. Using PGDATABASE we provide the name of our test database.
Next, we setup Jest. The config in jest.config.js looks as follows:
module.exports = {
preset: "ts-jest",
testEnvironment: "node",
roots: ["src"],
setupFiles: ["<rootDir>/setup-jest.js"],
}
And setup-jest.js like this:
require("dotenv").config({
path: ".env.test",
})
This snippet ensures that the appropriate environment variables are loaded from the provided .env file before running the tests.
Testing with database
Let's start with the assumption that we have a test database that we can use. This might be one in a GitHub Actions CI/CD pipeline for example. Later on I'll show you how to test your application without a database connection.
3 rules
At the beginning I said that there are some important things we want to consider that make life much easier when testing Node APIs:
- Separate the database layer
- Outsource the database connection initialization
- Outsource the HTTP server initialization
What do I mean by that?
Separate the database layer
You should have your own layer, separated from your business logic, that takes care of the communication with the database. In the example Git repo you can find this layer in a component's repository.ts file. This allows us to easily mock a layer's methods when we have no database available for testing.
Moreover, it's easier to replace your database system with another one.
export class UserRepository {
readAll(): Promise<IUser[]> {
return new Promise((resolve, reject) => {
client.query<IUser>("SELECT * FROM users", (err, res) => {
if (err) {
Logger.error(err.message)
reject("Failed to fetch users!")
} else resolve(res.rows)
})
})
}
Outsource the database connection initialization
You should already know that when writing tests against a database, you do not run them against the production one. Instead, you setup a test database. Otherwise, you run the risk messing up your production data.
Most of the time, your application connects to the database in a startup script, like index.js. After the connection is established, you start the HTTP server. That's what we want to do in our test setup too. This way we can connect to the database and disconnect from it gracefully before and after each test case.
Outsource the HTTP server initialization
It's a good practice, whether you use a database or not, to start the HTTP server from inside your tests. Just as we do for the database connection, we create a new HTTP server before and stop it after each test case.
This might look as follows: (you'll see the concrete implementation later on)
describe("Component Test", () => {
beforeEach(() => {
// Connect to db pool && start Express Server
});
afterEach(() => {
// Release db pool client && stop Express Server
});
afterAll(() => {
// End db pool
});
In particular the execution order is:
- Connect to the database pool
- Run the SQL seed (create tables)
- Start the Express server
- Run the tests
- Shutdown the Express server & release db pool client
- Repeat step 1 - 5 for each test case and close the pool at the end
Tests
Each component consists of two test files:
- repo.spec.ts
- http.spec.ts
Both of them make use of so called TestFactories which prepare the test setup. You'll see their implementation in the next chapter.
Note: If you have a look at the example Git repo, you'll see that there are two more: mock.spec.ts and dto.spec.ts. Former one is discussed later on. The latter is not covered in this article.
repo.spec.ts
A repository is an additional abstract layer that is responsible for interacting with the database like reading and inserting new data. That layer is what we test in here. Since a database is required in this case, a new pool client is created to connect to the database using the RepoTestFactory before each test case. And it is released, right after the test case is completed. At the end, when all test cases are finished, the pool connection is closed.
Example on GitHub
describe("User component (REPO)", () => {
const factory: RepoTestFactory = new RepoTestFactory()
const dummyUser: IUser = userFactory.build()
const dummyUserDTO: UserDTO = userDTOFactory.build()
// Connect to pool
beforeEach(done => {
factory.prepareEach(done)
})
// Release pool client
afterEach(() => {
factory.closeEach()
})
// End pool
afterAll(done => {
factory.closeAll(done)
})
test("create new user", async () => {
const repo = new UserRepository()
const user = await repo.create(dummyUserDTO)
expect(user).to.be.an("object")
expect(user.id).eq(1)
expect(user.email).eq(dummyUser.email)
expect(user.username).eq(dummyUser.username)
const count = await factory.getTableRowCount("users")
expect(count).eq(1)
})
})
http.spec.ts
Here, we test the integration of the user component's routes, controller and repository. Before each test case, a new pool client is created just as we did above. In addition, a new Express server is started using the HttpTestFactory. At the end, both are closed again.
Example on GitHub
ddescribe("User component (HTTP)", () => {
const factory: HttpTestFactory = new HttpTestFactory()
const dummyUser: IUser = userFactory.build()
const dummyUserDTO: UserDTO = userDTOFactory.build()
// Connect to pool && start Express Server
beforeEach(done => {
factory.prepareEach(done)
})
// Release pool client && stop Express Server
afterEach(done => {
factory.closeEach(done)
})
// End pool
afterAll(done => {
factory.closeAll(done)
})
test("POST /users", async () => {
const res = await factory.app
.post("/users")
.send(dummyUserDTO)
.expect(201)
.expect("Content-Type", /json/)
const user: IUser = res.body
expect(user).to.be.an("object")
expect(user.id).eq(dummyUser.id)
expect(user.email).eq(dummyUser.email)
expect(user.username).eq(dummyUser.username)
const count = await factory.getTableRowCount("users")
expect(count).eq(1)
})
})
Factories
The test factories are actually the heart of our tests. They are responsible for setting up and preparing the environment for each test case. That includes:
- Droping & creating all db tables
- Initializing the db connection
- Initializing the HTTP server
- Closing both of them again
There are four factories in total: AbsTestFactory, RepoTestFactory, HttpTestFactory and MockTestFactory. Each of them has its own Typescript class.
The last one is discussed in the chapter "Testing without database".
AbsTestFactory
The first one AbsTestFactory is an abstract base class that is implemented by the other three. It includes among others a method for connecting to the database pool and one for disconnecting from it.
export abstract class AbsTestFactory implements ITestFactory {
public poolClient: PoolClient
abstract prepareEach(cb: (err?: Error) => void): void
abstract closeEach(cb: (err?: Error) => void): void
public async getTableRowCount(name: string) {
const { rows } = await this.poolClient.query(
`SELECT COUNT(*) FROM ${this.poolClient.escapeIdentifier(name)};`
)
return rows.length ? +rows[0].count : 0
}
protected connectPool(cb: (err?: Error) => void) {
pool
.connect()
.then(poolClient => {
this.poolClient = poolClient
this.poolClient.query(this.seed, cb)
})
.catch(cb)
}
protected releasePoolClient() {
this.poolClient.release(true)
}
protected endPool(cb: (err?: Error) => void) {
pool.end(cb)
}
private seed = readFileSync(
join(__dirname, "../../db/scripts/create-tables.sql"),
{
encoding: "utf-8",
}
)
}
Using the create-tables.sql script, the factory drops and recreates all the tables after the connection is established:
DROP TABLE IF EXISTS users;
CREATE TABLE users (
id SERIAL PRIMARY KEY,
email VARCHAR(50) UNIQUE NOT NULL,
username VARCHAR(30) UNIQUE NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
RepoTestFactory
The RepoTestFactory is used by each component's repository test (repo.spec.ts) that you just saw above. All it does is use the parent class AbsTestFactory to connect to the database.
export class RepoTestFactory extends AbsTestFactory {
prepareEach(cb: (err?: Error) => void) {
this.connectPool(cb)
}
closeEach() {
this.releasePoolClient()
}
closeAll(cb: (err?: Error) => void) {
this.endPool(cb)
}
}
The methods prepareEach, closeEach and closeAll are called for each test case in the Jest beforeEach, afterEach and afterAll lifecycle.
HttpTestFactory
The last one HttpTestFactory is used by each component's HTTP test (http.spec.ts). Just like RepoTestFactory, it uses the parent class for the database connection. Furthermore, it initializes the Express server.
export class HttpTestFactory extends AbsTestFactory {
private readonly server: Server = new Server()
private readonly http: HttpServer = createServer(this.server.app)
get app() {
return supertest(this.server.app)
}
prepareEach(cb: (err?: Error) => void) {
this.connectPool(err => {
if (err) return cb(err)
this.http.listen(process.env.NODE_PORT, cb)
})
}
closeEach(cb: (err?: Error) => void) {
this.http.close(err => {
this.releasePoolClient()
cb(err)
})
}
closeAll(cb: (err?: Error) => void) {
this.endPool(cb)
}
}
Helpers
In helper.ts, there are two fishery objects which we can use to create dummy data during the tests.
export const userFactory = Factory.define<IUser>(({ sequence, onCreate }) => {
onCreate(
user =>
new Promise((resolve, reject) => {
pool.query<IUser>(
"INSERT INTO users (email, username) VALUES($1, $2) RETURNING *",
[user.email, user.username],
(err, res) => {
if (err) return reject(err)
resolve(res.rows[0])
}
)
})
)
return {
id: sequence,
email: "john@doe.com",
username: "johndoe",
created_at: new Date(),
}
})
export const userDTOFactory = Factory.define<UserDTO>(
() => new UserDTO("john@doe.com", "johndoe")
)
Usage:
// Returns new `IUser` instance
const dummyUser1 = userFactory.build()
// Returns new `IUser` instance & creates db entry
const dummyUser2 = await userFactory.create()
Rewind
Let's jump back to the repo.spec.ts and http.spec.ts test files from above. In both of them we used the factories' prepareEach method before each and its afterEach method after right each test case. The closeAll method is called at the very end of the test file. As you have just seen, depending on the type of factory, we establish the database connection and start the HTTP server if needed.
describe("Component Test", () => {
beforeEach(done => {
factory.prepareEach(done)
})
afterEach(() => {
factory.closeEach()
})
afterAll(done => {
factory.closeAll(done)
})
})
One important thing you should keep in mind is that for each test case that uses the database, the factory drops all the tables and recreates them using the provided SQL script afterwards. This way we have a clean database with empty tables in each test case.
Testing without database
So far we have run our tests against a test database, but what if we have no access to a database? In this case, we need to mock our database layer implementation (repository.ts), which is quite easy if you have separated it from the business logic, as I recommended in rule #1.
With mocks, the layer does not depend on an external data source any more. Instead, we provide a custom implementation for the class and each of its methods. Be aware that this does not affect the behavior of our controller since it does not care about where the data comes from.
Example on GitHub
const dummyUser = userFactory.build()
const dummyUserDTO = userDTOFactory.build()
const mockReadAll = jest.fn().mockResolvedValue([dummyUser])
const mockReadByID = jest
.fn()
.mockResolvedValueOnce(dummyUser)
.mockResolvedValueOnce(dummyUser)
.mockResolvedValue(undefined)
const mockCreate = jest.fn().mockResolvedValue(dummyUser)
const mockReadByEmailOrUsername = jest
.fn()
.mockResolvedValueOnce(undefined)
.mockResolvedValueOnce(dummyUser)
const mockDelete = jest.fn().mockResolvedValue(true)
jest.mock("../repository", () => ({
UserRepository: jest.fn().mockImplementation(() => ({
readAll: mockReadAll,
readByID: mockReadByID,
readByEmailOrUsername: mockReadByEmailOrUsername,
create: mockCreate,
delete: mockDelete,
})),
}))
After mocking the database layer, we can write ours tests as usual. Using toHaveBeenCalledTimes() we make sure that our custom method implementation has been called.
describe("User component (MOCK)", () => {
const factory: MockTestFactory = new MockTestFactory()
// Start Express Server
beforeEach(done => {
factory.prepareEach(done)
})
// Stop Express Server
afterEach(done => {
factory.closeEach(done)
})
test("POST /users", async () => {
const res = await factory.app
.post("/users")
.send(dummyUserDTO)
.expect(201)
.expect("Content-Type", /json/)
const user: IUser = res.body
cExpect(user).to.be.an("object")
cExpect(user.id).eq(dummyUser.id)
cExpect(user.email).eq(dummyUser.email)
cExpect(user.username).eq(dummyUser.username)
expect(mockCreate).toHaveBeenCalledTimes(1)
expect(mockReadByEmailOrUsername).toHaveBeenCalledTimes(1)
})
})
Note: cExpect is a named import from the "chai" package.
MockTestFactory
Just as we did in the other tests files, we use a test factory here as well. All the MockTestFactory does is run a new Express HTTP instance. It does not establish a database connection since we mock the database layer.
export class MockTestFactory extends AbsTestFactory {
private readonly server: Server = new Server()
private readonly http: HttpServer = createServer(this.server.app)
get app() {
return supertest(this.server.app)
}
prepareEach(cb: (err?: Error) => void) {
this.http.listen(process.env.NODE_PORT, cb)
}
closeEach(cb: (err?: Error) => void) {
this.http.close(cb)
}
}
One drawback we have using this approach is that the layer (repository.ts) is not tested at all because we overwrite it. Nevertheless, we can still test the rest of our application, like the business logic for example. Great!
Running
Using the the commands below we can run the tests with or without a database. Depending on the scenario, the files we do not want to test are excluded from execution.
{
"test:db": "jest --testPathIgnorePatterns mock.spec.ts",
"test:mock": "jest --testPathIgnorePatterns \"(repo|http).spec.ts\""
}
GitHub Actions
The final step is to create a CI/CD pipeline using GitHub actions that runs our tests. The according yaml file is available here. There's also a very good tutorial published on GitHub. You can decide whether to run the tests against a test database or use the mocked data layer. I decided to go with the former.
When running the pipeline with a test database, we need to make sure that we set the correct environment variables for it. Here you can find a test run.
Last words
My last tip is to have a look at the example repository on GitHub and to read it carefully There are some more tests and code snippets that I did not cover in this article. Moreover, checkout the links below. Happy coding!
Top comments (0)