Building Backend with Composable Services

About Composable Systems

Working inside a well-structured backend code feels like playing with Lego blocks. Many backend developers love opinionated frameworks (like Spring, Ruby on Rails, NestJS) because these frameworks make it easier to split their code into simple components.

In this post, we'll explore the essence of composable systems, and apply it to an (unopinionated) ExpressJS project.

To build a composable system, each component should have a simple & clear boundary. There are two main offenders that hurt composability of a component:

Global & Top-level Variable

Global variables or module-level variables smudge the boundary between components. Environment variables are a good example of this pattern. It is so easy to access process.env from anywhere, so we tend to use them without thinking too much about it. This makes it harder to check which function depends on which environment variable. In other words, the function's dependency gets blurry.

Big Interface

Sometimes, interfaces are very clear but too big. This is as bad as blurry dependencies because it is difficult to swap out a component with big interface. ORMs (object-relational mapping) and database adapters are good examples of this pattern.

By using an ORM, developers can access database with a clean interface; however, most ORMs have extremely big interfaces. It is near impossible to swap out an ORM with another component.

Refactoring for Simpler Structure

Ok, so we have identified the problem, but how do we fix it? Let's look at some examples and refactoring techniques.

Global & Top-level Variable

Example

Environment variables are often used like this:

// env.ts
import dotenv from "dotenv";
dotenv.configure();
export const SESSION_SECRET = process.env.SESSION_SECRET;
export const DATABASE_URI = process.env.DATABASE_URI;

import { DATABASE_URI } from "./env";
mongoose.connect(DATABASE_URI);

Top-level variables in this example blur the boundary between components as mentioned above.

Refactoring

Instead of exposing DATABASE_URI as top-level variable, we can put it in a function scope. Let's call the wrapper function EnvService.

// env.ts
import dotenv from "dotenv";
export const EnvService = () => {
  dotenv.configure();
  return {
    SESSION_SECRET: process.env.SESSION_SECRET,
    DATABASE_URI: process.env.DATABASE_URI,
  };
};
export type EnvService = ReturnType<typeof EnvService>;

To express that our database depends on environment variables, we can write a wrapper function for database as well.

import { EnvService } from "./env";
export const DatabaseService = (env: EnvService) => {
  mongoose.connect(env.DATABASE_URI);
  ...
};

Now it is super clear that our database depends on environment variables. We don't need global variables anymore.

Big Interface

Then what about big interfaces?

Example

It is common to use ORM classes directly inside request handlers.

app.get("/user/:id", async (req, res, next) => {
  try {
    const user = await UserModel.findById(req.params.id);
    //                 ^ like this one
    res.send({ user });
  } catch (err) {
    next(err);
  }
});

There are two problems with this implementation.

1. UserModel is a top-level class. The "/user/:id" handler depends on UserModel, but this dependency is not explicit.
2. UserModel has big interface. Most ORM libraries expose huge interfaces. It is unavoidable because an ORM needs to support a lot of database features & configurations.

In short, the request handler is tightly coupled with UserModel. There is no simple way to slice them apart cleanly.

Refactoring

We can attempt the same approach to make dependencies explicit. Let's start from the ORM.

// user.ts
export const UserService = () => {
  return {
    getById: (userId: string) => UserModel.findById(userId),
  };
};
export type UserService = ReturnType<typeof UserService>;
// type: { getById: (userId: string) => Promise<User> }

We have wrapped the complex top-level UserModel in a UserService function. UserService has much smaller interface compared to UserModel.

It's time to express the dependency between UserService & the request handler.

First, extract the handler into a separate function.

const userHandler: RequestHandler = async (
  req, res, next
) => {
  try {
    const user = await UserModel.findById(req.params.id);
    res.send({ user });
  } catch (err) {
    next(err);
  }
};
app.get("/user/:id", userHandler);

Then, express the dependency by wrapping the request handler with a function.

const UserHandler = (
  userService: UserService
): RequestHandler => async (req, res, next) => {
  try {
    const user = await userService.getById(req.params.id);
    res.send({ user });
  } catch (err) {
    next(err);
  }
};
app.get("/user/:id", UserHandler(userService));

Nice, now we have a clean interface between the request handler and the service!

Final Touch

Wrapping top-level variables & expressing dependencies as parameters are excellent ways to simplify boundaries between components. (This technique is called "inversion of control")

Inversion of control

In software engineering, inversion of control (IoC) is a programming principle. IoC inverts the flow of control as compared to traditional control flow. In IoC, custom-written portions of a computer program receive the flow of control from a generic framework. A software architecture with this design inverts control as compared to traditional procedural programming: in traditional programming, the custom code that expresses the purpose of the program calls into reusable libraries to take care of generic tasks, but with inversion of control, it is the framework that calls into the custom, or task-specific, code.

View on Wikipedia

But you may have noticed that we still don't have a way to provide required services to request handlers.

app.get("/user/:id", UserHandler(userService));
// Where does `userService` come from?

We can create a meta service for this purpose. A service that provides other services.

import { EnvService } from "./env";
import { SessionService } from "./session";
import { UserService } from "./user";

type ServiceMap = {
  user: UserService;
  env: EnvService;
  session: SessionService;
};

// Meta service
export const ServiceProvider = () => {
  // Initialize services.
  const env = EnvService();
  // A service can depend on another service,
  const user = UserService(env);
  // or multiple services.
  const session = SessionService(env, user);

  const serviceMap: ServiceMap = {
    user,
    env,
    session,
  };

  /**
   * Get service by service name.
   */
  const getService = <TServiceName extends keyof ServiceMap>(
    serviceName: TServiceName
  ) => serviceMap[serviceName];

  return getService;
};

export type ServiceProvider = ReturnType<typeof ServiceProvider>;

This meta service can be used like this:

const service = ServiceProvider();
const env = service("env"); // EnvService

Let's express that our app depends on ServiceProvider:

// app.ts
import express from "express";
import { ServiceProvider } from "./service";
import { UserHandler } from "./handlers";

export const App = (service: ServiceProvider) => {
  const app = express();
  app.get("/user/:id", UserHandler(
    // Provide required service with `ServiceProvider`.
    service("user"),
  ));
  return app;
};

Finally, initialize ServiceProvider and pass it to App!

// index.ts
import { App } from "./app";
import { ServiceProvider } from "./service";

const service = ServiceProvider();
const app = App(service);
app.listen(service("env").PORT);

What's Good About This Pattern?

By organizing functionalities into services, we get very straightforward project structure to reason about.

• Modifying service implementation is easier when the interface is small & explicit. Swapping out Firestore with MongoDB? That only affects the service that directly interacts with the database. The rest of the codebase is largely unaffected. The same can't be said if our request handlers are using ORM classes directly. Of course we don't change database every week, but the point is that changes are easier to make.
• Testing gets much easier. This is related to the first point, because we replace services with mocks during testing. Creating a test double (a.k.a. mock, fake, stub) is easier when the component has small & explicit interface. Also, checking which component to mock is effortless with inversion of control. All dependencies are expressed as parameters, so we just need to check those parameters.

Summary

Here's a summary of the refactoring technique we used.

1. Wrap top-level variables/functions into a service function. The service function's interface should be small.
2. Extract request handlers as separate functions.
3. Express dependencies as parameters (inversion of control).
4. Create a meta service (ServiceProvider).
5. Provide services to request handlers via ServiceProvider.

If you want to take a look at this pattern in action, please check out my personal project:

• service function
• request handler
• ServiceProvider
• test code