Containerize your multi-services app with docker compose

This article was posted first on my personal blog mariusniemet.me

In this previous article, we learned what a container is and took a brief introduction by learning how to create an image and then run a container, it was a good starting point.

But with the new industry standard most of the time you will work on apps made by multiple services that have dependencies between them, frontend, backend, database, caching, queue, and so on.

Project architecture

The image above describes the architecture of the project we want to containerize, as you can see we have four different services

• Front: a single-page app built with React
• Back: a graphQL server built with Nestjs
• Database: a MySQL db
• Cache: a Redis cache

If you want to follow along you have to download the source code here.

What is docker compose?

Docker Compose is a tool for defining and running multi-container applications. Compose simplifies the control of your entire application stack, making it easy to manage services, networks, and volumes in a single, comprehensible YAML configuration file.

How to create a stack?

First, you have to create a Dockerfile for each custom service (service for which you are responsible for building the image), for our current project, we have to create a Dockerfile for the front and back services.

Front

cd client

create a file named Dockerfile and add the content below:

FROM node:18-alpine

WORKDIR /app

COPY package.json .

RUN npm install

COPY . .

EXPOSE 5173

CMD [ "npm", "run", "dev" ]

Back

cd graphql-server

create a file named Dockerfile and add the content below:

FROM node:16-alpine AS base

# Set working directory
WORKDIR /app

# Copy package.json and package-lock.json
COPY package*.json ./

FROM base AS dev
# Install dependencies
RUN npm install 

# Copy source code
COPY . .

# Expose port 3000
EXPOSE 3000

# Start app
CMD ["npm", "run", "start:dev"]

FROM base as prod 
ENV NODE_ENV production 

# Install app dependencies
RUN npm install

# Bundle app source
COPY . .

# Creates a "dist" folder with the production build
RUN npm run build

# Start the server using the production build
CMD [ "node", "dist/main.js" ]

The compose file

Once we have the Dockerfile for our services the next step is to create a compose.yaml file, docker compose relies on that file for the stack configuration. Compose also supports docker-compose.yaml and docker-compose.yml for backward compatibility with earlier versions. If both files exist, Compose prefers the canonical compose.yaml.

name: composeintroduction

If you don’t set a name compose will use the name of the directory in which the compose file is located.

Services

A service is an abstract definition of a computing resource within an application. A service is defined by a Docker image and a set of runtime arguments (mostly docker run arguments). While creating a service you can use an existing image by setting the image name and tag compose will fetch it from the registry for you or you can ask compose to build a new image for you by specifying the path to the Dockerfile.

Let’s create the compose service for the backend:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile

services is a top-level argument and all our services will be declared here.

First, we have the service name in this case it’s backend and since for that service, we won’t rely on an existing image we have to specify how compose should build the image that’s the role of the build section, context is used to specify the subfolder in which the Dockerfile is located and the dockerfile argument is to specify the name. If your file is named Dockerfile you can remove that argument but if the name is different for any reason you have to specify it.

If your Dockerfile uses a multi-stage build you can specify the stage you are targeting and the command you want docker-compose to run inside your container once it’s ready.

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev

From now we will have a service named backend that will be built using the Dockerfile inside the graphql-server subfolder, the target stage will be dev and the command used will be npm run start:dev

Port definition:

As we have mentioned above all the arguments you usually pass to the docker run command are available here so to define ports for our container we use the syntax below:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      -"3000:3000"

The port at the left is the external accessible on the host machine and the right is the port inside the container (the same exposed in the Dockerfile).

Environnment variable

There are multiple ways to pass environment variables to a service:

• By using a file: An .env file in Docker Compose is a text file used to define environment variables that should be made available to Docker containers when running docker compose up. This file typically contains key-value pairs of environment variables, and it allows you to centralize and manage configuration in one place.

The .env file content:

DB_PASSOWORD=password
DB_USERNAME=user

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      -"3000:3000"
    env_file:
      - .env

• By passing key-value directly: There are two syntaxes for that

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      -"3000:3000"
    environment:
      DB_PASSWORD: password
      - DB_USERNAME=user

It’s up to you to choose the way you want to set variables, for the rest of this article we will use the syntax below:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      -"3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user

Now let’s create the service for the database.

services: 
   database: 
      image: mysql:latest
      environment: 
        MYSQL_ROOT_PASSWORD: password
        MYSQL_DATABASE: db
        MYSQL_USER: user
        MYSQL_PASSWORD: password

For the database service, since we will use the MySQL official image we don’t need the build section here, compose will fetch the image with the tag latest for us from the registry.

Last update the compose file by adding the remaining services:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      - "3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user

  database: 
    image: mysql:latest
    environment: 
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: db
      MYSQL_USER: user
      MYSQL_PASSWORD: password

  cache: 
    image: redis:latest
    ports:
    - "6380:6379"


  frontend: 
    build: 
      context: ./client
      dockerfile: Dockerfile
    command: npm run dev
    ports: 
    - "5173:5173"

Now we have added the frontend service and the cache service. We can now run our stack. First, we have to build the images by calling the command below inside our project’s folder :

docker compose build

Then compose will build the backend and the frontend image by using the specifications we have added in the compose file, your terminal should display something like the image below:

Once the images are built you can run the stack by calling the command below:

docke compose up -d 

The flag -d is for running the command in detach mode. By running this command compose will run all the containers declared in the compose file, for the backend and frontend compose will use the images built first for the database and the cache compose will fetch the images from the registry.

If everything went well you should have the result below just make sure the ports we have exposed aren’t used by other containers on your machine.

Each container’s name includes the stack name, the service name, and a number. Here we have one since there is only one instance by service.

Communication

Once we have our services running we want to establish communication between them, the backend should talk to the database and the cache, and the frontend should talk to the backend.

There are two things to consider here:

• The first is that compose creates a default network for your entire stack if you pay attention to the image above you have the first line Network composeintroduction_default that shows the name of the default network.
• The second is that compose comes with a DNS so if your services are in the same network you can use the service name + the port to communicate compose will figure out the IP address of the service you want to communicate with.

Based on these two pieces of information we understand that we can already establish the communication between our services just by using the service name, let’s update the backend first:

import { CacheModule } from '@nestjs/cache-manager';
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import * as redisStore from 'cache-manager-redis-store';
import { AppController } from './app.controller';
import { AppService } from './app.service';

@Module({
  imports: [
    TypeOrmModule.forRoot({
      type: 'mysql',
      host: 'database',
      port: 3306,
      username: process.env.DB_USERNAME,
      password: process.env.DB_PASSWORD,
      database: process.env.DB_NAME,
      entities: ['dist/**/*entity.js'],
      synchronize: true,
    }),
    CacheModule.register({
      isGlobal: true,
      store: redisStore,
      host: 'cache',
      port: 6379,
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Then the front:

import { ApolloClient, ApolloProvider, InMemoryCache } from "@apollo/client";
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App.tsx";
import "./index.css";

const client = new ApolloClient({
  uri: "http://backend:3000/graphql",
  cache: new InMemoryCache(),
});

ReactDOM.createRoot(document.getElementById("root")!).render(
  <React.StrictMode>
    <ApolloProvider client={client}>
      <App />
    </ApolloProvider>
  </React.StrictMode>
);

Now if we rebuild our images using docker compose build then run the containers using docker compose up -d we can access the frontend at http://localhost:5174 and fetch the data from the backend that will communicate with the cache and the database.

You may think that everything is great since our services can communicate with each other but we have a tiny issue here, all our services are inside the same default network exposed externally which means we can access the db, the cache, and the backend from the host machine which is something we don’t necessarily like, the idea is to have something like the image below:

Here we have two networks, public which is accessible from the host machine and private which is accessible only internally. Since the frontend will be accessed from the host and will communicate with the backend it should be inside the two networks.

Now let’s see how to do that in compose:

First, we start by adding a new top-level section named networks all the networks have to be declared there then we add two entries public and private which are the names of our networks. The private should have a parameter name internal set to true which will tell compose to not expose those services outside.

networks:
  public: 
  private: 
    internal: true

Once our networks are declared the last thing to do is to update the services definition by adding a network section:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      - "3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user
    networks:
      - private

  database: 
    image: mysql:latest
    environment: 
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: db
      MYSQL_USER: user
      MYSQL_PASSWORD: password
    networks:
      - private

  cache: 
    image: redis:latest
    ports:
    - "6380:6379"
    networks:
      - private


  frontend: 
    build: 
      context: ./client
      dockerfile: Dockerfile
    ports: 
    - "5173:5173"
    networks:
      - private 
      - public

networks:
  public: 
  private: 
    internal: true

We have added a network section for each service and only the frontend service has two networks now the services inside the private network can’t be accessed from the host. Note that the names private and public are not mandatory you can give any name you want the thing that makes the network private is the internal: true

Persistent volume

If we don’t want to lose our database and cache data each time the services are started we have to create volumes and assign them to the services.

The top-level volumes declaration lets you configure named volumes that can be reused across multiple services let’s do it :

volumes:
  db-data: 
  cache-data:

Then update the services definition:

cache: 
    image: redis:latest
    ports:
    - "6380:6379"
    networks:
      - private
    volumes:
      - cache-data:/data
database: 
    image: mysql:latest
    environment: 
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: db
      MYSQL_USER: user
      MYSQL_PASSWORD: password
    networks:
      - private
    volumes: 
      - db-data:/var/lib/mysql

Once the volumes are declared and assigned to the services we can restart the stack how many times we want the data won’t be lost.

Startup order

When we run the docker compose up command, from the logs printed we can see in which order the services are started

We can notice that the order is the following cache, database, frontend then backend but there are two issues here:

• First, the order won’t always be the same
• Second, the front will be available before the backend although it is the first that calls the second. To fix these issues compose provides an attribute nameddepends_on available for each service that will be used to control the startup and shutdown order.

The depends_on attribute takes a list of services for which the current service depends and for each service, we have to set a condition that can be service_started or service_healthy .

Let’s update the backend service then explain:

backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      - "3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user
    networks:
      - private
    depends_on:
      database:
        condition: service_started
      cache:
        condition: service_started

We have updated the backend service definition by adding the depends_on attribute. We have added two dependencies the database and the cache. Their condition is service_started that means compose will run the backend container only if the database and the cache have been already started.

Now let’s update the frontend service:

frontend: 
    build: 
      context: ./client
      dockerfile: Dockerfile
    ports: 
    - "5173:5173"
    networks:
      - private 
      - public
    depends_on:
      backend: 
        condition: service_healthy

Here the condition is service_healthy which is different than service_started the idea is to give the developers more control over when a dependency can be considered ready to receive requests cause having the backend started doesn’t always mean it’s ready to receive requests.

Now we have to define how compose will define if the backend service is healthy and to do that we are going to update the backend service definition by adding a new attribute named healthcheck :

backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      - "3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user
    networks:
      - private
    depends_on:
      database:
        condition: service_started
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD", "curl", "-f", "http://backend:3000/healthcheck"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
      start_interval: 5s

When a container has a healthcheck specified, it has a health status in addition to its normal status. This status is initially starting. Whenever a health check passes, it becomes healthy (whatever state it was previously in). After a certain number of consecutive failures, it becomes unhealthy. The value of the test attribute can be whatever you want here we are making an HTTP request since it’s a web service but for example, in the case of a database, you can replace it with a mysql ping command.

The final compose.yaml file should look like this:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      - "3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user
    networks:
      - private
    depends_on:
      database:
        condition: service_started
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD", "curl", "-f", "http://backend:3000/healthcheck"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
      start_interval: 5s

  database: 
    image: mysql:latest
    environment: 
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: db
      MYSQL_USER: user
      MYSQL_PASSWORD: password
    networks:
      - private

  cache: 
    image: redis:latest
    ports:
    - "6380:6379"
    networks:
      - private


  frontend: 
    build: 
      context: ./client
      dockerfile: Dockerfile
    ports: 
    - "5173:5173"
    networks:
      - private 
      - public
    depends_on:
      backend: 
        condition: service_started

networks:
  public: 
  private: 
    internal: true

Secrets

A secret is a blob of data, such as a password, SSH private key, SSL certificate, or another piece of data that should not be transmitted over a network or stored unencrypted in a Dockerfile or your application’s source code.

To use a secret in our compose stack we have to do the following actions:

First, declare the secret by using the secrets top-level element. The source of the secret is either file or external.

secrets:
  my_certificate: 
    file: ./basic-certificate.cert

Second, grant the access to the secret for the frontend service

    build: 
      context: ./client
      dockerfile: Dockerfile
    ports: 
    - "5173:5173"
    networks:
      - private 
      - public
    depends_on:
      backend: 
        condition: service_started
    secrets: 
      - my_certificate

The final compose file should look like this:

name: composeintroduction

services: 
  backend:
    build:
      context: ./graphql-server
      dockerfile: Dockerfile
      target: dev
    command: npm run start:dev
    ports:
      - "3000:3000"
    environment:
      DB_PASSWORD: password
      DB_USERNAME: user
    networks:
      - private
    depends_on:
      database:
        condition: service_started
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD", "curl", "-f", "http://backend:3000/healthcheck"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
      start_interval: 5s

  database: 
    image: mysql:latest
    environment: 
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: db
      MYSQL_USER: user
      MYSQL_PASSWORD: password
    networks:
      - private

  cache: 
    image: redis:latest
    ports:
    - "6380:6379"
    networks:
      - private
    volumes:
      - cache-data:/data


  frontend: 
    build: 
      context: ./client
      dockerfile: Dockerfile
    ports: 
    - "5173:5173"
    networks:
      - private 
      - public
    depends_on:
      backend: 
        condition: service_started
    secrets: 
      - my_certificate

secrets:
  my_certificate: 
    file: ./basic-certificate.cert

volumes:
  db-data: 
  cache-data:

networks:
  public: 
  private: 
    internal: true

Conclusion

We have come to the end of this article that aimed to show how to containerize a multi-container app using docker-compose. You can find the final version of the project here. Don’t hesitate to check the compose documentation there are a lot of fun things to discover like working with multiple compose files or docker-compose watch.

I hope you enjoy this article as much as I enjoyed writing it.

Feel free to reach out to me on LinkedIn or Twitter.