Creating a development dockerfile and docker-compose.yml for yarn 1.22 monorepos using Turborepo.

(Be sure to check out the example repo)

If you've ever worked in a yarn monorepo project with inter-dependent workspaces, you know that creating docker containers for development in a way that avoids unnecessary rebuilds can be a challenge.

Turborepo, "...an intelligent build system optimized for JavaScript and TypeScript codebases", provides tools that make this task much easier.

Let's assume a project that looks like something like this:

- /project-directory
    - /apps
        - /frontend
            - package.json
            - other files...
        - /backend
            - package.json
            - other files...
    - /packages
        - /shared-stuff
            - package.json
            - other files...
    - .dockerignore
    - package.json
    - turbo.json
    - Dockerfile.dev
    - docker-compose.yml

.dockerignore

**/node_modules
**/.next
**/dist

This ensures that node_modules and build artifacts are excluded when copying directories to the container from the host in our Dockerfile.

package.json

{
  "name": "turbo-docker-monorepo",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "private": true,
  "workspaces": ["apps/*", "packages/*"],
  "packageManager": "yarn@1.22.19",
  "devDependencies": {
    "turbo": "^1.9.6"
  }
}

This is the minimal package.json you'll need for a yarn monorepo that makes use of turborepo with workspaces in the 'apps' and 'packages' directories. Not much else to say!

turbo.json

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".next/**", "!.next/cache/**"]
    },
    "lint": {},
    "dev": {
      "cache": false,
      "persistent": true
    }
  }
}

This turbo.json file is taken from the basic turborepo example.

Let's assume that the backend app has the packages/shared-stuff workspace as a dependency (meaning the backend package.json has "shared-stuff": "*" in its list of dependencies).

We can use turborepo to build backend in a way that respects its dependence on the shared-stuff package with the command turbo run build --filter=backend, which will build (in order) the shared-stuff package and then backend app. Cool!

Dockerfile.dev

# syntax=docker/dockerfile:1.5.2
# based on: https://github.com/vercel/turbo/blob/main/examples/with-docker/apps/api/Dockerfile

FROM node:20.2-alpine3.17 as base

# adding apk deps to avoid node-gyp related errors and some other stuff. adds turborepo globally
RUN apk add -f --update --no-cache --virtual .gyp nano bash libc6-compat python3 make g++ \
      && yarn global add turbo \
      && apk del .gyp

#############################################
FROM base AS pruned
WORKDIR /app
ARG APP

COPY . .

# see https://turbo.build/repo/docs/reference/command-line-reference#turbo-prune---scopetarget
RUN turbo prune --scope=$APP --docker

#############################################
FROM base AS installer
WORKDIR /app
ARG APP

COPY --from=pruned /app/out/json/ .
COPY --from=pruned /app/out/yarn.lock /app/yarn.lock

# Forces the layer to recreate if the app's package.json changes
COPY apps/${APP}/package.json /app/apps/${APP}/package.json

# see https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

COPY --from=pruned /app/out/full/ .
COPY turbo.json turbo.json

# For example: `--filter=frontend^...` means all of frontend's dependencies will be built, but not the frontend app itself (which we don't need to do for dev environment)
RUN turbo run build --no-cache --filter=${APP}^...

# re-running yarn ensures that dependencies between workspaces are linked correctly
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

#############################################
FROM base AS runner
WORKDIR /app
ARG APP
ARG START_COMMAND=dev

COPY --from=installer /app .

CMD yarn workspace ${APP} ${START_COMMAND}

Let's go over what happens in each layer. For the following discussion, assume that the backend app service is being created, which has shared-stuff as a dependency.

base

FROM node:20.2-alpine3.17 as base

RUN apk add -f --update --no-cache --virtual .gyp nano bash libc6-compat python3 make g++ \
      && yarn global add turbo \
      && apk del .gyp

This layer adds container dependencies to ensure NPM modules that use node-gyp build correctly. It also adds turborepo globally. The remaining layers are built from 'base'

pruned

FROM base AS pruned
WORKDIR /app
ARG APP

COPY . .

RUN turbo prune --scope=$APP --docker

This layer copies over the project files and runs turbo prune for the service in question. The $APP argument will be either 'frontend' or 'backend' (assuming the later, for this example), which is set in docker-compose.yml (this will be covered last). Regarding the --docker flag:

With the docker flag, the prune command will generate folder called out with the following inside of it:

• A folder json with the pruned workspace's package.jsons
• A folder full with the pruned workspace's full source code, but only including the internal packages that are needed to build the target.
• A new pruned lockfile that only contains the pruned subset of the original root lockfile with the dependencies that are actually used by the packages in the pruned workspace.

Assuming we're building the backend service container, here is what the 'out' directory in the ‘pruned’ layer would look like at this point:

- /out
    - /full
       - /apps
           - /backend (all files)
       - /packages
           - /shared-stuff (all files)
       - .gitignore
       - package.json
       - turbo.json
    - /json
        - /apps
            - /backend
                - package.json
        - /packages
            - /shared-stuff
                - package.json
        - package.json
    - yarn.lock

Note that the 'frontend' app is not present and (according to the turborepo docs) its dependencies were excluded from yarn.lock, which is what we want!

installer

FROM base AS installer
WORKDIR /app
ARG APP

# COPY 1
COPY --from=pruned /app/out/json/ .
COPY --from=pruned /app/out/yarn.lock /app/yarn.lock
COPY apps/${APP}/package.json /app/apps/${APP}/package.json

# RUN 1
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

# COPY 2
COPY --from=pruned /app/out/full/ .
COPY turbo.json turbo.json

# RUN 2
RUN turbo run build --no-cache --filter=${APP}^...

# RUN 3
RUN \
      --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,sharing=locked \
      yarn --prefer-offline --frozen-lockfile

The COPY 1 command series begins by copying the json directory in out from the pruned layer, as well as the 'scoped' yarn.lock file. It also redundantly copies the app's package.json from the host, which I've found necessary for the container to recreate correctly when shared workspace dependencies change.

The first run command (RUN 1) runs yarn and uses a cache mount targeting the yarn cache directory, which will help speed up subsequent re-builds.

The next two statements (COPY 2) copy the contents of out/full from the pruned layer as well as turbo.json from the host.

Since this container will run in a dev environment, it means we can safely skip building the backend app. However, we do need to build packages/shared-stuff, which in the current example is a dependency of backend. This is done with the command in RUN 2: RUN turbo run build --no-cache --filter=${APP}^.... In this case, since we're building the backend service container, the filter flag will resolve to --filter=backend^.... What this means is that all workspace dependencies of backend, but not backend itself, will be built, which is what we want.

This is a huge time saver for certain frameworks (looking at you, Next.js).

The final RUN command (RUN 3) is just a copy of the first (RUN 1). This final yarn call ensures that dependencies between workspaces are linked correctly.

runner

FROM base AS runner
WORKDIR /app
ARG APP
ARG START_COMMAND=dev

COPY --from=installer /app .

CMD yarn workspace ${APP} ${START_COMMAND}

The final layer copies over the installer layer and runs the command to start the app (based on the START_COMMAND argument).

Now let's look at docker-compose.yml, specifically the config for the backend service (almost done!)

version: '3.8'

x-defaults:
  &defaults
  init: true
  tty: true
  networks:
    - my_monorepo_network

networks:
  my_monorepo_network:

services:
    backend:
        <<: *defaults
        ports:
          - "3333:3333"
        command: yarn workspace backend dev
        environment:
          - PORT=3333
        build:
          args:
            APP: backend
            START_COMMAND: dev
          context: .
          dockerfile: ./Dockerfile.dev
        volumes:
          - ./apps/backend:/app/apps/backend
          - /app/apps/backend/node_modules
    ...other services...

This is a pretty typical definition for a Node service. Here you can see the build args, which are used in Dockerfile.dev.

Changing dependencies

tl;dr: docker compose up -d -V --build <service>

When you change dependencies for your node services on the host, you need to rebuild the container for it to reflect those changes.

You don't need to use docker compose build --no-cache <service> for this. There are better methods which don't require completely busting cache and starting from scratch.

The complicating factor for node services, as commonly configured in docker compose projects, is anonymous volumes:

...

volumes:
   - ./apps/backend:/app/apps/backend
   - /app/apps/backend/node_modules # <-- this guy

...

In the context of yarn 1.22 monorepos, apps/backend/node_modules contains development dependencies that are unique to that workspace.

The point of the anonymous volume /app/apps/backend/node_modules is to make it so node_modules in the backend workspace within the container is excluded from the bind mount ./apps/backend:/app/apps/backend.

Because of that volume, simply running docker compose build <service> or docker compose up -d --build <service> won't work like you expect it to when development dependencies for backend change.

There are two ways I've found to reliably recreate containers when deps change, while dealing with anonymous volumes:

1 The -V, --renew-anon-volumes flag

docker compose up -d -V --build <service>

You don't need to stop or restart the service.

2 rm container and build

docker compose rm -f -s <service> && docker compose up -d --build <service>

Either of these methods will do the trick.

Personally, I add --force-recreate to the first command, making it docker compose up -d -V --force-recreate --build <service>, because I have PTSD from dealing with Node dependency Docker issues. I'm only partly kidding. The flag doesn't doesn't noticeably increase the build time, in any case.

Note that everything described above also applies when you're not working in a monorepo and your (single) node service has volumes defined like this in docker-compose.yml:

...

working_dir: /app
volumes:
   - .:/app
   - /app/node_modules

...

One more thing: I've noticed that sometimes Nginx services will occasionally lose the plot when related compose service containers are rebuilt while it’s running, when it’s in a reverse-proxy role and you're using docker-compose service names in the config. I would assume this has something to do with it caching the resolved ip addresses of services (relative to the docker subnet) for those service names. Restarting the proxy service will resolve the issue.

Using file watch instead of bind mounts (experimental)

Check the docs to see what this is all about.

I would not recommend using this for actual development work yet, since it's little wonky at the moment (it's experimental, after all), but for the curious this is how the backend service definition would be changed to use watch for syncing files, instead of bind mounts:

  backend:
    <<: *defaults
    ports:
      - "3333:3333"
    command: yarn workspace backend start:dev
    environment:
      - PORT=3333
      - IN_CONTAINER=1
    build:
      args:
        APP: backend
        START_COMMAND: start:dev
      context: .
      dockerfile: ./dockerfiles/Dockerfile.dev
    x-develop:
      watch:
        - action: sync
          path: ./apps/backend
          target: /app/apps/backend

Then, start your compose project like so:

docker compose up -d && docker compose alpha watch

Two observations:

1. When I tested this out in an existing project, I had to completely recreate the project before it would work. I even changed the project name for good measure to completely dissassociate it from the prior compose project (you can do this in docker-compose.yml with the top-level name prop or the -p flag), though I'm not sure going that far was strictly necessary.

2. I could not get the rebuild action to work. While it would start a rebuild when the targeted file changed, as advertised, it would get stuck at Recreating <service> for the other services (not the one that was rebuilt). I have only observed this happening with a single test project, however.