In this post I'm going to create a super-basic Nx-backed mono-repo app and dockerize it.
Mono-repos have a number of benefits:
- If you're working on features that span multiple parts of the system you can do one PR encompassing them all.
- It also makes integration testing even easier.
- It allows you to easily share code (interfaces) between front-end and back-end
- One (consistent) way to run the app locally
- Everything at the current commit works together
Nx is a set of Extensible Dev Tools for Monorepo development. It bakes in a lot of solid tools (Typescript, prettier, Cypress, Jest, React, Express, etc) with some handy CLI commands to make development easier. Read more about it here
After setting up a basic hello-world Nx app in docker I'm going to discuss how the same process could be used to migrate from a poly-repo-app to a mono-repo-app.
Table of Contents
- Scaffolding the Basic App
- Dockerizing
- Scaffolding the Migration
- The Actual Migration? (but not really)
- What'd we end up with?
The Basic App
Code for the Basic App is here:
Scaffolding
Scaffold the app using NX commands
npx create-nx-workspace@latest
npx: installed 184 in 14.703s
? Workspace name (e.g., org name) blog-nx-docker
? What to create in the new workspace react-express [a workspace with a full
stack application (React + Express)]
? Application name blog-nx-docker
? Default stylesheet format emotion [ https://emotion.sh]
Creating a sandbox with Nx...
... lots of output ...ls
cd blog-nx-docker
npm start and npm start api will run the api and the front-end... The Nx scaffold automatically includes a super basic api call that gets displayed on the page ("Welcome to api!").
But how would we get this up and running in Docker?
Dockerizing
We'll need a dockerignore file, an nginx.conf file, one multi-stage Dockerfile, and a docker-compose.yml.
Boring Files
Getting the easy ones out of the way... here's the nginx.conf
file.
worker_processes 1;
events {
worker_connections 1024;
}
http {
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream api {
server blog-nx-api:3333;
}
server {
listen 80;
server_name localhost 10.*;
root /usr/share/nginx/html;
index index.html index.htm;
include /etc/nginx/mime.types;
gzip on;
gzip_min_length 1000;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
location / {
try_files $uri $uri/ /index.html;
}
}
server {
listen 3333;
server_name localhost 10.*;
location / {
proxy_pass http://api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}
}
And the .dockerignore
file... this prevents unnecessary docker image builds by ignoring changes to files.
node_modules
coverage
docker
tools
Dockerfile*
README.md
LICENSE
.vscode
.dockerignore
.git
.gitignore
Multi-stage Dockerfile
Next the multi-stage Dockerfile
. This file will build itself in stages and be used by the docker-compose.yml
file.
FROM node:12 AS blog-nx-base
WORKDIR /app
COPY . .
RUN npm ci -S
RUN npm run build -S
RUN npm prune --production -S
FROM nginx:alpine AS blog-nx-ui
COPY nginx.conf /etc/nginx/nginx.conf
WORKDIR /usr/share/nginx/html
COPY --from=blog-nx-base /app/dist/apps/blog-nx-docker .
FROM node:12 AS blog-nx-api
EXPOSE 3333
WORKDIR /app
COPY --from=blog-nx-base /app/node_modules /app/node_modules
COPY --from=blog-nx-base /app/dist/apps/api .
CMD ["node", "main.js"]
This Dockerfile
pulls the node 12.x image from the docker registry and does the install and build for both the front-end and api. From there, the UI is copied into an nginx image and the built-backend is copied into a fresh node:12 image.
At this point you could run docker build commands to tag images, like:
docker build --target blog-nx-api -t blog-nx-api .
docker build --target blog-nx-ui -t blog-nx-ui .
...and use those in docker-compose... but docker-compose 3.4 supports using the multi-stage Dockerfile straight out.
Multi-stage docker-compose file
For the docker-compose.yml
file:
version: '3.4'
services:
blog-nx-api:
container_name: blog-nx-api
build:
context: .
target: blog-nx-api
networks:
martzcodes:
blog-nx-ui:
container_name: blog-nx-ui
build:
context: .
target: blog-nx-ui
ports:
- 80:80
- 3333:3333
networks:
martzcodes:
networks:
martzcodes:
The secret sauce here is the build.target... that refers to the AS
name in the Dockerfile. This wasn't supported in earlier versions of Docker, which is why you see a lot of projects still use multiple Dockerfiles.
So now if you run docker-compose up -d
it will build the images via the Dockerfile and then if you go to http://localhost/ you get the same satisfying template page with an api call.
Advanced
That's great for a basic hello-world app, but how do we handle a more advanced app that already exists in the real-world that is stored in several repos?
My strategy is to scaffold it out using basic Nx commands, merge the package.json dependencies and follow the same general process as before.
I'm starting from a place where I have only two repos with 3 components of the app (there are several others installed via docker-compose / images... not worth moving those for the moment). Mine looks like:
- Service Repo
- Express Service
- Database seeds
- UI Repo
- React App
Both repos have their own Dockerfile(s) and docker-compose.ymls... and they also have their own CI yaml files with integration tests that span both repos. Which you may have seen from my previous post:
Integration testing across multiple repos with Codefresh
Matt Martz ・ Mar 7 '20
As it stands I typically get ~3 Pull Requests for a feature that spans front-end -> back-end -> db... one for each.
Scaffolding the Migration
I'm going to start with an empty Nx workspace.
$ npx create-nx-workspace@latest
npx: installed 184 in 15.125s
? Workspace name (e.g., org name) team-name
? What to create in the new workspace empty [an empty workspace]
? CLI to power the Nx workspace Nx [Extensible CLI for JavaScript and TypeScript applications]
Creating a sandbox with Nx...
warning " > @nrwl/workspace@9.1.2" has incorrect peer dependency "prettier@^1.19.1".
[-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0/354
new team-name --preset="empty" --interactive=false --collection=@nrwl/workspace
✔ Packages installed successfully.
Successfully initialized git.
CREATE team-name/nx.json (268 bytes)
CREATE team-name/tsconfig.json (509 bytes)
CREATE team-name/README.md (2552 bytes)
CREATE team-name/.editorconfig (245 bytes)
CREATE team-name/.gitignore (503 bytes)
CREATE team-name/.prettierignore (74 bytes)
CREATE team-name/.prettierrc (26 bytes)
CREATE team-name/workspace.json (1059 bytes)
CREATE team-name/package.json (1108 bytes)
CREATE team-name/apps/.gitkeep (1 bytes)
CREATE team-name/libs/.gitkeep (0 bytes)
CREATE team-name/tools/tsconfig.tools.json (218 bytes)
CREATE team-name/tools/schematics/.gitkeep (0 bytes)
CREATE team-name/.vscode/extensions.json (109 bytes)
I need to supplement the empty workspace with npm i --save-dev @nrwl/express @nrwl/react
to get the express and react generators.
With those installed I can use some of the nrwl generators. Create the react app:
$ npm run nx g @nrwl/react:app team-name-ui
> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/react:app" "team-name-ui"
? Which stylesheet format would you like to use? emotion [ https://emotion.sh ]
? Would you like to add React Router to this application? Yes
✔ Packages installed successfully.
CREATE ...
and the express app:
$ npm run nx g @nrwl/express:app team-name-service
> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/express:app" "team-name-service"
? In which directory should the node application be generated?
CREATE ...
but now I want to add a UI -> Service interface library (so I don't have to define interfaces twice) and a UI component library...
$ npm run nx g @nrwl/workspace:lib team-name-interfaces
> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/workspace:lib" "team-name-interfaces"
CREATE ...
and
$ npm run nx g @nrwl/react:lib team-name-components
> team-name@0.0.0 nx /Users/mattmartz/Development/team-name
> nx "g" "@nrwl/react:lib" "team-name-components"
CREATE ...
Now that that's out of the way... it doesn't really do anything... especially with each other. Now let's migrate one piece at a time.
The Actual Migration? (but not really)
I can't share a repository for this section so instead I'm going to provide some high-level strategy and discuss a few things I ran into.
Docker-compose Migration
First I'm going to move the integrated docker-compose.yml file that lives in one of the apps. The goal is to be able to run docker-compose up -d
and have the original app up and running (pulling everything from docker registry images / building nothing locally).
I'm moving this first to ensure I don't have to troubleshoot it later.
One thing I quickly noticed is that my old docker-compose file was using version 2 instead of version 3.4... notably one of my services inside of it was using the old volumes_from
. That means I needed to define the shared data model outside of the services list and update the links...
version: '3.4'
services:
mysql:
image: mysql:5.6
container_name: mysql
ports:
- '3306:3306'
networks:
martzcodes:
environment:
- TZ=America/New_York
volumes:
- data:/var/lib/mysql # CHANGED
data:
image: ...
container_name: mysql-data
environment:
- TZ=America/New_York
networks:
- martzcodes
volumes:
- data:/var/lib/mysql # CHANGED
volumes: # ADDED
data:
Moving the UI, Backend and DB
First merge the package.json
dependencies... I tend to copy one into the other by section... and "Sort Lines Ascending" and then remove the duplicates... there may be a cleaner way but that works for me.
Then copy the rest of the files over. Nx uses an entry file named main.tsx
... you can either keep that name and replace the contents or rename it in the workspace.json
file.
Alternately you could create each file individually and re-build the app from the ground up... it depends how compatible your old version is with the new
Next would be setting up the Dockerfile. Since this will be the first thing going into the Dockerfile you could probably largely base it off of what was created in the Basic Example (unless you had a bunch of custom code in yours...). With the Dockerfile updated... change docker-compose.yml to build from it and use the correct target.
Moving the backend follows the same general principle... merge the dependencies and get it running locally.
Finally... the DB is the easiest because Nx doesn't really have any database hooks... basically just copy the files.
What'd we end up with?
If you didn't have a ton of troubleshooting to do, you probably ended up with a single repo that contains all of your files for an app. The benefits for this were described at the top of this post... which is probably why you got this far.
Top comments (3)
Hi Matt. really nice tutorial! Its really needed to copy the node_modules folder in the service folder? doing that you are adding dependencies that are not used by the API for example (angular, angular material, etc).
thanks! I haven't really thought about this in a while...
the strategy i used above was to have it running as a dev-server via Nx... you could also build the api and just copy the built stuff to the docker image and then run that directly (making sure it bundles everything it needs)... that'd probably be the better way to do it. good question!
Hi Martz! It's me again with bad news! :/ nestjs (nodejs at the end) needs the node_module folder because the build does not add all the dependencies, some libraries like typeorm cause problems if you don't include the node_modules.
I found this github.com/ZenSoftware/bundled-nest
"""
In many real-world scenarios (depending on what libraries are being used), you should not bundle Node.js applications (not only NestJS applications) with all dependencies (external packages located in the node_modules folder). Although this may make your docker images smaller (due to tree-shaking), somewhat reduce the memory consumption, slightly increase the bootstrap time (which is particularly useful in the serverless environments), it won't work in combination with many popular libraries commonly used in the ecosystem. For instance, if you try to build NestJS (or just express) application with MongoDB, you will see the following error in your console:
Error: Cannot find module './drivers/node-mongodb-native/connection'
at webpackEmptyContext
Why? Because mongoose depends on mongodb which depends on kerberos (C++) and node-gyp.
"""
So the workaround that I found was to move all the angular dependencies to the DevDependency list and copy the node_modules as you did in this tutorial.