Over the last several weeks I have been working to get to know Docker at a much deeper level. Even after using it off and on for a year or so now, I still feel like I have much to learn. It is an ever changing landscape, but it's really a lot of fun to learn at a deeper level.
I had been struggling with understanding how to start up and application, and ensure that Postgres was running. I was experiencing this issue while working on some new tutorials for Codeship. Let's walk through the problem together, and how it manifests itself.
Docker has a couple methods to boot your containers in order, one of those being
depends_on. Typically - this is a great solution, but it also doesn't really tell you if the service is running, just that the container has started.
Docker explains it this way:
The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.
Dockerfile was fairly basic as well -
FROM node:7.7.2-alpine COPY package.json /tmp/package.json RUN cd /tmp && npm install --quiet RUN mkdir -p /usr/app && cp -a /tmp/node_modules /usr/app WORKDIR /usr/app COPY ./ /usr/app/
Nothing too crazy - just installing npm modules, copying the app files over, and then I let compose run the commands to start the app itself. In order to start my app, Postgres had to run a quick check to verify the table was set up and ready to roll.
Where I was running into trouble is that I would see occasional failures with the
pg module not able to connect to the client, typically when rebuilding the images, and during testing specifically. What I would notice is that the Postgres container was still getting itself ready to take connections, and the node app was already trying to start testing - leading to a race condition.
It's important to note, that using Codeship to deploy the app to Heroku,
depends_on isn't an option for me, so I'm using
links to connect everything. Since we aren't really guaranteed anything with
depends_on either, I'm ok with that, and it hasn't contributed to the issue, but the tests must work or I'll never get a deployment to run.
Let me preface this with a couple things:
- Docker can solve this multiple ways. I don't know them all, but this is a viable solution.
- I am always looking for new methods, so comments that are constructive are welcomed.
- Docker is always changing, so this may be out of date if you are reading this further out from the original post date.
I don't plan on getting exhaustive with the
entrypoint review here, however what I've been able to learn so far is that in combination with
CMD, it will essentially work as a wrapper script that can run a script after something has been fulfilled.
The idea here is that I can wait for
psql to respond, and then I can start my app. I'm still working through some of the details but the script looks like this:
#!/bin/sh # wait-for-postgres.sh set -e cmd="$@" while ! pg_isready -h "postgres" -p "5432" > /dev/null 2> /dev/null; do echo "Connecting to postgres Failed" sleep 1 done >&2 echo "Postgres is up - executing command" exec $cmd
Between documentation, and Stack Overflow, I was able to piece together a shell script to use the
pg_isready command to determine if the
postgres container is accepting connections or not. Once it is, then I can pass any command that will execute, and become
PID 1 because I'm using
exec $cmd to run
I did need to modify some things. I was originally using the
alpine image which is using Busy Box, a super lightweight Linux flavor. The difficulty I had was that I needed to install the postgres tooling to have the
pg_isready function available. I switched to the
slim image, and modified my
Dockerfile a bit.
FROM node:7.7.2-slim RUN apt-get update && apt-get install -f -y postgresql-client COPY package.json /tmp/package.json RUN cd /tmp && npm install --quiet RUN mkdir -p /usr/app && cp -a /tmp/node_modules /usr/app WORKDIR /usr/app COPY ./ /usr/app/ ENTRYPOINT ["./wait-for-postgres.sh"]
Three main changes:
- Changing to the
node:7.7.2-slimimage to have more Linux tools available.
RUN apt-get update && apt-get install -f -y postgresql-clientis run to install the tooling needed for the
wait-for-postgres.shscript to work.
ENTRYPOINT ["./wait-for-postgres.sh"]to run the shell script, and then whatever command I pass to it is run once the loop exits in the script.
Now - I have the ability to run
npm run dev or
npm test and know that postgres is ready to go.
I'm still working on things - but wanted to share this first. E.g. - test that the tables are also setup before executing the app, explore better retry logic in the node app itself, and probably more - but this works.
It's exciting to learn about everything Docker and how to deliver on Codeship. Like everything, there are ups and downs, so to speak. At times the debugging can be tedious, but it's a small price to pay for learning this.
Looking forward to your comments - here or @kellyjandrews on Twitter.