Deploying containerized NginX to Heroku - how hard can it be?

I'm currently taking a Docker MOOC course. In part 3 of the course, there is an exercise about deploying a dockerized software to Heroku with CI/CD pipeline. The software can be anything, so I decided to deploy the course materials, and use Github Actions to make the deployment. Github Actions was very straightforward to use in my opinion, and I had no problems with that. Instead, I ran into some issues to get the materials available in Heroku. In this blog post, I thought I'd share the challenges I faced and tell how I solved them.

Course materials and how they're built

So the course material is a simple website built with Jekyll. The website was containerized, so it already had a Dockerfile in the course material repository. I forked the repository to be able to set up a CI/CD pipeline for it, and to deploy it to Heroku.
The Dockerfile that already existed in the course repository looks like this:

FROM jekyll/jekyll:3.8.3 as build-stage

WORKDIR /tmp

COPY Gemfile* ./

RUN bundle install

WORKDIR /usr/src/app

COPY . .

RUN chown -R jekyll .

RUN jekyll build

FROM nginx:alpine

COPY --from=build-stage /usr/src/app/_site/ /usr/share/nginx/html

So from the Dockerfile you can see that it uses a multi-stage build: in the first phase it uses Jekyll image, and in the second phase NginX. The repository doesn't contain any configuration for NginX, so it just uses the default configuration file.

The problem with NginX and Heroku

NginX is a widely used web server that can also act for example as a load balancer or reverse proxy.
As a reverse proxy, it takes a request coming from a client and forwards it to a server. This way the client doesn't communicate with the server itself, because the proxy is internet-facing, not the server. It can also act as a load balancer and send requests from clients evenly to different servers if there's multiple.
In this case, NginX is used as a basic web server, serving the static pages produced with jekyll build command.

Heroku is a PaaS (platform as a service) where you can deploy your software and host it in the cloud. It supports multiple programming languages, and also docker containers.

By default, NginX listens to the port 80. Now when I build the docker image locally on my laptop, and then run it with the command docker run -p 8080:80 docker-course-material I can open localhost:8080 in browser, and it will display the course materials. Now when I first deployed the materials to Heroku, the deployment to Heroku was successful, but opening the website showed a message that something went wrong. I noticed an error in the logs:

State changed from starting to crashed
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

After googling a while, I found the information that Heroku assigns a random port for the web apps when the application is deployed. It says in the Heroku documentation that

Each web process simply binds to a port, and listens for requests coming in on that port. The port to bind to is assigned by Heroku as the PORT environment variable.

This means that I would need to make NginX listen to the random port that Heroku assigns for the application. How can I accomplish that?

First version of the solution

Let me be clear, although I have worked as a software developer before I went to data engineering, I had no experience in NginX whatsoever. I however realised I had to create a configuration file for it to make it listen to another port. My first goal was to make it listen to some other port, such as 8080 locally so that I could test the configuration.

I read that NginX uses configuration file in path /etc/nginx/conf.d/default.conf. My first solution was to create a configuration file that would replace the default file parent image had added, and that in the file a different port was defined. I struggled with the configuration a bit, but the first working version of the configuration file with hardcoded port looks like this:

server {
  listen 0.0.0.0:8080;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }   
}

The line with listen 0.0.0.0:8080; tells the NginX to listen localhost and port 8080. After copying the configuration file in Dockerfile, I was able to bind to the port 8080 and access the materials. This didn't solve the problem with Heroku yet, though.

Binding PORT environment variable

So now that I had succeeded in changing to port to 8080, I would still need to figure out how to define the port dynamically when the Heroku assigns it. I couldn't then just hardcode the port number, but instead I needed to use some placeholder and replace it when the container is run.

So I replaced the hardcoded 8080 in the configuration with $PORT and added a line to Dockerfile that would replace it with the environment variable. Because it needed to be done at runtime, the only option was to use CMD. I couldn't use RUN command in the file, because the environment variable PORT set by Heroku is available only when the container is started. RUN command is executed once at build time.

For replacing the placeholder with the value of environment variable I used sed s command. Sed is a simple stream editor and can be used to transform text. sed s is probably to most commonly known sed command, and it can be used to replace text with some other using regular expression.
The example of sed s command
sed -i 's/cat/dog/g input-file.txt'
replaces all occurrences of a string cat with string dog in the file input-file.txt. Option -i means that the substitution is done in-place.

Final version

So in the end, this is how the configuration file looks like:

server {
  listen 0.0.0.0:$PORT;

  location / {
    root /usr/share/nginx/html;
    index index.html;
  }   
}

and this is how the Dockerfile looks like:

FROM jekyll/jekyll:3.8.3 as build-stage

ARG PORT

WORKDIR /tmp

COPY Gemfile* ./

RUN bundle install

RUN echo $PORT

WORKDIR /usr/src/app

COPY . .

RUN chown -R jekyll .

RUN jekyll build

FROM nginx:alpine

COPY --from=build-stage /usr/src/app/_site/ /usr/share/nginx/html

COPY nginx.conf /etc/nginx/conf.d/default.conf

CMD sed -i -e 's/$PORT/'"$PORT"'/g' /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'

Dockerfile has only two new lines: copying nginx configuration file, and replacing the placeholder with value of $PORT. The final command nginx -g 'daemon off' is needed when the Docker is used, so that the NginX stays on the foreground and the container is not stopped immediately.

The final thing that I need to mention is that the problem I encountered is not, in fact, NginX-specific, but rather Heroku-specific. This problem can occur with any web server and Heroku, and needs to be solved by binding on port that Heroku assigns. I just happened to encounter this problem for the first time with NingX.

The solution is pretty simple, but I managed to use a couple of hours with this problem. If you do encounter the issue, I hope this blog post helps!