More than "Hello World" in Docker: Build Rails + Sidekiq web apps in Docker

This is the first post in the More than "Hello World" in Docker series. The series will help you ready your app: from setting it up locally to deploying it as a production-grade workload in AWS.

• Build Rails + Sidekiq web apps in Docker - we are here
• Deploy Rails in Amazon ECS
  • Concepts
  • Push an image to ECR
  • Create the RDS database, Task Definition, and Load Balancer
  • Create the ECS Cluster and connect it all together
  • Configure Sidekiq
• Automate Deploys with AWS CodeDeploy
  • Deep Dive on AWS CodeBuild
• Advanced ECS Concepts:
  • Service Discovery and Auto Scaling - coming soon
  • Fargate - coming soon

There is no shortage of web tutorials on how to display "Hello World" in Docker. The problem is that the path from getting your app to say "Hello World" in Docker, to running production-grade applications in AWS, is often not as straightforward. Even if your app isn't Rails / Ruby, reading through this series will help you know the challenges that lie ahead of you after you decide to use Docker in production.

I've used Docker in production for 7 months and have studied it for 4 months before that. In simple setups, doing Docker is very straightforward. Using it in production, however, increases the complexity as there would be a lot of other things to think about. This post will help you manage that complexity.

For this post, we would:

• Learn about basic Docker concepts and why Docker is used.
• Connect a Rails app to a PostgreSQL Server.
• Have a Sidekiq server running to get jobs from a Redis database.
• Feed simple tasks to the Sidekiq server by adding jobs in Redis.
• Run the Rails app using Docker compose

This is a foundational post. The Rails app we will build here will be the sample app that we will be using in later posts. If you're just starting in Docker, I recommend following along this post.

If you had some prior experience working with Docker, I recommend going to the next post in the series. The source code for this tutorial is available on GitHub, under release v0.0.1.

0 | Concepts

In this section, I will do a quick run-through on the basic concepts behind Docker. For a deeper conceptual understanding of Docker, I recommend Chris Noring's introduction.

In essence, Docker allows us to spawn containers on top of a Host OS (like Ubuntu, Mac, or Windows). We can run multiple containers inside a single Host OS.

Container - This is a process running on top of the Linux kernel. It is allocated its own resources: CPU, memory, and filesystem. By having its own resources, a container becomes isolated from other containers running inside your instance. When a container fails for whatever reason, it does not affect other containers/processes running inside the same Host OS.

Image - This is a file containing everything you need to execute your code: dependencies, binaries, source code, etc. They are built by executing commands in a script-like file called Dockerfile. Images are used to spawn multiple, identical containers. The containers are an "instantiation" of these images - and are the ones serving your customer traffic.

Dockerfile - This is a file containing instructions required to build an image. The commands you put in here should specify how you can put everything your application needs into an image.

If all this seems unclear, the hands-on in this post will illustrate the concepts in practice.

1 | Why Docker?

Lower Cost

Traditionally, you provision one instance and have just one application server running there. If you need to scale your application, you add more instances with one application server running on each. You don't usually scale by adding more application servers in one instance - because it's hard to manage. But this also results in most of your instances having unused capacity.

In Docker, you can run multiple containers in one instance. Since you can allocate an instance's resources to a container, you can add as many containers in the instance as long as there are enough resources on the instance. For example, if you have an instance with 8GB in memory and your containers only require 1GB each, you can have as much as 8 containers in that single instance.

Another famous problem in a production environment is memory bloat. Memory bloat happens when your application consumes a lot more memory than it's supposed to. The common fix is just to increase the size of the instance, which of course, increases cost.

In Ruby, the memory bloat problem is notorious in heavily multi-threaded applications such as Sidekiq. In Docker, you can set a hard memory allocation to a container. If the application consumes more than the hard allocation (say, 1GB), Docker kills the container and starts a new one. This prevents memory bloat and helps you use your resources more efficiently.

With this, you get to make the most out of your instance's resources. You don't have to provision as many instances as you would traditionally.

Standardization

The environment that your app runs in your local will be exactly the same as it is in production. No more "it works in my machine but not in prod."

Onboarding new developers also become easy: all you have to do is make them install Docker and have them run your Docker image in their local machine. No more 7-page document on how to install your app and all its dependencies in their machine.

Faster Deployments

Deployments pre-Docker look like this: AWS detects a spike in your traffic, it provisions a new instance, installs all the dependencies, tests your instance if it displays traffic, then the new instance takes in traffic. That process can take a few minutes because it loads an entire OS every time.

Deployments in Docker are much faster. Since containers are just processes, starting it is much lightweight that loading an entire OS.

Others

Still not convinced? You can read about the other benefits here.

2 | Installation

For this tutorial, I used Rails 5.2.3 and Ruby 2.5.1. You can install Ruby and Rails from this tutorial. Make sure to select Ruby 2.5.x and Rails 5.2.x in the dropdown, and choose the OS you'd be using.

We're also going to use Docker. If you're a Mac user like me, use this tutorial. For Windows users, you can use this.

3 | Creating a Rails app

(3.0) Go to a folder you want to make your Rails app on and execute:

rails new rails_docker_app
cd rails_docker_app
git init && git add .
git commit -m "Initialize Rails project"

Then, let's create the routes, controller and view necessary to display a message on the homepage.

(3.1) Open the config/routes.rb and make a route to the homepage. We will create the homepage in the next steps.

  root to: "home#index"

(3.2) Create the file home_controller.rb in the app/controllers folder. Put a basic message inside the variable @message. Since the variable name is prepended with the @ symbol, it is going to be accessible from the view.

class HomeController < ApplicationController
  def index
    @message = "static page"
  end
end

(3.3) Create the app/views/home folder. In that folder, create a file called index.html.erb. Notice that we referenced the @message variable we created in the previous step. This should display the message "static page".

Home Page:

<%= @message %>

(3.4) Now, start the rails server and you should see the simple homepage we just created.

rails server -p 3000

You should see a simple page like the one below:

(3.5) When you're satisfied, commit your progress

git add .
git commit -m "Add basic routes, views, controller"

4 | Setting up Docker for Rails

(4.1) Create the Dockerfile in the app root directory with the touch Dockerfile command.

# gets the docker image of ruby 2.5 and lets us build on top of that
FROM ruby:2.5.1-slim

# install rails dependencies
RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs libsqlite3-dev

# create a folder /myapp in the docker container and go into that folder
RUN mkdir /myapp
WORKDIR /myapp

# Copy the Gemfile and Gemfile.lock from app root directory into the /myapp/ folder in the docker container
COPY Gemfile /myapp/Gemfile
COPY Gemfile.lock /myapp/Gemfile.lock

# Run bundle install to install gems inside the gemfile
RUN bundle install

# Copy the whole app
COPY . /myapp

Essentially, the Dockerfile is just a bunch of human-readable shell commands put together in a file. The first line fetches a Docker image called "ruby:2.5.1-slim" from Docker Hub. This image is called the base image. It has a lightweight operating system called "slim", with ruby 2.5.1 pre-installed. We will build upon this base image with the next commands.

The next command installs dependencies for Rails.

Images have their own "folder structure" inside. With the next commands, we create the /myapp folder and make that the 'default' folder.

We copy the Gemfile and Gemfile.lock, and then run bundle install. This installs all the required gems inside the image. We then proceed with copying the entire application into the image.

(4.2) Now that we understand what it does, let's execute docker build . for Docker to go through the instructions and create a Docker image for us.

You should see Docker run the commands in your Dockerfile:

(4.3) Once the execution is finished, run docker images to see the docker image you just built:

You can see that it's a pretty huge file (~641MB). That's because it literally has everything it needs to run the application.

(4.4) From the docker images, look for the image_id of your newly created image. Mine looks like this: "2f1a0fabe8c6". Let's tag our image so it would have a pretty name we can remember easily.

# for me
docker tag 2f1a0fabe8c6 rails-docker-app:v0.0.1

# for you: replace --chash-- with your image_id
docker tag --chash-- rails-docker-app:v0.0.1

Running a docker images again, you'd see your image with the proper name.

(4.5) Now that we have a Docker image, let's run it.

docker run -it -p 3000:3000 rails-docker-app:v0.0.1 bundle exec rails server -b 0.0.0.0 -p 3000

The -it flags allow you to have a terminal where you can see logs from your Rails server. The -p 3000:3000 flag exposes the port 3000 of the container to the port 3000 of your local machine. Then, I specified the Docker image we just tagged earlier and the command to run the Rails server.

You should see the simple page again, this time running on top of Docker:

(4.6) If you're satisfied with the output, commit your progress.

git add .
git commit -m "Add Dockerfile and run via docker run"

5 | Make running Docker easy with docker-compose

We now have a functioning Ruby on Rails application on top of Docker 🎉 🎉 🎉 .

Usual production workloads aren't this simple though. We usually have other services we run in conjunction with the Rails app: databases, Redis, Sidekiq, etc. So a local setup would typically have 5 different terminal windows with 5 separate docker run commands. Now that's a lot to keep track. Docker thought of that and introduced docker-compose.

The service docker-compose allows you to declare all these services neatly in one YAML file. It also gives you the power to start all these services in one window.

Let's dive in:

(5.1) Create a docker folder in your app's root directory, and a development folder inside it: mkdir docker && mkdir docker/development. Then, create a file called docker-compose.yml and enter the following:

version: '3'
services:
  db:
    image: postgres
    volumes:
      - ../../tmp/db:/var/lib/postgresql/data
  web:
    build: ../../
    command: bundle exec rails s -p 3000 -b '0.0.0.0'
    volumes:
      - ../../:/myapp
    ports:
      - "3000:3000"
    depends_on:
      - db

This docker-compose folder defines 2 services: db and web. The "db" service is a PostgreSQL container that will serve as our database service. The "web" service is the application server for our Rails app.

The content inside the "web" service is what we would typically put inside the docker run command - but it's much easier to read this way! 📖

(5.2) Now, we use docker-compose build command to build the docker images necessary to run this setup. Then, we use docker-compose up to simultaneously run these services.

cd docker/development
docker-compose build
docker-compose up

You will see logs coming from these two services:

(5.3) If you're satisfied with the output, commit your progress.

git add .
git commit -m "Add docker-compose.yml for easier docker runs"

6 | Going Further: Integrating PostgreSQL

Now, most web tutorials stop at section 5 and call it a day. This post aims to help with your production needs as much as possible. Connecting a PostgreSQL database and integrating Sidekiq via Redis are two common production tasks that aren't given much coverage in blog posts around.

(6.1) It's time to create our models. Run the 2 commands below to generate the models and migrations for Author and Post.

rails generate model Author name:string kind:string
rails generate model Post title:string body:text author:references likes_count:integer

(6.2) Open your Gemfile and replace gem 'sqlite3' with the line below. We would be using PostgreSQL as our database, and delete references to SQLite3.

gem 'pg', '>= 0.18', '< 2.0'

Then do bundle install.

(6.3) To be able to connect to our database, replace the contents of the config/database.yml with the contents below (if the file isnt there, create it yourself).

development:
  adapter: postgresql
  encoding: unicode
  host: db
  username: postgres
  password:
  pool: 5
  database: sample_app_docker_development

Then, add seeds to the db/seeds.rb file so we can have sample data in the database.

author = Author.create(name: "Raphael Jambalos", kind: "Programmer")
post = Post.create(title: "Redis", body: "This is a in-memory database often used for caching.", author_id: author.id)
post = Post.create(title: "PostgreSQL", body: "This is a transactional database used for transactions", author_id: author.id)
post = Post.create(title: "DynamoDB", body: "This is a NoSQL database used for concurrent workloads.", author_id: author.id)

(6.4) We want to display the sample data to the view. To do that, let's change the contents of the app/controllers/home_controller.rb file to:

class HomeController < ApplicationController
  def index
    @message = "Dynamic"

    @posts = Post.all
  end
end

Then, let's change the contents of the app/views/home/index.html.erb so we can render the posts neatly to the view.

Home Page:

<%= @message %>

<% @posts.each do |post| %>
  <h1> <%= post.title %> </h1>
  <h2> <%= post.author.name %> </h2>
  <p> <%= post.body %>
<% end %>

(6.5) Now, let's run docker-compose again.

cd docker/development/
docker-compose build
docker-compose up

On a separate window, run these commands sequentially

docker-compose run web rake db:create
docker-compose run web rake db:migrate
docker-compose run web rake db:seed

(6.6) If you're satisfied with your progress, commit it:

git add .
git commit -m "Add models, migration, seeds; Modify views to show seeded data."

7 | Going Further: Integrating Sidekiq

(7.1) Add gem 'sidekiq' to the Gemfile and run bundle.

Sidekiq is used for background processing. Imagine a website with a form in it. After filling up all the information required in the form, you press Submit. The browser sends all the info you wrote to the web server. By default, the web server has 60 seconds to process the request; otherwise, it results in a request timeout. This type of processing is foreground processing.

Now, we wouldn't want to make our customers wait as the info gets processed (much less show them a timeout page). So we throw the job of processing the information to a background processor ("Sidekiq"). Once we assign the job to Sidekiq, the customer will see a page with the text "Please wait a few moments". Sidekiq does the request and in a few minutes, the customer can see the results on refresh.

(7.2) Run rails g sidekiq:worker IncrementCount to create a worker file. We go to the file Sidekiq just created app/workers/increment_count_worker.rb and replace its contents with the snippet below.

Essentially, we take a post and add 1 to its like_count property.

class IncrementCountWorker
  include Sidekiq::Worker

  def perform(post_id)
    post = Post.find(post_id)

    post.likes_count ||= 0
    post.likes_count += 1
    post.save
  end
end

(7.3) By default, the files in the app/workers/ folder aren't read by Rails. You have to go to the config/application.rb and insert this line inside the class Application block.

  config.autoload_paths += %W(#{config.root}/app/workers)

(7.4) To show the likes of each post and add a like button for each post, let's change the contents of the app/views/home/index.html.erb

Home Page:

<%= @message %>

<% @posts.each do |post| %>
  <h1> <%= post.title %> </h1>
  <h2> <%= post.author.name %> </h2>
  <p> <%= post.body %>

  <br>

  <p> 
    <%= link_to "Like", increment_async_path(post_id: post.id), method: :post %>

    Likes:
    <%= post.likes_count %> 
  </p>
<% end %>

(7.5) Then, we add a route in config/routes.rb and a controller method in app/controllers/home_controller.rb to accommodate this request.

  post "/increment_async", to: "home#increment_async"

The controller method calls our Sidekiq service and invokes the worker "IncrementCountWorker".

class HomeController < ApplicationController
  def index
    @message = "Dynamic"

    @posts = Post.all
  end

  def increment_async
    ::IncrementCountWorker.perform_async(params[:post_id])
    redirect_to root_path
  end
end

(7.6) Our new setup requires a Sidekiq service and a Redis service.

Redis is an in-memory database. Every time a customer presses the button we made in 7.4, an entry in the Redis database is added. The Sidekiq service polls the Redis service for new entries in its database. If there is one, it executes it.

version: '3'
services:
  db:
    image: postgres
    volumes:
      - ../../tmp/db:/var/lib/postgresql/data
  web:
    build: ../../
    command: bundle exec rails s -p 3000 -b '0.0.0.0'
    volumes:
      - ../../:/myapp
    environment:
      RAILS_ENV: "development"
      REDIS_URL: "redis://redis:6379/12"
    ports:
      - "3000:3000"
    depends_on:
      - db
  redis:
    image: redis
    volumes:
      - ../../tmp/db:/var/lib/redis/data
  sidekiq:
    build: ../../
    command: 'bundle exec sidekiq'
    volumes:
      - ../../:/myapp
    environment:
      RAILS_ENV: "development"
      REDIS_URL: "redis://redis:6379/12"
    depends_on:
      - redis

(7.7) Now, let's run docker-compose again.

cd docker/development/
docker-compose build
docker-compose up

Going back to localhost:3000 on your browser, you should be able to see this:

Pressing on the link button, you would be able to increment the like count of each post. When Rails receives the request, it puts a task on Redis and Sidekiq fetches it, adds '1' to the specified post and saves it.

8 | What's next?

Now, you have a fully-fledged Rails 5 application running on Docker. It ain't much, but having Sidekiq and PostgreSQL is a good foundation for you to build on top of. You can even use this template to migrate existing Rails applications to Docker.

After developing your Rails application, the next step is deploying it with a cloud provider so your customers can see your app. For the next post, we will deploy this project to AWS Elastic Container Service. Stay tuned!

Acknowledgments

Special thanks to Allen, my editor, for helping this post become more coherent.

Credits to Patrick Brinksma for the cover photo.

I'm happy to take your comments/feedback on this post. Just comment below, or message me!