How to make an infrastructure to easily deploy containers

There's a good chance you've heard about Containers. They are very popular these days, and for change it comes with great documentation.
There are good articles and tutorials covers the foundations, what are containers and their components (images, containers, Dockerfile, etc). There are also step-by-step tutorials that help you to start out with Docker.
When I thought how am I gonna use it, I looked for a workflow. I scratched my head and thought:

How can I automate the process of build and deploy of (almost) any app in a container?

In this post, I talk about the workflow I came up with for my use-case.
One that is automated, easy to modify, re-build and deploy on new or existing container instances.
I had a task to dockerize a third-party application, not a common one, that doesn't have an existing image on DockerHub. This is my first post experimenting with Docker.

I assume you are familiar with Docker concepts of Dockerfile, images and containers (if not, this is a great place to start) and Ansible.

• The Task
• The Flow
• Deploy
• The Automation
  • The builder
  • Deploy process
• Conclusions

The Task

I had a task to deploy a third party app that monitors our production network. It's a small distributed app, which works with master and slaves. I won't get into detail of the app (because this is out of scope for this article).
This was the first docker project I did, so I had to develop the infrastructure too.

What do I mean by develop an infrastructure for Docker?

I need to define the process first, then abstract it, so I can re-use as many parts as possible.
This would allow me to add new dockerized apps faster, build and deploy them with minimum additional code and more important - no duplicate code. Now is great time to define such process, because this is my first Docker project.

The Flow

Here is a decipt that define the process I want to implement.
As I said in the introduction, I'm not going to cover the basics in this post. However, I'll describe and explain each component in the process and some of it is considered basic.

• Dockerfile: "A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image." Once we have a Dockerfile defined, we use the docker command line to build it. The output of the execution is an image.

• Image: "An image is an executable package that includes everything needed to run an application--the code, a runtime, libraries, environment variables, and configuration files." It is the blueprint you can instantiate instances (containers) from. A good analogy to this is OOP - where you create a class, that allows you to create objects (instances).

• Build Server: Is a centralized server that executes the build process of Dockerfiles, composing images.

• Push: Once we have an image, pushing it to a registry makes it available for download on other machines. Think of it like Linux repository for apps, just for docker images.

• Registry: "The Registry is a stateless, highly scalable server side application that stores and lets you distribute Docker images." In my case, it resides on a different machine than the build server. But it can be the same one. It is just an app that runs inside a container :)

• Pull: You pull images on the machines you want to run the containers. If you use a private registry, you need to connect those clients to it.

• Container: "A container is a runtime instance of an image--what the image becomes in memory when executed (that is, an image with state, or a user process)."

For more info about concepts above, check the Docker Get-Started link.

Deploy

Once we have the image available locally, we can run an instance of it.

The syntax is: docker run [ OPTS ] <IMAGE:TAG> [ CONTAINER_NAME ]. There are many useful options, such as volumes and port mapping.

Before I describe the automation infrastructure and process, what are the advantages of using Containers anyway?
Here are the pros I wrote down for myself:

1. Docker runs on any platform with Docker engine installed. I don't care about dependencies and startup scripts. I totally eliminate that concern.
2. The deployment of an App, is very often similar to another App. Once I have an infrastructure to deploy containers, the time it would take me to support another App got shorten.
3. The speed of starting a Container is faster than a VM and it consumes less resources.
4. It gives the flexability to utilize the hardware more efficient.
5. Upgrading an App to a newer version is faster.
6. You have a rollback capability with no effort. Something broke? Just re-deploy from previous image.

The points made are from Ops point-of-view. There are more advantages (and disadvantages) when developing an App using containers.

The Automation

To create an automated process, we need to define such. Let's define concrete process, for building and deploying new images.

The builder

The role of the builder is implicit - to build images. Altough docker images can be built anywhere (on dev machine for example), I find it better to do it on a Build Server. Why?

• I like to have single entry point. I set up the environment once, define the requirements for the Job, standardize the naming and tagging of the images. All my images follow the same process, regardless of who builds them.
• If for some reason I need to change my build process, I need to update it on one place.
• I have a script that automatically tags the image, removing this task from the developer that builds it.

Jenkins is perfect for this task.

I created a new Job, which simply builds Dockerfile's and tags them automatically. I do that by using a Makefile to build the image and push it to our registry.

Why do I use a Makefile? Well, I made an abstract Makefile. This allows me to build ANY Dockerfile. I provide it with a name and tag for the image. It then builds, tags and pushes the new image to our registry in a single process.

Deploy process

This section defines how am I pulling an image and run a container on an instance.

We use Ansible to manage our infrastructure. It has a great module for Docker. I'm pretty sure any IaaC has Docker module that makes your life easier.
Deployment of a new image is made with a single task, that takes care for everything - pulling new images, spin up new containers and provide specific container options.

I create a new Ansible role, that prepare the Docker host. It generates the directories and copy relevant files. The responsibility of the role is to make the host ready to run the container instance. Note that the steps here are probably different depending on the App we want to deploy.

Since I let Ansible manage the configuration, and Docker to actually run the App, I have a short Ansible playbook that is clean and very easy to manage. Prior to that, I used to define much larger playbooks to prepare everything and deploy the App.

Using my new playbook, I can instantiate a container on any machine with docker engine in seconds. Ansible and Docker will get the job done.

Conclusions

That's the workflow I ended up with. I think it is simple, and convinient to dockerize new Apps now that I have the right tools and infrastructure.
Before that I had used Ansible playbooks and roles to build and deplpoy new apps. Most of the time it worked as expected, but sometimes things broke. It usually depends on the platform. Something that works on Ubuntu 14, not always works on Ubuntu 18 without modifications. Sometimes a driver is missing, or a package dependency. Or it is something else.
These pitfalls can't happen with Containers, since everything the app needs is inside the container.

To deploy a new App, these are my steps:

1. I need to define a Dockerfile
2. Validate it works
3. Once I have a ready image - use Jenkins to build an official image and push to the registry
4. I made a blueprint of the deploy process for Ansible - I update relevant parameters and tasks to run new images in a container
5. Play the ansible playbook on the new host where I want to run the container

If you haven't tried out using containers yet, I really encourage you to put the time and learn this technology. They are great. It is a great tool in your toolbox and it's easy to try it out.

Some great resources are this blog post, and this tutorial. Just try and play around with it.

Do you have a different workflow you foster? Feel free to share.

On another note, this is the first blog post I publish. Any feedback would be appreciated.