CircleCI + AWS How to create CI/CD pipeline from scratch Part 1 - Build and push images

Prepare environment for containers

Create an IAM user

Right after the creation of an AWS account, you are logging in under the root user. You have full access to every service and the billing management console. To secure interaction with AWS it is a good practice to create a new user inside the group that has only required permissions.

A few words about managing permissions. There are two main ways to add them to users through groups and roles. The main difference is in that groups are a collection of users with the same policies. Roles, in turn, can be used to delegate access not only to users but also to other services and applications, we will use both. Let's create them.

On the second step select the next policies

• AmazonEC2ContainerRegistryFullAccess
• AWSCodeDeployRoleForECS
• AmazonEC2ContainerServiceFullAccess
• AmazonECSTaskExecutionRolePolicy

Then create the role that we will give to ECS to deploy our containers to EC2 instances.

and on the second step select in policies AmazonEC2ContainerServiceforEC2Role

More about it is showed there

And finally, let's add a new user and add to the previously created group:

This user should have only programmatic access because you will use it only from the circleci. After it, generate access keys and save them, they will need you later.

Create ECR

ECR a place where you will store containers from they will be deployed. Just go to the ECR and click on the "Create repository" button. You will see the window where you should select the name for the repository. Other
settings use by default

Great! You have a repository and all required credentials to build and push images. Time to automatize it.

Configuring Circle CI

The main idea is to run tests after each commit for all branches and deploy after changes in the development and master.

Before you start to configure the pipeline, you will need to prepare the application
following this fantastic getting started page.

Tests

The most popular use case of the Circle CI that I've seen is running tests (not all developers trust to external
services to deploy applications). To run them you should define a job
and add it as a step to a workflow. There is an example of test workflow
for the simple_plug_server application:

version: 2.1
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test:
    jobs:
      - test

It has only the one workflow test with the one job test. This job has three parts:

• docker - where is defined as a container inside which you will deploy test environment and run tests
• working_directory - the name of the folder where everything is happening
• steps - the set of commands where you download code, setup dependencies and finally run tests. You can also cache dependencies on this step.

We can also improve the representing of the failed tests, for it you should add a JUnit formatter for test results
(for elixir it is the hex package JUnitFormatter) and specify
the directory containing subdirectories of JUnit XML or Cucumber JSON test metadata files.
More information about it and how to add support for other languages and test frameworks look here.

Build and push containers

On the previous steps we created the ECR repository and the user that can push images, time to setup CircleCI config.

For work with images we will use the official orb for ECR circleci/aws-ecr@6.9.1
It significantly simplifies building and pushing images, let's add the new step to our config file:

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@6.9.1
  aws-ecs: circleci/aws-ecs@1.2.0
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test-and-build:
    jobs:
      - test
      - aws-ecr/build-and-push-image:
          repo: "simple_plug_server"
          tag: "${CIRCLE_BRANCH}_${CIRCLE_SHA1},${CIRCLE_BRANCH}_latest"
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - development

Briefly about the steps of this job:

• repo - the name of the repository (last part of the 815991645042.dkr.ecr.us-west-2.amazonaws.com/simple_plug_server)
• tag - tags that we apply to the built container, for the master branch it will add two tags: master_02dacfb07f7c09107e2d8da9955461f025f7f443 and master_latest
• requires - there you should describe the previous necessary steps, in this example we build an image only if all tests pass
• filters - describe for which branches this job should execute. There are a lot of other filters that you can use to customize a workflow

But before you start to run this workflow you should add the next environment variables:

• AWS_ACCESS_KEY_ID - access key for circleci that you obtained on this step
• AWS_SECRET_ACCESS_KEY - secret key for circleci that you obtained on this step
• AWS_REGION - region where placed your ECR instance
• AWS_ECR_ACCOUNT_URL - url of the ECR(looks like 815991645042.dkr.ecr.us-west-2.amazonaws.com)

After successful build of the development and master branches you will see something like there:

Great! You automatized process of running tests and building images in the next chapter you will see how to
setup servers on the AWS infrastructure and redeploy them after successfully passed tests.