DEV Community

Cover image for CI/CD Pipeline for Amazon ECS/FARGATE with Terraform
erozedguy
erozedguy

Posted on • Updated on

CI/CD Pipeline for Amazon ECS/FARGATE with Terraform

#terraform #aws #devops #cloud

DESCRIPTION

In this post I am going to explain how to build the infrastructure on AWS with Terraform to implement a CI / CD pipeline for ECS / Fargate.
The Architecture consists of a VPC with 2 public subnets in different Availability Zones. The desired tasks are 2 and each task is deployed on each public subnet with Fargate and each task belongs to the same ECS service.
An Application Load Balancer is used to balance the load between the two tasks.
In this case, the main goal is to implement a docker container that contains a simple HTTP server built with GOLANG. This HTTP server allows you to obtain the private IP of each task.
If I push new changes to the CodeCommit repository, CodePipeline detects those changes, triggers the pipeline and creates a new docker image and then deploys it to the ECS service to update the tasks.

ARCHITECTURE

aws cicd pipeline

RESOURCES

https://github.com/erozedguy/CICD-Pipeline-for-Amazon-ECS-Fargate

STEPS

STEP 01 - Create a IAM Role and CodeCommit Credentials

  • Create a Service Role for Elastic Container Service Task (
    Allows ECS tasks to call AWS services on your behalf.)
    taskRole

  • Attach AWSCodeCommitPowerUser Policy to my USER
    commitpolicy

  • Generate HTTPS Git credentials for AWS CodeCommit to clone, push, pull to the CodeCommit Repository

credentials

STEP 02: Terraform scripts to build the infrastructure

PROVIDERS 

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.51"
    }
  }
}
provider "aws" {
  profile = "default"
  region  = "us-east-1"
}
Enter fullscreen mode Exit fullscreen mode

VPC

The vpc script has VPC, SUBNETS and INTERNET GATEWAY resources.

resource "aws_vpc" "ecs-vpc" {
  cidr_block = "${var.cidr}"

  tags = {
    Name = "ecs-vpc"
  }
}

# PUBLIC SUBNETS
resource "aws_subnet" "pub-subnets" {
  count                   = length(var.azs)
  vpc_id                  = "${aws_vpc.ecs-vpc.id}"
  availability_zone       = "${var.azs[count.index]}"
  cidr_block              = "${var.subnets-ip[count.index]}"
  map_public_ip_on_launch = true

  tags = {
    Name = "pub-subnets"
  }
}

# INTERNET GATEWAY
resource "aws_internet_gateway" "i-gateway" {
  vpc_id = "${aws_vpc.ecs-vpc.id}"

  tags = {
    Name = "ecs-igtw"
  }
}
Enter fullscreen mode Exit fullscreen mode

VARIABLES TO VPC 

variable "cidr" {
  type    = string
  default = "145.0.0.0/16"
}

variable "azs" {
  type = list(string)
  default = [
    "us-east-1a",
    "us-east-1b"
  ]
}

variable "subnets-ip" {
  type = list(string)
  default = [
    "145.0.1.0/24",
    "145.0.2.0/24"
  ]

}
Enter fullscreen mode Exit fullscreen mode

IAM ROLES & POLICIES

For the CodeBuild is necessary to create a IAM Role&Policy to allow access to ECR to push and pull the Docker images in the ECR repository. Also, is necessary a permission to access a S3 bucket to store the artifacts.

resource "aws_iam_role" "codebuild-role" {
  name = "codebuild-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "codebuild.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy" "codebuild-policy" {
  role = "${aws_iam_role.codebuild-role.name}"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["codecommit:GitPull"]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:CompleteLayerUpload",
          "ecr:GetAuthorizationToken",
          "ecr:InitiateLayerUpload",
          "ecr:PutImage",
        "ecr:UploadLayerPart"]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
        "logs:PutLogEvents"]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:GetObjectVersion",
          "s3:GetBucketAcl",
        "s3:GetBucketLocation"]
        Effect   = "Allow"
        Resource = "*"
      }
    ]


  })

}
Enter fullscreen mode Exit fullscreen mode

ROUTE TABLES

A single Route Table for both Public Subnets

# TABLE FOR PUBLIC SUBNETS
resource "aws_route_table" "pub-table" {
  vpc_id = "${aws_vpc.ecs-vpc.id}"
}

resource "aws_route" "pub-route" {
  route_table_id         = "${aws_route_table.pub-table.id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = "${aws_internet_gateway.i-gateway.id}"
}

resource "aws_route_table_association" "as-pub" {
  count          = length(var.azs)
  route_table_id = "${aws_route_table.pub-table.id}"
  subnet_id      = "${aws_subnet.pub-subnets[count.index].id}"
}
Enter fullscreen mode Exit fullscreen mode

SECURITY GROUPS

The first Sec-Group is for the ECS Service

resource "aws_security_group" "sg1" {
  name        = "golang-server"
  description = "Port 5000"
  vpc_id      = aws_vpc.ecs-vpc.id

  ingress {
    description      = "Allow Port 5000"
    from_port        = 5000
    to_port          = 5000
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  egress {
    description = "Allow all ip and ports outboun"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
Enter fullscreen mode Exit fullscreen mode

The second Sec-Group is for the Application Load Balancer

resource "aws_security_group" "sg2" {
  name        = "golang-server-alb"
  description = "Port 80"
  vpc_id      = aws_vpc.ecs-vpc.id

  ingress {
    description      = "Allow Port 80"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  egress {
    description = "Allow all ip and ports outboun"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
Enter fullscreen mode Exit fullscreen mode

APPLICATION LOAD BALANCER 

resource "aws_lb" "app-lb" {
  name               = "app-lb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.sg2.id]
  subnets            = ["${aws_subnet.pub-subnets[0].id}", "${aws_subnet.pub-subnets[1].id}"]

}
Enter fullscreen mode Exit fullscreen mode

Port #5000 is used in the Target Group because that port is used for the container

resource "aws_lb_target_group" "tg-group" {
  name        = "tg-group"
  port        = "5000"
  protocol    = "HTTP"
  vpc_id      = "${aws_vpc.ecs-vpc.id}"
  target_type = "ip"

}
Enter fullscreen mode Exit fullscreen mode

Port #80 is used for the Listener

resource "aws_lb_listener" "lb-listener" {
  load_balancer_arn = "${aws_lb.app-lb.arn}"
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = "${aws_lb_target_group.tg-group.arn}"
  }
}
Enter fullscreen mode Exit fullscreen mode

ECS & ECR

ECR repository 
resource "aws_ecr_repository" "ecr-repo" {
  name = "ecr-repo"
}
Enter fullscreen mode Exit fullscreen mode
ECS Cluster 
resource "aws_ecs_cluster" "ecs-cluster" {
  name = "clusterDev"
}
Enter fullscreen mode Exit fullscreen mode
Task Definition
  • In this part is important to specify the containerPort
  • Create a ENV VAR: export TF_VAR_uri_repo = <ID_ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com/<ECR_REPOSITORY_NAME> 
resource "aws_ecs_task_definition" "task" {
  family                   = "HTTPserver"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 256
  memory                   = 512
  execution_role_arn       = data.aws_iam_role.ecs-task.arn

  container_definitions = jsonencode([
    {
      name   = "golang-container"
      image  = "${var.uri_repo}:latest" #URI
      cpu    = 256
      memory = 512
      portMappings = [
        {
          containerPort = 5000
        }
      ]
    }
  ])
}
Enter fullscreen mode Exit fullscreen mode
ECS Service

Specify the load balancer block

resource "aws_ecs_service" "svc" {
  name            = "golang-Service"
  cluster         = "${aws_ecs_cluster.ecs-cluster.id}"
  task_definition = "${aws_ecs_task_definition.task.id}"
  desired_count   = 2
  launch_type     = "FARGATE"


  network_configuration {
    subnets          = ["${aws_subnet.pub-subnets[0].id}", "${aws_subnet.pub-subnets[1].id}"]
    security_groups  = ["${aws_security_group.sg1.id}"]
    assign_public_ip = true
  }

  load_balancer {
    target_group_arn = "${aws_lb_target_group.tg-group.arn}"
    container_name   = "golang-container"
    container_port   = "5000"
  }
}
Enter fullscreen mode Exit fullscreen mode

CI/CD PIPELINE

CodeCommit Repository 
resource "aws_codecommit_repository" "repo" {
  repository_name = var.repo_name
}
Enter fullscreen mode Exit fullscreen mode
CodeBuild Project 
resource "aws_codebuild_project" "repo-project" {
  name         = "${var.build_project}"
  service_role = "${aws_iam_role.codebuild-role.arn}"

  artifacts {
    type = "NO_ARTIFACTS"
  }

  source {
    type     = "CODECOMMIT"
    location = "${aws_codecommit_repository.repo.clone_url_http}"
  }

  environment {
    compute_type    = "BUILD_GENERAL1_SMALL"
    image           = "aws/codebuild/standard:5.0"
    type            = "LINUX_CONTAINER"
    privileged_mode = true
  }
}
Enter fullscreen mode Exit fullscreen mode
buildspec.yml
  • This file is very important to create the Docker Image and Pull it to ECR repository
  • To update the ECS service is important to specify the containerName and imageUri in a JSON file with the name imagedefinitions.json. This file is an artifact
  • This file must be in the CodeCommit repository 
version: 0.2

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - echo $AWS_DEFAULT_REGION
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin 940401905947.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - REPOSITORY_NAME="ecr-repo"      
      - REPOSITORY_URI=940401905947.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$REPOSITORY_NAME
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Building the Docker image...
      - docker build -t $REPOSITORY_NAME:latest .
      - docker tag $REPOSITORY_NAME:latest $REPOSITORY_URI:latest
      - docker tag $REPOSITORY_NAME:latest $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - printf '[{"name":"golang-container","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json

artifacts:
  files: imagedefinitions.json
Enter fullscreen mode Exit fullscreen mode
S3 Bucket to store the artifacts 
resource "aws_s3_bucket" "bucket-artifact" {
  bucket = "eroz-artifactory-bucket"
  acl    = "private"
}
Enter fullscreen mode Exit fullscreen mode
CodePipeline

Specify Source, Build, Deploy Stages

NOTE: to code the stages check the official documentation
https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference.html

resource "aws_codepipeline" "pipeline" {
  name     = "pipeline"
  role_arn = "${data.aws_iam_role.pipeline_role.arn}"

  artifact_store {
    location = "${aws_s3_bucket.bucket-artifact.bucket}"
    type     = "S3"
  }
  # SOURCE
  stage {
    name = "Source"
    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeCommit"
      version          = "1"
      output_artifacts = ["source_output"]

      configuration = {
        RepositoryName = "${var.repo_name}"
        BranchName     = "${var.branch_name}"
      }
    }
  }
  # BUILD
  stage {
    name = "Build"
    action {
      name             = "Build"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      version          = "1"
      input_artifacts  = ["source_output"]
      output_artifacts = ["build_output"]

      configuration = {
        ProjectName = "${var.build_project}"
      }
    }
  }
  # DEPLOY
  stage {
    name = "Deploy"
    action {
      name            = "Deploy"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "ECS"
      version         = "1"
      input_artifacts = ["build_output"]

      configuration = {
        ClusterName = "clusterDev"
        ServiceName = "golang-Service"
        FileName    = "imagedefinitions.json"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

DATA

This section is for using created IAM roles

data "aws_iam_role" "pipeline_role" {
  name = "codepipeline-role"
}

data "aws_iam_role" "ecs-task" {
  name = "ecsTaskExecutionRole"
}
Enter fullscreen mode Exit fullscreen mode

OUTPUTS

To get the ALB DNS and the CodeCommit Repository URL

output "repo_url" {
  value = aws_codecommit_repository.repo.clone_url_http
}

output "alb_dns" {
  value = aws_lb.app-lb.dns_name
}
Enter fullscreen mode Exit fullscreen mode

EXTRA VARIABLES 

variable "repo_name" {
  type    = string
  default = "dev-repo"
}

variable "branch_name" {
  type    = string
  default = "master"
}

variable "build_project" {
  type    = string
  default = "dev-build-repo"
}

variable "uri_repo" {
  type = string
  #The URI_REPO value is in a TF_VAR in my PC
}
Enter fullscreen mode Exit fullscreen mode

STEP 03: HTTP Simple Server with GOLANG

This code is useful to get the PRIVATE IP of the ecs tasks

package main

import (
    "fmt"
    "log"
    "net"
    "net/http"
)

func main() {
    log.Print("HTTPserver: Enter main()")
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe("0.0.0.0:5000", nil))
}

// printing request headers/params
func handler(w http.ResponseWriter, r *http.Request) {

    log.Print("request from address: %q\n", r.RemoteAddr)
    fmt.Fprintf(w, "%s %s %s\n", r.Method, r.URL, r.Proto)

    fmt.Fprintf(w, "Host = %q\n", r.Host)
    fmt.Fprintf(w, "RemoteAddr = %q\n", r.RemoteAddr)
    if err := r.ParseForm(); err != nil {
        log.Print(err)
    }
    for k, v := range r.Form {
        fmt.Fprintf(w, "Form[%q] = %q\n", k, v)
    }
    fmt.Fprintf(w, "\n===> local IP: %q\n\n", GetOutboundIP())
}

func GetOutboundIP() net.IP {
    conn, err := net.Dial("udp", "8.8.8.8:80")
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    localAddr := conn.LocalAddr().(*net.UDPAddr)

    return localAddr.IP
}
Enter fullscreen mode Exit fullscreen mode

STEP 04: Dockerfile

  • This Dockerfile create a image with a HTTP Server with GOLANG
  • This file must be in the CodeCommit repository 
FROM golang:alpine AS builder

ENV GO111MODULE=on \
    CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=amd64
WORKDIR /build
COPY ./HTTPserver.go .

# Build the application
RUN go build -o HTTPserver ./HTTPserver.go

WORKDIR /dist
RUN cp /build/HTTPserver .

# Build a small image
FROM scratch
COPY --from=builder /dist/HTTPserver /
EXPOSE 5000
ENTRYPOINT ["/HTTPserver"]
Enter fullscreen mode Exit fullscreen mode

STEP 05: Create TF_VAR

uri_repo

STEP 06: Create the Infrastructure

Commands

  • terraform init
  • terraform validate
  • terraform plan
  • terraform apply -auto-approve

When the creation is finished we get the OUTPUTS
outputs

STEP 07: Upload Dockerfile, Code and buildspect files to the CodeCommit repository

  • Clone the repository
    Alt Text

  • Copy buildspect.yml, Dockerfile and Golang Code to the cloned repository folder and then do a commit
    git

  • Git push to the CodeCommit Repository
    git push

STEP 08: Check the Pipeline

pipeline

  • When the "Build" stage is done, check the docker image in the ECR repository build

repo

STEP 09: Check the ECS Service

When the "Deploy" stage is done, check the Tasks in the ECS Service

deploy

service

tasks

STEP 10: Check the Target Group

target_group

STEP 11: Check the operation of the Application Load Balancer

alb1

alb2

FINAL STEP: Delete the Infrastructure

terraform destroy -auto-approve

Top comments (4)

Collapse
 
amarg_12 profile image
Amar •

Through terraform i have created ecs and services and codepipeline integrated with org git as source but when i deploy through GitHub the task definition is updated with latest version but when i modify any different resources in terraform and ecs get deployed with initial task definition which is old there in tf file how to overcome this issue.

Collapse
 
erozedguy profile image
erozedguy •

Hey, It's been a long since I created it but, I'm not sure maybe you should revie the container_definitios block in terraform (task_definition resource), there you have the image you are using and the tag. As best practices for docker in general we shouldn't use latest tag, also take a look in the CI/CD pipeline, the image should be tagged using the commit hash. I hope I've helped you :)

Collapse
 
amarg_12 profile image
Amar •

I didn't get you sorry , commit has unique value every time , i want to tell tf about the changes happened at outside and stop it to deploy and maintain the state .

Thread Thread
 
erozedguy profile image
erozedguy •

What did you get as outputs during the tf plan ?. You must review what changes will be applied. Maybe the problem is in the task definition resource, you need to review if the declaration you have in the tf code corresponds to the current configuration you have in AWS, for example the tag of the docket image in the containers_definitions block

Read next

steeve profile image

S3 Client against disasters (hacks, fires, catastrophes)

Steeve -

depaa profile image

Building a Serverless Newsletter: Your Guide to AWS and Amazon SES

Matteo Depascale -

lukedoesinfra profile image

Terraform Modules at Super

Luke Livingstone -

zirkelc profile image

A Simple Way to Sign AWS Requests with Signature V4

Chris Cook -