Streamlining AMIs using Packer, Vault & GitHub Actions

Nowadays, if you want to minimize human errors and maintain a consistent process for how software is released then you are going to rely on Continuous integration and continuous deployment (CI/CD). It's really hard to imagine how much productivity they bring into the plate.

In this tutorial, we are going to take entire AWS instance backup using tools like Packer and see how it solves our problem and make our life much easier.

Amazon Machine Image (AMI)

An Amazon Machine Image (AMI) is a special type of virtual appliance that is used to create a virtual machine within the Amazon Elastic Compute Cloud ("EC2"). It serves as the basic unit of deployment for services delivered using EC2. -- Wikipedia

An AMI includes the following:

• A template for the root volume for the instance (for example, an operating system, an application server, and applications)
• Launch permissions that control which AWS accounts can use the AMI to launch instances.
• A block device mapping that specifies the volumes to attach to the instance when it's launched.

What is Packer ?

Packer is a tool for building identical machine images for multiple platforms from a single source configuration.

Image Source : https://www.hashicorp.com/

Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer comes out of the box with support for many platforms.

To know more about Packer, visit : https://developer.hashicorp.com/packer

Project Structure

GitHub Repository : https://github.com/mukulmantosh/Packer-Exercises

• .github - Workflow files for GitHub Actions
• packer - Contains HCL2 Packer templates, Shell Scripts etc.
• Dockerfile - Building Docker Image
• main.py - FastAPI Routes handling two endpoints
• requirements.txt - listing all the dependencies for a specific Python project

Let's Begin

I have used Amazon Linux 2 with arm64 architecture as our base AMI.

The custom AMI name is FastAPI_Base_Image. It's a clean AMI without any OS/Software dependencies.

If you are not sure how to create an AMI, follow this link : https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html

Dockerfile

I will create a container from the Dockerfile which is taking Python 3.9 as the base image and followed with python dependencies installation and starting the uvicorn server.

The image is already hosted in DockerHub.

URL : https://hub.docker.com/r/mukulmantosh/packerexercise

We have compiled for three architectures. Thanks to Docker Buildx.

• amd64
• arm64
• arm/v7

Packer Template

packer/build.pkr.hcl

variable "ami_name" {
  type        = string
  description = "The name of the newly created AMI"
  default     = "fastapi-nginx-ami-{{timestamp}}"
}

variable "security_group" {
  type        = string
  description = "SG specific for Packer"
  default     = "sg-064ad8064cf203657"
}

variable "tags" {
  type = map(string)
  default = {
    "Name" : "FastAPI-NGINX-AMI-{{timestamp}}"
    "Environment" : "Production"
    "OS_Version" : "Amazon Linux 2"
    "Release" : "Latest"
    "Creator" : "Packer"
  }
}
source "amazon-ebs" "nginx-server-packer" {
  ami_name          = var.ami_name
  ami_description   = "AWS Instance Image Created by Packer on {{timestamp}}"
  instance_type     = "c6g.medium"
  region            = "ap-south-1"
  security_group_id = var.security_group
  tags              = var.tags

  run_tags        = var.tags
  run_volume_tags = var.tags
  snapshot_tags   = var.tags


  source_ami_filter {
    filters = {
      name                = "FastAPI_Base_Image"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }

    most_recent = true
    owners      = ["self"]
  }
  ssh_username = "ec2-user"



}

build {
  sources = [
    "source.amazon-ebs.nginx-server-packer"
  ]

  provisioner "shell" {
    inline = [
      "sudo yum update -y",
    ]
  }

  provisioner "shell" {
    script       = "./scripts/build.sh"
    pause_before = "10s"
    timeout      = "300s"
  }

  provisioner "file" {
    source      = "./scripts/fastapi.conf"
    destination = "/tmp/fastapi.conf"
  }


  provisioner "shell" {
    inline = ["sudo mv /tmp/fastapi.conf /etc/nginx/conf.d/fastapi.conf"]
  }

  error-cleanup-provisioner "shell" {
    inline = ["echo 'update provisioner failed' > packer_log.txt"]
  }

}

User Variables

User variables allow your templates to be further configured with variables from the command-line, environment variables, Vault, or files. This lets you parameterize your templates so that you can keep secret tokens, environment-specific data, and other types of information out of your templates. This maximizes the portability of the template.

Builders

Builders create machines and generate images from those machines for various platforms (EC2, GCP, Azure, VMware, VirtualBox) etc. Packer also has some builders that perform helper tasks, like running provisioners.

Provisioners

Provisioners use built-in and third-party software to install and configure the machine image after booting. Provisioners prepare the system, so you may want to use them for the following use cases:

• installing packages
• patching the kernel
• creating users
• downloading application code

Post-Processors

Post-processors run after builders and provisioners. Post-processors are optional, and you can use them to upload artifacts, re-package files, and more.

On Error Provisioner

You can optionally create a single specialized provisioner called an error-cleanup-provisioner. This provisioner will not run unless the normal provisioning run fails. If the normal provisioning run does fail, this special error provisioner will run before the instance is shut down. This allows you to make last minute changes and clean up behaviors that Packer may not be able to clean up on its own.

The amazon-ebs Packer builder is able to create Amazon AMIs backed by EBS volumes for use in EC2.

source "amazon-ebs" 

This builder builds an AMI by launching an EC2 instance from a source AMI, provisioning that running machine, and then creating an AMI from that machine. This is all done in your own AWS account. The builder will create temporary keypairs, security group rules, etc. that provide it temporary access to the instance while the image is being created. This simplifies configuration quite a bit.

The builder does not manage AMIs. Once it creates an AMI and stores it in your account, it is up to you to use, delete, etc. the AMI.

To know more, visit this link : https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs

In the “source_ami_filter” section, We are filtering based on the base AMI which we created earlier.

  source_ami_filter {
    filters = {
      name                = "FastAPI_Base_Image"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }

    most_recent = true
    owners      = ["self"]
  }

most_recent - Selects the newest created image when true.
owners - You may specify one or more AWS account IDs, "self" (which will use the account whose credentials you are using to run Packer)

We are using a Packer function called “timestamp” to generate UNIX timestamp, which helps to get a unique AMI name on every build.

By default the AMI’s you create will be private. If you want to share the AMI’s with other accounts you can make use of the “ami_users” option in packer.

If you want to build images in multi-region, you can specify the below code in the source section.

  ami_regions   = ["us-west-2", "us-east-1", "eu-central-1"]

In the provisioner section we will be updating the OS along-with installing scripts and copy nginx configuration.

packer/scripts/build.sh

Installing Docker, NGINX, and pulling latest application image from DockerHub and starting the container.

#!/bin/bash
sudo yum install jq -y
sudo yum install -y git

sudo yum install -y docker
sudo usermod -a -G docker ec2-user
sudo systemctl enable docker.service
sudo systemctl start docker.service

sudo amazon-linux-extras install nginx1 -y
sudo systemctl enable nginx.service
sudo systemctl start nginx.service

IMAGE_TAG=`curl -L -s 'https://hub.docker.com/v2/repositories/mukulmantosh/packerexercise/tags'|jq '."results"[0]["name"]' | bc`

sudo docker pull mukulmantosh/packerexercise:$IMAGE_TAG
sudo docker run -d --name fastapi --restart always -p 8080:8080 mukulmantosh/packerexercise:$IMAGE_TAG

packer/scripts/fastapi.conf

Copy the configuration to NGINX configuration folder. So, NGINX will proxy the request to backend.

upstream fastapi {
    server 127.0.0.1:8080;
}
server {

    listen 80;

    location / {
        proxy_pass http://fastapi;
        proxy_set_header X-Forwarded-For 
        $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

}

Building Template

Before you begin to build, make sure you have setup the following keys in your system and aws-cli is installed in your machine.

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

There are two commands which you need to run before you execute build.

packer fmt build.pkr.hcl

The packer fmt Packer command is used to format HCL2 configuration files to a canonical format and style

packer validate build.pkr.hcl

The packer validate Packer command is used to validate the syntax and configuration of a template

Starting the Build

packer build build.pkr.hcl

The packer build command takes a template and runs all the builds within it in order to generate a set of artifacts.

You can see the new AMI has been successfully created and tag has been assigned.

You must have observed in the packer template, that we are using a custom security group. By default, Packer creates security group which access port 22 (0.0.0.0) from anywhere.

This posses security risk and to minimize that, I created a custom security group (Packer_SG) which allows only My IP.

variable "security_group" {
  type        = string
  description = "SG specific for Packer"
  default     = "sg-064ad8064cf203657"
}

You can add more security by taking leverage of Session Manager Connections.

Session Manager Connections
Support for the AWS Systems Manager session manager lets users manage EC2 instances without the need to open inbound ports, or maintain bastion hosts.

GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline.

Self-hosted runners

For our setup we will be using self-hosted Github runners.

Self-hosted runners offer more control of hardware, operating system, and software tools than GitHub-hosted runners provide. With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitHub-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud.

Don't know how to setup ? Follow the below link :

• Deploying Self-hosted Runners for GitHub Actions
• About self-hosted runners

As from security standpoint, we will make sure "Packer_SG" security group allow inbound port 22 for Github Action IP.

Execute Pipeline

Before proceeding, make sure to create the secrets which will be required in the build process.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

.github/workflows/build-packer.yml

name: Packer

on:
  push:
    branches: main

jobs:
  packer:
    runs-on: self-hosted
    name: packer

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1-node16
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-south-1


      # validate templates
      - name: Validate Template
        uses: hashicorp/packer-github-actions@master
        with:
          command: validate
          arguments: -syntax-only
          target: build.pkr.hcl
          working_directory: ./packer


      # build artifact
      - name: Build Artifact
        uses: hashicorp/packer-github-actions@master
        with:
          command: build
          arguments: "-color=false -on-error=abort"
          target: build.pkr.hcl
          working_directory: ./packer

On inspecting the YAML file, you can clearly observe that we will be validating packer templates and then followed by building the artifact.

Let me make a small change in main branch. So, the pipeline will get triggered.

You can see now, the new AMI is created.

Vault

HashiCorp Vault tightly controls access to secrets and encryption keys by authenticating against trusted sources of identity such as Active Directory, LDAP, Kubernetes, Cloud Foundry, and cloud platforms. Vault enables fine grained authorization of which users and applications are permitted access to secrets and keys.

To know more about Vault, visit this link :

• https://developer.hashicorp.com/vault

The reason we are using Vault over here is to create dynamic user credentials.

This helps us to avoid setting up environment variables for

export AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXX"
export AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXX"

Putting this keys in local machine, might expose some risks. So, I would recommend trying out AWS Secrets Engine.

AWS Secrets Engine

The AWS secrets engine generates AWS access credentials dynamically based on IAM policies. This generally makes working with AWS IAM easier, since it does not involve clicking in the web UI. Additionally, the process is codified and mapped to internal auth methods (such as LDAP). The AWS IAM credentials are time-based and are automatically revoked when the Vault lease expires.

I have already setup Vault in my local machine.

Follow the below link for setting up Vault.

• https://developer.hashicorp.com/vault/downloads

You can either setup in your local machine or use HashiCorp Cloud.

Let's now begin by enabling the AWS secrets engine in our Vault server which is running locally.

Now, click on Configuration to setup our credentials.

Provide the AWS credentials and region which will be used to create user and attach role to them.

Next, I will modify lease time to 15 minutes. So, once the user is created it will be deleted automatically after 15 minutes.

Click on Save.

I have configured the AWS credential. Now, I will create the role which is going to be attached to the new user.

Policy Document

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CopyImage",
        "ec2:CreateImage",
        "ec2:CreateKeypair",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteKeyPair",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSnapshot",
        "ec2:DeleteVolume",
        "ec2:DeregisterImage",
        "ec2:DescribeImageAttribute",
        "ec2:DescribeImages",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DetachVolume",
        "ec2:GetPasswordData",
        "ec2:ModifyImageAttribute",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifySnapshotAttribute",
        "ec2:RegisterImage",
        "ec2:RunInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": "*"
    }
  ]
}

I would recommend follow defense in depth and principle of least privilege.

Most of them don't encourage that policy document should contain delete permissions.

I came across an interesting article for tightening your policy document and make it more secure. So, it won't interfere with other instances.

Please checkout the below link :

• Towards a more Restricted Set of AWS IAM Permissions for Packer

Now, I will click on Generate Credentials.

Now, it's going to create a IAM user which is valid for 15 minutes (900 seconds)

You can see below, the new user is appearing in the IAM User section.

The PackerRole with assigned permissions are also being reflected.

Now, we are going to make sure that Packer should generate this credentials automatically.

Let's begin by editing the build.pkr.hcl file.

You need to add this line before closing of the source block.

  vault_aws_engine {
    name = "PackerRole"
  }

Next, you need to setup environment variables.

Windows :

set VAULT_ADDR=http://127.0.0.1:8200
set VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX

Linux :

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX

Once, we are done setting up our environment variables. We need to validate everything is working as expected by running the validate command.

packer validate build.pkr.hcl

If you receive this message The configuration is valid then you are good to proceed.

To initiate the build run the below command :

packer build build.pkr.hcl

• Note : Make sure before your begin build. The security group Packer_SG allows inbound access to port 22 from MyIP, as you are running the build from local machine.

Observe the message : You're using Vault-generated AWS credentials

This is going to pick the credentials from Vault, which is going to dynamically create a new user and attach the role.

The user will get automatically deleted based on the expiry specified.

Once, the build is complete. You will find the new image appearing in the AMI section.

Final Destination

Congratulations !!! You did it 🏆🏆🏆

If you liked this tutorial 😊, make sure to share across your friends and colleagues.

References

• Build a Golden Image Pipeline with HCP Packer
• How to Set Up Continuous Golden AMI Vulnerability Assessments with Amazon Inspector
• Packer Docs