Docker & Azure Pipelines: Better Together!

If you want to use an Azure Pipelines Self Hosted agents, you have some different options. But only few of them allow you to be truly flexible and scalable.

In this post we will see how to run the Azure Pipelines Agents in Docker, so you can take advantage of orchestrators like Kubernetes, Azure Container Instances, and many more.

Let's get into it.

Intro

Today I wanna show you how you can run your Azure Pipelines agents in Docker so you can then take advantage of orchestrators like Kubernetes, Azure Container Instances, and others to make your CI CD agents truly scalable and flexible.

I will walk you through a complete container example.

Video

As usual, if you are a visual learner, or simply prefer to watch and listen instead of reading, here you have the video with the whole explanation and demo, which to be fair is much more complete than this post.

(Link to the video: https://youtu.be/rO-VKProMp8)

If you rather prefer reading, well... let's just continue :)

Supported Platforms

First things first: Let's talk about the supported platforms.

The good news is that both Windows and Linux are supported as container hosts. If your CICD process requires tools that are platform specific there is no problem.

Prerequisites

Second thing, prerequisites. On Windows you would need to remember to enable Hyper-V, which is disabled by default, while on Linux it just works.

Of course you would need to have Docker installed on both platforms, and that could be either Docker Community Edition or Docker Enterprise Edition.

Container Image Creation

Alright, let see now how to create a basic image. The process is slightly different between Windows and Linux, so I will cover both.

I will start with Windows, but you can skip to the Linux section if you want.

Windows

Click here to check the video and see this explained in much greater details.

To create a container image for Windows you need a Docker file, of course.

FROM mcr.microsoft.com/windows/servercore:ltsc2019

WORKDIR /azp

COPY start.ps1 .

CMD powershell .\start.ps1

We derive from the Server Core image, which contains already all the dependencies we need.

Next we need to download the PowerShell script Microsoft provides (here):

if (-not (Test-Path Env:AZP_URL)) {
  Write-Error "error: missing AZP_URL environment variable"
  exit 1
}

if (-not (Test-Path Env:AZP_TOKEN_FILE)) {
  if (-not (Test-Path Env:AZP_TOKEN)) {
    Write-Error "error: missing AZP_TOKEN environment variable"
    exit 1
  }

  $Env:AZP_TOKEN_FILE = "\azp\.token"
  $Env:AZP_TOKEN | Out-File -FilePath $Env:AZP_TOKEN_FILE
}

Remove-Item Env:AZP_TOKEN

if ($Env:AZP_WORK -and -not (Test-Path Env:AZP_WORK)) {
  New-Item $Env:AZP_WORK -ItemType directory | Out-Null
}

New-Item "\azp\agent" -ItemType directory | Out-Null

# Let the agent ignore the token env variables
$Env:VSO_AGENT_IGNORE = "AZP_TOKEN,AZP_TOKEN_FILE"

Set-Location agent

Write-Host "1. Determining matching Azure Pipelines agent..." -ForegroundColor Cyan

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$(Get-Content ${Env:AZP_TOKEN_FILE})"))
$package = Invoke-RestMethod -Headers @{Authorization=("Basic $base64AuthInfo")} "$(${Env:AZP_URL})/_apis/distributedtask/packages/agent?platform=win-x64&`$top=1"
$packageUrl = $package[0].Value.downloadUrl

Write-Host $packageUrl

Write-Host "2. Downloading and installing Azure Pipelines agent..." -ForegroundColor Cyan

$wc = New-Object System.Net.WebClient
$wc.DownloadFile($packageUrl, "$(Get-Location)\agent.zip")

Expand-Archive -Path "agent.zip" -DestinationPath "\azp\agent"

try
{
  Write-Host "3. Configuring Azure Pipelines agent..." -ForegroundColor Cyan

  .\config.cmd --unattended `
    --agent "$(if (Test-Path Env:AZP_AGENT_NAME) { ${Env:AZP_AGENT_NAME} } else { ${Env:computername} })" `
    --url "$(${Env:AZP_URL})" `
    --auth PAT `
    --token "$(Get-Content ${Env:AZP_TOKEN_FILE})" `
    --pool "$(if (Test-Path Env:AZP_POOL) { ${Env:AZP_POOL} } else { 'Default' })" `
    --work "$(if (Test-Path Env:AZP_WORK) { ${Env:AZP_WORK} } else { '_work' })" `
    --replace

  Write-Host "4. Running Azure Pipelines agent..." -ForegroundColor Cyan

  .\run.cmd
}
finally
{
  Write-Host "Cleanup. Removing Azure Pipelines agent..." -ForegroundColor Cyan

  .\config.cmd remove --unattended `
    --auth PAT `
    --token "$(Get-Content ${Env:AZP_TOKEN_FILE})"
}

This script checks the environment, create some folders and install and configure the agent.

If you want a fresh agent container for every pipeline job, pass the --once flag to the run command. You must also use a container orchestration system, like Kubernetes or Azure Container Instances, to make sure you start new copies of the container when the work completes.

Finally, let's just create the image:

docker build -t dockeragentwin:latest -f Dockerfile.windows .

Once again, you may want to check the video and see this explained in much greater details.

Linux

Click here to check the video and see this explained in much greater details.

For Linux, the Dockerfile looks a little different:

FROM ubuntu:18.04

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        jq \
        git \
        iputils-ping \
        libcurl4 \
        libicu60 \
        libunwind8 \
        netcat \
        libssl1.0

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]

The main differences here are that we use a Linux base image, and that we need to install the dependencies manually using apt-get.

Then we need the startup script as well, but this time is a bash script (also provided by Microsoft here)

#!/bin/bash
set -e

if [ -z "$AZP_URL" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "$AZP_TOKEN_FILE" ]; then
  if [ -z "$AZP_TOKEN" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE=/azp/.token
  echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE"
fi

unset AZP_TOKEN

if [ -n "$AZP_WORK" ]; then
  mkdir -p "$AZP_WORK"
fi

rm -rf /azp/agent
mkdir /azp/agent
cd /azp/agent

export AGENT_ALLOW_RUNASROOT="1"

cleanup() {
  if [ -e config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    ./config.sh remove --unattended \
      --auth PAT \
      --token $(cat "$AZP_TOKEN_FILE")
  fi
}

print_header() {
  lightcyan='\033[1;36m'
  nocolor='\033[0m'
  echo -e "${lightcyan}$1${nocolor}"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_RESPONSE=$(curl -LsS \
  -u user:$(cat "$AZP_TOKEN_FILE") \
  -H 'Accept:application/json;api-version=3.0-preview' \
  "$AZP_URL/_apis/distributedtask/packages/agent?platform=linux-x64")

if echo "$AZP_AGENT_RESPONSE" | jq . >/dev/null 2>&1; then
  AZP_AGENTPACKAGE_URL=$(echo "$AZP_AGENT_RESPONSE" \
    | jq -r '.value | map([.version.major,.version.minor,.version.patch,.downloadUrl]) | sort | .[length-1] | .[3]')
fi

if [ -z "$AZP_AGENTPACKAGE_URL" -o "$AZP_AGENTPACKAGE_URL" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent - check that account '$AZP_URL' is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and installing Azure Pipelines agent..."

curl -LsS $AZP_AGENTPACKAGE_URL | tar -xz & wait $!

source ./env.sh

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "$AZP_URL" \
  --auth PAT \
  --token $(cat "$AZP_TOKEN_FILE") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

# To be aware of TERM and INT signals call run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh & wait $!

Although it looks different, this script does exactly the same things of the Windows one: prepare the system, and install and configure the agent.

Let's build the image:

docker build -t dockeragentlin:latest -f Dockerfile.linux .

Once again, if you want a fresh agent container for every pipeline job, pass the --once flag to the run command.

Remember to check the video and see this explained in much greater details.

Run the container

That was not too bad right? And the fact that, as we have seen, Microsoft provides the full script for both platforms makes it even easier.

If you are interested in how you can Drastically improve you Docker Build Performance, I have a video which talks just about that.

To run the container, of course we can use the docker run command:

docker run -e AZP_URL=<Azure DevOps instance> -e AZP_TOKEN=<PAT token> -e AZP_AGENT_NAME=<Agent name> dockeragentlin:latest

As you've probably noticed, there are some environment variables we can use to control the execution. Needless to say, those works in both a local Docker instance as well as any orchestrator.

We can use the _AZP_URL_ variable to specify the URL of the instance of Azure DevOps we want to connect to. This parameter, as you may imagine, is required.

We also need to pass the _AZP_TOKEN_ variable and its value must be a Personal Access Token with the Read and Manage Agent Pools scope that is valid on the instance of Azure DevOps we are connecting to.

The other 3 variables are instead optional.

• _AZP_AGENT_NAME_ lets you set a name for the agent and it defaults to the container hostname
• _AZP_POOL_ is used to specify the Agent Pool that you want that specific agent to be part of
• And finally _AZP_WORK_ lets you cange the working directory of the agent, which by default is _work.

Extend the basic image

At this point we have a fully working basic build agent.

You can of course extend the Dockerfile to include additional tools and their dependencies, or build your own container by using this one as a base layer.

Just make sure that the start script is called and it is the last command in the Dockerfile, and ensure that any of the derivative containers don't remove any of the dependencies of the original one.

The agent in Azure DevOps

Check out this section of the video to see how a new agent appears in Azure DevOps.

Important Notes

Before we close, just a few notes.

First of all, as we have seen before, the startup script installs the agents every time the image is run from scratch. If you want the startup to be quicker, you can consider installing the agent as part of your docker image creation instead.

Also, remember that each agent automatically updates itself when it runs a task that requires a newer version of the agent.

Second thing to note is that in order to use Docker from within a Docker container, you have to bind-mount the Docker socket.

As this very noticeable warning message on the official documentation site says, remember that doing so it is not recommended because it has serious security implications. The code inside the container can inf cat run as root on your Docker host.

Lastly, if you are trying to run this on Windows and you experience an error like the one below, you need to install Git Bash by downloading and installing git-scm.

Conclusions

Alright, that's it for today. Now you can take those container images and run them in any orchestration of your choice, to have a truly flexible and scalable CICD platform with Azure Pipelines.

Let me know in the comment section below what you think about this, and if you'd like me to make a follow-up post/video on this showing how to deploy the containerized Azure Pipeline agents to Kubernetes.