DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» is a community of 964,423 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Cover image for Containerize CI pipelines with Earthly
Richard Kovacs
Richard Kovacs

Posted on

Containerize CI pipelines with Earthly

I was a big fan of GitHub Actions to run CI pipelines on our open-source stuff. Compare to other vendors, they did a really good job. It is easy to edit, nicely documented, and has tons of actions to reuse. On the other hand, the Github Actions vendor locks our projects. We ❀️ GitHub, but vendor locking should be a burning problem.

Before I jump into Earthly, I would like to write about the biggest competitor in space: Dagger. Dagger is a new project from Docker Inc. (whatever they called actually), targeting locally reproducible containerized CI pipelines. Sounds fancy, except for one point: it uses CUE language to define jobs. I guess they have the reasons, but I don't want to learn a language to create a build task. Nonsense for me, sorry.

That's where Earthly comes into the picture. It combines Makefile and Dockerfile syntax, so we should jump into with almost zero learning curve πŸ‘. It has a huge amount of features like caching, reusability, live debugging, secret management, etc., for a detailed list please follow the documentation (+100 for documentation).

It's craft time baby!

Let's start with a simple Go lint job and learn Docker In Docker first:

# cat Earthfile
VERSION 0.6

lint:
    FROM earthly/dind:alpine
    WORKDIR /workdir
    COPY . ./
    WITH DOCKER --pull golangci/golangci-lint:v1.49.0
        RUN docker run -w $PWD -v $PWD:$PWD golangci/golangci-lint:v1.49.0 golangci-lint run --timeout 240s
    END
Enter fullscreen mode Exit fullscreen mode

To execute the job we have to install Earthly:

# cat Makefile
BIN_PATH = $(shell pwd)/bin
$(shell mkdir $(BIN_PATH) &>/dev/null)

EARTHLY = $(BIN_PATH)/earthly
earthly:
ifeq (,$(wildcard $(EARTHLY)))
    curl -L https://github.com/earthly/earthly/releases/download/v0.6.23/earthly-linux-amd64 -o $(EARTHLY)
    chmod +x $(EARTHLY)
endif
Enter fullscreen mode Exit fullscreen mode
make earthly && ./bin/earthly --allow-privileged +lint
Enter fullscreen mode Exit fullscreen mode

I think it can't be more simple.

Build and cache dependencies

# cat Earthfile
VERSION 0.6
FROM golang:1.18.0
WORKDIR /workdir

deps:
    COPY go.mod go.sum ./
    RUN go mod tidy
    RUN go mod download
    SAVE ARTIFACT go.mod AS LOCAL go.mod
    SAVE ARTIFACT go.sum AS LOCAL go.sum

test:
    FROM +deps
    COPY . ./
    RUN make _test
Enter fullscreen mode Exit fullscreen mode

I want to write something here, but code describes everything. To spin up job, you have to execute the following command, this time without --allow-privileged:

./bin/earthly +test
Enter fullscreen mode Exit fullscreen mode

Automation, automation, automation, ...

First I would like to execute jobs before I push the change into remote Git. Husky looks like a pretty handy tool to manage Git hooks.

Download Husky:

# cat Makefile
husky:
    GOBIN=$(BIN_PATH) go install github.com/automation-co/husky@v0.2.5
Enter fullscreen mode Exit fullscreen mode
make husky && ./bin/husky init
Enter fullscreen mode Exit fullscreen mode

Edit hook files:

# cat .husky/hooks/pre-commit
#!/bin/bash

husky install
Enter fullscreen mode Exit fullscreen mode
# cat .husky/hooks/pre-push
#!/bin/bash

if [[ "$SKIP_GIT_PUSH_HOOK" ]]; then exit 0; fi

set -e

if git status --short | grep -qv "??"; then
    git stash
    function unstash() {
        git reset --hard
        git stash pop
    }
    trap unstash EXIT
fi

make lint test

git diff --exit-code --quiet || (git status && exit 1)
Enter fullscreen mode Exit fullscreen mode

Install hooks:

./bin/husky install
Enter fullscreen mode Exit fullscreen mode

Create Makefile targets:

# cat Makefile
lint: earthly
    $(EARTHLY) -P +lint

test: earthly
    $(EARTHLY) +test
Enter fullscreen mode Exit fullscreen mode

And GitHub Action integration too, we still need to run jobs at the central place:

name: Lint and test
on: [push]
jobs:
  lint:
    name: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: make lint

  go-test:
    name: go tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: make test
Enter fullscreen mode Exit fullscreen mode

Enjoy πŸŽ‰

In this way, before each commit, Husky installs the latest version of hooks and then executes all the tasks before the push. GitHub executes actions after the push, so we did what we could to deliver high-quality software.

Top comments (0)

Update Your DEV Experience Level:

Settings

Go to your customization settings to nudge your home feed to show content more relevant to your developer experience level. πŸ›