Earthly: Beyond Docker

We've been using containers for software packaging for a long time. But let’s face it, Docker's user experience isn't really the greatest. Building and testing usually entails having a bunch of Makefiles, fiddly Dockerfiles, a bit of black magic, and a bundle of Bash scripts. It’s a path that can lead into insanity.

And it’s a shame because Docker could be much friendlier. Earthly developers certainly believe in this: they have created a tool that fills the space between Docker and Make.

At the end of this tutorial, you’ll know how Earthly works. As a bonus, you'll have a pipeline that builds, tests, and pushes a Docker image into a remote repository.

How Earthly works

Earthly is a Docker-based build tool. It does not, however, replace language-specific tools like Maven, Gradle, or Webpack. Instead, it leverages and integrates with them — acting as a glue.

Using Docker containers as the core mechanic to achieve repeatability, every Earthly command runs in an isolated environment and is effortlessly parallelized whenever possible.

Earthly consists of the CLI binary and a Docker image called earthly/buildkit, based on Docker’s BuildKitd.

Earthly wants you to get the same experience everywhere. BuildKit’s role is to yield portable Docker-compatible layers. Thus, we get the same result whether on our local machine and in our CI systems.

Does Earthly deliver on its promises?

Earthly promises better, repeatable builds. But, does it deliver? Here are some of the best features and some of the problems found while doing a test drive.

Pros:

• Earthly, like Docker, is language-agnostic. Its syntax feels familiar since it mixes Make and Dockerfiles.
• Their site has good documentation.
• Configuration is low maintenance. There are only a few settings to tweak.
• Useful even in the cases you don’t need Docker. Container images are only one of the possible outputs. For instance, you can use Earthly to test and compile binaries in a clean environment.
• The import system works well in multirepo and monorepos.

Cons:

• It doesn’t actually replace Makefile or Docker Compose. And you will still need some shell scripts.
• It’s slower than native Docker because it relies on copying files instead of mounts. Of course, this is by design to ensure repeatability.
• Multiplatform builds sometimes use emulation, so it may be too slow for some use cases.
• Error messages can get really confusing.

Setting up Earthly

Earthly configuration files are called, as typical in software engineering, Earthfiles. Opening one reminds us of Dockerfiles with a twist of Makefiles.

Earthly takes the syntax from Docker and expands it for a general use case. While Dockerfiles are only meant to produce container images, Earthly can generate artifacts.

The following Earthfile was taken from Earthly’s official guide. It describes two targets: build and docker.

FROM golang:1.15-alpine3.13
WORKDIR /go-example

build:
    COPY main.go .
    RUN go build -o build/go-example main.go
    SAVE ARTIFACT build/go-example /go-example AS LOCAL build/go-example

docker:
    COPY +build/go-example .
    ENTRYPOINT ["/go-example/go-example"]
    SAVE IMAGE go-example:latest

The build target compiles a Go program inside a container, while docker copies it into a production-ready image that can be pushed into any container registry and run anywhere.

The FROM keyword works as you would expect if familiar with Dockerfiles. Likewise, COPY, ENTRYPOINT, WORKDIR, and RUN, behave similarly. Two concepts deserve a pause to explain further:

• targets: the thing we want to build. Earthly uses a target-based system inspired by Make.
• references: targets can reference other targets. As in Make, Earthly will sort dependencies and do the right thing.
• extended commands: Earthly features specialized commands such as SAVE IMAGE and SAVE ARTIFACT.

To run this Earthfile we need to run: earthly +TARGET. The first time, Earthly will pull the image and create a cache volume. The cache will persist the build files between runs.

$ earthly +build

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
golang:1.15-alpine3.13 | --> Load metadata linux/arm64
             context | --> local context .
               +base | --> FROM golang:1.15-alpine3.13
             context | transferred 1 file(s) for context . (2.1 MB, 4 file/dir stats)
               +base | *cached* --> WORKDIR /go-example
              +build | *cached* --> COPY main.go .
              +build | *cached* --> RUN go build -o build/go-example main.go
              output | --> exporting outputs
================================ SUCCESS [main] ================================
              +build | Artifact +build/go-example as local build/go-example

Or we can directly go to the Docker generation target with:

$ earthly +docker

               +base | --> FROM golang:1.15-alpine3.13
               +base | *cached* --> WORKDIR /go-example
             context | transferred 3 file(s) for context . (2.1 MB, 4 file/dir stats)
              +build | *cached* --> COPY main.go .
              +build | *cached* --> RUN go build -o build/go-example main.go
              +build | *cached* --> SAVE ARTIFACT build/go-example +build/go-example AS LOCAL build/go-example
             +docker | --> COPY +build/go-example ./
              output | [██████████] exporting layers ... 100%
              output | [          ] exporting manifest
================================ SUCCESS [main] ================================
             +docker | Image +docker as go-example:latest
              +build | Artifact +build/go-example as local build/go-example

Since the docker target depends on build, Earthly first runs the build step and then proceeds to build and export the image into the local Docker directory.

Extending Earthfiles

Let’s try Earthly in a more involved example. Please go ahead and fork this demo repository.

The provided Dockerfile builds a Node Alpine container image.

FROM node:14.17-alpine3.12
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY src src
EXPOSE 3000
ENTRYPOINT ["npm", "start"]

You can build it with: docker build . -t mydemo and then run it as: docker run -it -p 3000:3000 mydemo. Browsing the localhost on port 3000 should print a "hello, world" message.

How would we go about turning this into a working Earthfile? Let’s start one from scratch.

The first two lines remain the same and will be used for every target.

FROM node:14.17-alpine3.12
WORKDIR /app

Now we add the build target, which downloads dependencies and copies the source.

build:
    COPY package.json package-lock.json ./
    COPY --if-exists node_modules node_modules
    RUN npm install
    COPY src src

We have added a COPY statement to copy the node_modules folder into the build environment.

Since npm runs in the container, we need to copy its output files back outside. We do this with Earthly’s special command SAVE ARTIFACT ... AS LOCAL.

   SAVE ARTIFACT node_modules AS LOCAL ./node_modules
    SAVE ARTIFACT package.json AS LOCAL ./package.json
    SAVE ARTIFACT package-lock.json AS LOCAL ./package-lock.json

Running earthly +build should install the Node dependencies. The end result of this process is similar to what we would get if we run npm install directly, with the added benefit that we get the same outcome everywhere.

Next, we should create the Docker image. We can achieve this with:

docker:
    FROM +build
    EXPOSE 3000
    ENTRYPOINT ["npm", "start"]
    SAVE IMAGE semaphore-demo-earthly:latest

The new target picks up from where the build stopped. It adds the start command and exports the image into the host’s local Docker registry.

Testing with Earthly

One of the most powerful features Earthly brings is the ability to extend Dockerfiles with tests. Let’s add unit tests by extending build with a target that runs: npm test.

tests:
    FROM +build
    COPY spec spec
    RUN npm test

Now we can run earthly +tests to get the results.

Adding a linting target is also trivial:

lint:
    FROM +build
    COPY .jshintrc ./
    RUN npm run lint

To run more involved tests, such as integration tests or end-to-end tests, we can reuse existing Docker Compose manifests in order to bring up other containers during the integration stage.

integration-tests:
    FROM +build
    COPY docker-compose.yml ./
    COPY integration-tests integration-tests
    WITH DOCKER --compose docker-compose.yml --service db
        RUN sleep 10 && npm run integration-tests
    END

Earthly uses a Docker-in-Docker approach to start helper containers:

1. Copies docker-compose.yml and the integration tests in the main container (the application container).
2. Starts a PostgreSQL container inside the main container. The special WITH DOCKER command accepts precisely one RUN statement.
3. Runs the integration tests.

Our Earthfile is good enough for now. Let’s upload it into the repository so we can configure a CI/CD pipeline in the next section.

$ git add Earthfile
$ git commit -m "push Earthfile"
$ git push origin master

Earthly Continuous Integration Pipeline

We are now all set to test the application with Earthly and continuous integration. Our pipeline will pull to and push from Docker Hub, something that will require authentication with the service.

Authentication in Semaphore happens via secrets. To create an encrypted secret, go to your organization menu and click on Settings > Secrets. Then, press Create secret and type the variables DOCKER_USERNAME and DOCKER_PASSWORD as shown below:

Next, initialize the forked demo repository. The procedure for adding a project is detailed in the getting started guide. Go check it out if this is the first time using Semaphore.

Since Earthly doesn’t come preinstalled in the Ubuntu CI image, we need to install it for every job. The simplest way of achieving this is with a pipeline-level prologue. The commands in the global prologue are executed before every job in the pipeline.

The Earthly installation command for Linux is:

sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'

Click on the pipeline and type the line in the prologue section.

Build block

The first block will start a build stage. In it, we will download and cache Node dependencies.

Type the following commands in the job:

echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly --ci +build
cache store

We’ll run these commands or a variation of them in every successive job in the pipeline:

• docker login: logs into Docker Hub with the credentials defined in the secret.
• checkout: clones the repository into the CI machine.
• cache: With the cache command, we can store node_modules in the project-level storage provided by Semaphore.
• earthly ci: runs Earthly with some CI-optimized settings.

To finish the block, enable the “dockerhub” secret, so the variable is decrypted and exported in the job.

Tests block

The next block will run fast tests such as unit tests and linting. Add the following commands in the block’s prologue:

echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore

Next, create two jobs:

earthly --ci +tests

And:

earthly --ci +lint

Integration tests block

The last job will run the integration test. Type the following commands. The -P switch is needed to enable privileged mode inside Docker (needed for Docker-in-Docker).

echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly --ci -P +integration-tests

Don’t forget to check that the secret is enabled in all blocks. When ready, run the workflow to try the CI pipeline.

Continuous Delivery with Earthly

The final result of a CI/CD pipeline is to deliver something that can be deployed. Here’s where the continuous delivery part comes into play. We’ll push the image to a remote repository so it can be later deployed in production.

A Semaphore workflow can span multiple pipelines, as long as they’re connected by promotions. Go back to the workflow editor and click on Add Promotion. Click the new pipeline and copy the global Earthly install commands, again:

sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'

The block in the pipeline will contain a single job to generate and push the Docker image into Docker Hub. The commands are as follows:

echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly +docker
docker tag semaphore-demo-earthly $DOCKER_USERNAME/semaphore-demo-earthly
docker push $DOCKER_USERNAME/semaphore-demo-earthly

We use earthly +docker to export the image into the CI machine, then a combination of docker tag and docker push to send it to the remote registry.

The only thing left is to enable the Docker Hub secret … and we’re done!

Rerun the workflow and click on promote to try pushing the image.

What do you think?

The Earthly project is young and under heavy development, with many features still in the experimental stage (we stuck to stable features during the course of the post). It’s not perfect, but it feels like a massive step in the right direction.

Skill up your Docker knowledge with these tutorials:

• Kubernetes vs. Docker: Understanding Containers in 2021
• How To Deploy a Go Web Application with Docker
• Dockerizing a Node.js Web Application
• Dockerizing a Ruby on Rails Application

Have you tried Earthly? Tell us about your experience.