Use Docker to build better CI/CD pipelines with Dagger

With the raises of DevOps practices, CI/CD (continuous integration & continuous deployment) takes a major place in every delivery workload.
CI/CD allow organizations to build, test and finally ship their applications more quickly and efficiently. It's a modern set of practices which allows to automatically trigger build, test or others types of jobs when the changes to the codebase are done.

In this quest of automation, we can use some CI/CD ecosystem like Github Actions, Gitlab-CI or many more.
However, a very promising new solution open-source is born called Dagger.

🤔 Dagger? What is it?

Dagger.io is a brand-new programmable CI/CD engine which is open-source. It was created by Solomon Hykes, the founder of Docker.
It's designed to use Buildkit from Docker in order to runs our pipelines inside containers.
For those who don't know, Buildkit is an improved backend used by Docker to build images. It's way more efficient than the old legacy Docker's builder due to its improved caching system, the parallelization of build tasks and the support of new features in Dockerfile.

Dagger is also programmable, so we must create our pipelines as code, Dagger himself is written in CUE (Cuelang), a Google's langage.

CUE is an acronym of "Configure Unify Execute" and so as its name suggests, it's not another general purpose language but a declarative one mainly used for data templating and validation, configuration or even code generation.

It's basically an evolution of more lambda languages like YAML or JSON and one of the thing which make it way better and modern is the presence of a package manager.

So going back to Dagger, you can use CUE to build your pipelines but you can also use the SDK available with multiples languages support:

• Go
• Python
• NodeJS

✨ Features and advantages

Dagger, thanks to his innovative containerized design and architecture, leads to many advantages over more conventional CI/CD methods.
As seen previously, Dagger works with Docker containers allowing it to be cross-compatible with every CI/CD runtime environment like Github Actions, Gitlab-CI, Travis-CI, etc.

One of the other main strengths of Dagger is the ability to do local testing of our pipeline using the Dagger CLI. This is made possible by the Dockerized design of it which makes the development and testing processes way easier compared to conventional CI/CD solutions.
Conversely, Dagger is also able to perform remote runs (directly on a self-hosted Github runner for example) with local sources.
To do this it is possible to use the environment variable DOCKER_HOST.

The Dockerized design also allows pipelines made with Dagger devkit to be run in every CI/CD runtime environment like, for example, Github Action (using the official Dagger Github Action from the marketplace).
Furthermore, it can also be run independently of the architecture of the platform. The only requirement is the Docker ecosystem support. So it can be run on a managed runner (eg. Github Runners), a self-hosted runner, a local machine, a serverless compute instance, etc.

Furthermore, Dagger has a solid caching system which caches every operation by default, but it's customizable.
It helps to reduce execution time of CI/CD jobs after the first runs by caching some unchanged required files like: the downloaded dependencies, some built binaries or just some CI/CD engine stuff.

Finally, Dagger is designed to be reusable thanks to his internal package manager (provided by the Cue language). Indeed, it's similar to the one of the language Golang.

package app

import (
    "dagger.io/dagger"
    "dagger.io/dagger/core"
    "universe.dagger.io/bash"
    "universe.dagger.io/docker"
)

Here you can see the import of some packages from the "universe". The Universe is a community package repository, all sources of these packages can be found here, you can also contribute to them.

❓ How does it works?

Dagger is language agnostic, the wish of the team behind is to allow developers to make their pipelines with the language of their choice.
For that, Dagger use a specific architecture. The SDKs (Go, Cue, Node and Python) don't actually run your pipelines themself. Instead, they send pipeline definitions to the Dagger GraphQL API which will after trigger the Dagger Engine.

Dagger's architecture scheme. Ref: https://dagger.io/blog/graphql

🎮 Demo application using Cue SDK

Dagger's team made a demo app with pipelines designed to execute build and test jobs. It's useful to discover and try Dagger.
To use it, you must, of course, have Docker installed since Dagger will use Buildkit but you must also install the CLI (the CUE one is available here: https://docs.dagger.io/sdk/cue/526369/install)
Once done, clone this repository: https://github.com/dagger/todoapp/blob/main/dagger.cue
Finally, open a terminal inside this freshly cloned project and run dagger-cue project update.

Now you're ready to get your hands in Dagger, if you look at the dagger.cue file you will see the plan.
A plan in Dagger orchestrates the Actions. It's the base component of your configuration.

Within this plan we can:

• interact with the client filesystem
  • read files, usually the current directory as .
  • write files, usually the build output as _build
• read and define env variables
• declare jobs like dependencies update, build & test

There is the one of this demo app:

package todoapp

import (
    "dagger.io/dagger"

    "dagger.io/dagger/core"
    "universe.dagger.io/netlify"
    "universe.dagger.io/yarn"
)

dagger.#Plan & {
    actions: {
        // Load the todoapp source code 
        source: core.#Source & {
            path: "."
            exclude: [
                "node_modules",
                "build",
                "*.cue",
                "*.md",
                ".git",
            ]
        }

        // Build todoapp
        build: yarn.#Script & {
            name:   "build"
            source: actions.source.output
        }

        // Test todoapp
        test: yarn.#Script & {
            name:   "test"
            source: actions.source.output

            // This environment variable disables watch mode
            // in "react-scripts test".
            // We don't set it for all commands, because it causes warnings
            // to be treated as fatal errors.
            // See https://create-react-app.dev/docs/advanced-configuration
            container: env: CI: "true"
        }

        // Deploy todoapp
        deploy: netlify.#Deploy & {
            contents: actions.build.output
            site:     string | *"dagger-todoapp"
        }
    }
}

You can see that three different jobs are defined:

• a build one which will compile the react app using yarn
• a test one
• a deploy one

To execute locally one of those, you can execute: dagger-cue do <job_name>.
So to build the project you can do dagger-cue do build.

⛏️ To go deeper

I hope to have given you a good overview of Dagger with this first chapter or at least to have made you want to go further with it.
Getting started with Dagger and CUE is not necessarily easy.
It's still an extremely young product (we are currently in version v0.2.36) and there is, therefore, still a lot of optimization work to be done as well as missing features.
Keep in mind that Dagger is an open-source project available on Github so don't hesitate to open issues or pull request if needed.

In the next (coming soon) chapter we will see how to use Dagger to build a deployment workflow for a Pulumi project and how to manage secrets.