Deploy your Pulumi project using Docker and Dagger.io

🕰️ In the previous episode

In the first part of this Dagger's series, I showed you what's Dagger.io, what's the features of it and it's benefits against others ci/cd solutions and finally the very basis of Dagger.

With this chapter, I will show you how we can overpower the CI/CD of any Pulumi project using Dagger.

🧰 Pulumi - An amazing IaC tool

First of all, I think that some of you may doesn't know what is Pulumi or even IaC (Infrastructure as Code) concept, so I will quickly present to you these two points.

Infrastructure as Code

Nowadays, IT extends to many areas, and so new needs are emerging leadingly to a necessity to adapt infrastructures in order to be able to support all of this.
They also have seen their prerequisites evolve given their multiplication and their increasingly large sizes. As a result, companies started wanting to automate and simplify their infrastructures.

To address this problem, Amazon unveiled in 2006 the concept of Infrastructure As Code (or IaC) allowing, on Amazon Web Services, the configuration of instances using computer code.
It was a revolution for infrastructure management and, although limited at the time, this method was quickly adopted by the market.
This event also coincides with the date of appearance of the DevOps movement in which it's part.

Most of the infrastructure as code tools are based on the use of descriptor files to organize the code which avoids duplication between environments. Some advanced tools support variability, the use of outputs and even deployment to several providers simultaneously.

There are three types of infrastructure as code:

• Imperative: resources (instances, networks, etc.) are declared via a list of instructions in a defined order to obtain an expected result.
• Functional: unlike the imperative mode, the order of the instructions does not matter. The resources are defined in such a way that their final configuration is as expected.
• Based on the environment: the resources are declared in such a way that their state and their final configuration are consistent with the rest of the environment.

The advantages of IaC compared to traditional management are numerous, such as: cost reduction, the possibility of versioning the infrastructure, the speed of deployment and execution, the ability to collaborate, etc.
It also allows complete automation, in fact, once the process is launched, there is no longer any need for human intervention. This advantage not only limits the risks due to human error and therefore increases reliability, but also allows teams to focus on projects and less on the deployment of applications.

Pulumi

Pulumi is an open-source IaC tool. It can be used to create, manage and deploy an infrastructure on many cloud provider like AWS, GCP, Scaleway, etc.
It haves few strengths compared to others IaC solutions like Terraform for example.

First of all, Pulumi support many programming languages like Go, TypeScript, Python, Java, F# and few others.
This is a great advantage for Pulumi and one of the reasons why we begin to use it at Camptocamp because the presence of a programming language like Go, rather than a configuration language like HCL (language used by Terraform) allows much more flexibility and adaptability.

Furthermore, Pulumi has new native providers like AWS, Google, and others in order to get new versions the same day they are released.
Additionally, Pulumi also supports Terraform providers to maintain compatibility with any infrastructure built using Terraform.

Another very interesting advantage is the support of what they call “Dynamic Providers” which allows to easily extend an existing provider with new types of personalized resources and that by directly programming new CRUD operations.
This can allow new resources to be added, for example, while adding complex migration or configuration logic.

Finally, there are still many advantages to Pulumi such as the ease of carrying out tests thanks to the native frameworks of the programming languages provided for this usage, the presence of aliases allowing a resource to be renamed while maintaining compatibility with the state of the infrastructure, better integrations with mainstream IDEs like VSCode, etc.

⚡ Supercharge your Pulumi project thanks to Dagger

Now that you know IaC, Pulumi and of course Dagger (if not, you can check the first part of this blog series), we will see how we can create CI/CD pipelines for any Pulumi project using Dagger and CUE and finally how can we run them.
For that, I will present to you the Dagger architecture we have built at Camptocamp, it was designed to be complete and reusable. It may be too complex for small projects but if you understand it you will normally be able to create your own !

Important note: Initially Dagger.io was developed using Cuelang, an amazing and modern declarative language. Cue was also the only way to do pipelines with it. However, Dagger's team have recently transformed Dagger to be language agnostic.
We now have different SDKs: Go (the main one), CUE, Node.js and Python.
As of today, I advice you to choose the Go SDK if you don't really know which one to take. For the CUE one, I only recommend it to you if:

• You like declarative language like YAML
• You know CUE or want to learn it

For the following chapters, I will present our implementation which is made in CUE. However, it will be easy to transpose it to other SDKs if you understand it.

📦 The Pulumi package

In order to make a reusable and powerful Dagger project, we decided to create a "Pulumi" package which embeds our objects definitions like:

• the Docker image
• the container definition
• the command object

As an exemple there is the command object definition:

// pulumi/command.cue
package pulumi

import (
    "dagger.io/dagger"
    "universe.dagger.io/bash"
    "universe.dagger.io/docker"
)

#Command: self = {
    image?: docker.#Image
    name:   string
    args: [...string]
    env: [string]: string
    source: dagger.#FS
    input?: dagger.#FS

    _forceRun: bash.#RunSimple & {
        script: contents: "true"
        always: true
    }

    _container: #Container & {
        if self.image != _|_ {
            image: self.image
        }

        source: self.source

        command: {
            name: self.name
            args: self.args
        }

        env: self.env & {
            FORCE_RUN_HACK: "\(_forceRun.success)"
        }

        if self.input != _|_ {
            mounts: input: {
                type:     "fs"
                dest:     "/input"
                contents: self.input
            }
        }

        export: directories: "/output": _
    }

    output: _container.export.directories."/output"
}

At the top of this file, you can see the package definition and all our imports. universe.dagger.io is a community repository where we can find pre-made package for Docker, bash, alpine, etc.
Just after that, we start defining our #Command object by adding public fields like command's name, args, custom Docker image (which is optional due to ? character, etc.
We also define our unexported fields (which aren't accessible from outside the package) like the container definition which will run our command (the container object is defined in the container.cue file in the same repository).

This is one of the few definitions that we have in this package, I will not show you directly all of them since it's really specific to our implementation and it's not very relevant to explain how it's working.
However, you can retrieve all of our source code here: https://github.com/camptocamp/dagger-pulumi

So one last file from this pulumi package that we will see is the main.cue one. It's where we defined all the Pulumi commands that we will use (using the #Command object seen just before).

// pulumi/main.cue
[...]
#Preview: self = {
    stack: string
    diff:  bool | *false

    #Command & {
        name: "preview"

        args: [
            "--stack",
            stack,
            "--save-plan",
            "/output/plan.json",

            if diff {
                "--diff"
            },
        ]
    }

    _file: core.#ReadFile & {
        input: self.output
        path:  "plan.json"
    }

    plan: _file.contents
}

#Update: {
    stack: string
    diff:  bool | *false
    plan:  string

    _file: core.#WriteFile & {
        input:    dagger.#Scratch
        path:     "plan.json"
        contents: planHere, you defined 
    }

    #Command & {
        name: "update"

        args: [
            "--stack",
            stack,
                        [...]
            if diff {
                "--diff"
            },

            "--skip-preview",
        ]

        input: _file.output
    }
}
[...]

I voluntary remove some parts of the file and leave only the definition for Pulumi preview and update commands in order to keep it simple.
In the Pulumi context, preview command is used to compare the codebase which defines the desired infrastructure state with the actual one and preview all the potential changes if we apply the configuration.
The update one, as it name says, update the deployed infrastructure to match the desired state.

Using these definitions, we finally defined a #Pulumi object in the main.cue from the parent ci package.
It's composed by few attributes used to set our project environment like, for example, the Pulumi stack (workspace), the env variables, the authorization to run destructive actions through this CI, etc.
We also have a list of all available commands construct using all #Command objects defined earlier.

Finally, the last step before we can be able to run our pipelines is to create the Dagger's plan.
This is the most important part where all jobs are defined but it's really specific to the project for which it's built.
I will present to you a plan from one of our projects where we used the pulumi package that I show you just before.

import (
    "github.com/camptocamp/pulumi-aws-schweizmobil/ci"

    "dagger.io/dagger"
)

dagger.#Plan & {
    client: {
        env: {
            PULUMI_GOMAXPROCS?: string
            PULUMI_GOMEMLIMIT?: string
        }

        filesystem: {
            sources: read: {
                path:     "."
                contents: dagger.#FS

                exclude: [
                    ".*",
                ]
            }

            // TODO
            // Simplify once https://github.com/dagger/dagger/issues/2909 is fixed
            plan: read: {
                path:     "plan.json"
                contents: string
            }

            planWrite: write: {
                path:     "plan.json"
                contents: actions.preview.plan
            }
        }
    }

    #Pulumi: ci.#Pulumi & {
        env: {
            if client.env.PULUMI_GOMAXPROCS != _|_ {
                GOMAXPROCS: client.env.PULUMI_GOMAXPROCS
            }

            if client.env.PULUMI_GOMEMLIMIT != _|_ {
                GOMEMLIMIT: client.env.PULUMI_GOMEMLIMIT
            }
        }

        source: client.filesystem.sources.read.contents
        stack:  string
        diff:   bool
        update: plan: client.filesystem.plan.read.contents
        enableDestructiveActions: bool
    }

    actions: {
        #Pulumi.commands
    }
}

Firstly, you can see at the begin of this file the import of the homemade ci package (as a reminder, it is fully available here)
After, we defined the dagger.#Plan object which is composed by few parts:

• the client config: this is where we will be able to set some runtime elements like: env variables, filesystems with read/write permissions, the sources location...
• the actions: this is all the possible jobs that you will be able to run using Dagger CLI or in your CI/CD environment, in our case we just use local variable #Pulumi.commands which is built using the #Pulumi object defined in the ci package.

🚀 Launch Dagger's jobs

Locally using CLI

As I told in the previous part of this blog series, Dagger can run any jobs locally thanks to his Dockerize design.
You must have installed the Dagger's CLI for Cue SDK (check the first part of this series or the official documentation if it's not already done).

Once ready, you can open your favorite terminal client located inside your project directory (composed by your Pulumi project and your Dagger plan) and run:

• dagger --plan=ci.cue do --with '#Pulumi: stack: "<your_stack>"' preview -> it will run a Pulumi preview on the desired stack
• dagger --plan=ci.cue do --with '#Pulumi: stack: "<your_stack>", #Pulumi: diff: true, #Pulumi: enableDestructiveActions: true' update -> it will run a Pulumi update on the desired stack

Remotely using a CI/CD environment

With Dagger you can also run your pipelines on every CI/CD environments!
As a simple example, you can easily run these jobs on Github Actions:

name: pulumi-project

on:
  push:
    branches:
      - main

jobs:
  dagger:
    runs-on: ubuntu-latest
    steps:
      - name: Clone repository
        uses: actions/checkout@v2

      - name: Install Dagger Engine
        run: |
            cd /usr/local
            wget -O - https://dl.dagger.io/dagger/install.sh | sudo sh
            cd -

      - name: Run Pulumi update using Dagger
        run: |
            dagger-cue --plan=ci.cue do update --log-format plain        

Note than the --log-format plain flag is useful in order to have well formatted logs in the Github Action output.
With this action, a pulumi update will be launched at every push on the main branch using Dagger.

🔚 Conclusion

If you have made it this far I thank you very much and I hope I have succeeded to give you a good overview of what is Dagger and how is it possible to exploit it in order to improve its practices in terms of CI/CD.

From my point of view, Dagger is a very promising solution. Right now we can still feel Dagger's young age, indeed, it's still not perfect.
In my opinion, each SDK should continue to develop and harmonize with each other. I also think that Dagger's execution performance and cache management should be improved (I'm basing myself on the 0.2 version of the engine since 0.3 is not yet available with the Cuelang SDK so maybe that's already better!).
Nevertheless, the advantages of Dagger are already essential for me, such as the possibility of running its pipelines locally or even being able to run them remotely on a remote VM using local sources.

To be followed very closely! 😉