Running your CI builds without the server

Perhaps the one feature that sets Djinn CI out from other CI platforms is the fact that is has an offline runner. The offline runner allows for CI builds to be run without having to send them to the server. There are some limitations around this, of course, but it provides a useful mechanism for sanity checking build manifests, testing custom images, and for building software without the need for a CI server.

Using the offline runner

We've given a high-level overview of the offline runner and how it works. But let's actually use see it in action to see how it is used. As part of this demonstration, we're going to be building the djinn-ci/imgsrv image server. This is what's used for serving the base QEMU images at https://images.djinn-ci.com.

Note: The following demonstration assumes all commands will be executed on a Linux distribution.

Installing the runner

First, the offline runner needs to be installed. We can grab the binary for this right from the Djinn CI namespace here.

Once downloaded, we can move the binary into our PATH,

$ sudo mv djinn /usr/local/bin

Next, we need to configure the offline runner, and the drivers it will use. So let's create the necessary configuration directory,

$ mkdir ~/.config/djinn
$ touch ~/.config/djinn/driver.conf

Now we can configure the qemu driver for the offline runner to use,

# ~/.config/djinn/driver.conf
driver qemu {
    disks  "/var/lib/djinn/images"
    cpus   2
    memory 2048
}

More information about how to configure drivers can be found in the documentation.

Finally, let's create the necessary sub-directories for the images we want to use, and download a base image for our build,

$ mkdir -p /var/lib/djinn/images/qemu/x86_64

For building the imgsrv program we would need the debian/oldstable image. We can download this from the image server previously mentioned.

With the image downloaded, we can create a sub-directory in our images directory to store it,

$ mkdir /var/lib/djinn/images/qemu/x86_64/debian
$ mv oldstable /var/lib/djinn/images/qemu/x86_64/debian

Building the image server

With the offline runner setup, let's clone the djinn-ci/imgrsrv repository, and build it,

$ git clone https://github.com/djinn-ci/imgsrv

Once cloned, all we need to do to execute the build locally, is change into the repository, and execute djinn,

$ cd imgsrv
$ djinn

When it starts running, we should see some output like this,

Running with Driver qemu...
Creating machine with arch x86_64...
Booting machine with image debian/oldstable...
Established SSH connection to machine as root...

Once done, the djinn-imgsrv artifact will be collected into the root of the repository.

Now, this begs the question. Why build it using the offline runner, when you could built it normally using the make.sh script in the repository? For the following reasons:

• The build manifest handles downloading and building dependencies needed
• The build manifest is executed on the Linux distribution the server is deployed to

The second point, is perhaps the most important. The djinn-imgsrv is a dynamically linked binary. This is because the program makes use of SQLite as the in-memory database to keep track of the images to serve. Because of this Cgo is used to interface Go with the C code in the sqlite library. So, a binary that is dynamically linked is built.

So, if the typical approach of building the program were to be taken, then a binary linking against the distribution's version of the libc will be produced. In some scenarios, this can be no good, as you may be developing on a different distribution than what you would be deploying to.

In this case, I develop on Arch, and deploy the image server to Debian 10. So, if I were to build this typically, then the binary may not execute on Debian. To remediate this, the offline runner is used instead, whereby I know it will be built against the distribution onto which it is deployed.

This is perhaps one of the benefits of the Djinn CI platform as a whole, and not just the offline runner itself. Which is the ability to utilise the qemu driver in builds, to build and test software on various Linux distributions.

More information

To learn more about Djinn CI, visit the about page which gives an overview of the features of the platform, and be sure to checkout the user documentation too.