DEV Community

Cover image for Setting up a self-hosted CI/CD pipeline
Jean-Yves Pellé
Jean-Yves Pellé

Posted on • Originally published at pelle.link on

Setting up a self-hosted CI/CD pipeline

As a reminder, a CI/CD pipeline (or Continuous Integration/Continuous Deployment pipeline) is a series of actions to be performed in order to consistently and reliably distribute a new version of a software.

In this article, we will see how to set up a minimalistic CI/CD pipeline (automatically triggering builds and deployments upon push events to a git repository) that is self-hosted and can be adapted according to your needs.

Prerequisites

We will need:

  • A build server that we will name buildsrv:
    • Running Linux
    • Accessible via SSH
    • With make and git tools
  • A git forge hosting the sources of our application. We will use GitHub here, but any other forge capable of calling webhooks upon receiving push events on its repositories will also do the job.
  • An instance of Ctfreak with an administrator account (the Free Edition will suffice). We will use https://demo.ctfreak.com here.

The sources

The sources of our application can be summarized in this Makefile:

build:
    @echo "Build my awesome app"
deploy: build
    @echo "Deploy my awesome app"

Enter fullscreen mode Exit fullscreen mode

You can find them in this git repository, with the master branch corresponding to the latest deployed version and the dev branch to the version currently under development.

Objectives

Let’s review the objectives to be achieved by setting up our CI/CD pipeline:

  • For the Continuous Integration part:
    • Compile the sources (via the make command) every time a developer pushes to the dev branch.
    • Receive a notification in case of failure.
  • For the Continuous Deployment part:
    • Compile the sources and deploy (via the make deploy command) every time a developer pushes to the master branch.
    • Receive a notification in case of success or failure.

To do this, Ctfreak will act as a gateway between the git forge and the build server.

That is, every time a push is received by the forge:

  • It will call a webhook defined in Ctfreak.
  • Ctfreak will then connect via SSH to the build server to execute the appropriate make command, then send a notification.

Build server configuration

Connect to your build server and run the following commands:

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.master
> Cloning into 'myawesomeapp.master'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 3.59 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.dev
> Cloning into 'myawesomeapp.dev'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 7.19 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ cd myawesomeapp.dev

ctfreakdemo@buildsrv:~/myawesomeapp.dev$ git checkout dev
> Branch 'dev' set up to track remote branch 'dev' from 'origin'.
> Switched to a new branch 'dev'
Enter fullscreen mode Exit fullscreen mode

As you can see, we have cloned the git repo 2 times (1 per branch), which will allow us to avoid potential conflicts if the builds of the master and dev branches were to run concurrently.

Ctfreak Configuration

Log in to your Ctfreak instance with an admin account.

Adding the build server

Start by adding the SSH key to connect to the build server via SSH CredentialNew SSH Credential:

Adding the SSH key

Then add the server itself via NodesInternal NodesNew node:

Adding the server

Make sure that the SSH Key for buildsrv key is selected as the Credential (Ctfreak will use it to connect to the server), then validate to create the buildsrv node.

Creating the CI/CD project

We are going to create a dedicated project to gather all the tasks related to our CI/CD pipeline.

Go to ProjectsNew Project

Project creation

Validate to create the project.

Adding a notifier

Go to ProjectsCI/CDNotifiersNew notifier

Selecting the notifier type

Choose Discord as the notifier type (for example).

Editing the notifier

Specify the discord webhook URL to call and validate.

List of notifiers

A test notification can be sent by clicking on the send button (the one shaped like a paper airplane).

Creating the build task

Go to ProjectsCI/CDNew task

Selecting the task type

Choose Bash Script as the task type.

Continuous integration task

As you may have guessed, this task will compile the dev branch on the build server and send a notification in case of failure.

The smart chaining multiple execution policy selected here will avoid launching unnecessary builds.

Let’s take an example: you have a build that lasts half an hour and 3 pushes are sent to the git repository at 5-minute intervals.

At the first push, the build is launched immediately.

At the 2nd push, a 2nd build is put on hold since the first one is not yet finished.

At the 3rd push, we don’t plan to do a 3rd build because the 2nd one is still on hold.

Creating the deployment task

To save time, create this task from the build task via ProjectsCI/CDBuid myawesomeapp / devDuplicate.

Continuous deployment task

Making the following modifications:

  • We want to receive a notification in case of success or failure.
  • The script content is updated to deploy from the master branch.

Adding webhooks

Now that our build and deployment tasks are created, we will add an incoming webhook for each one that can be called by our git forge.

Let’s start with the deployment task by going to ProjectsCI/CDBuid myawesomeapp / devIncoming webhooksNew webhook.

Selecting webhook type

Choose GitHub as the webhook type.

When GitHub calls our webhook following a push, it will do so regardless of the branch involved, but our task should only be executed when this push concerns the dev branch.

To solve this issue, we add an execution condition that will apply to the JSON payload sent by GitHub. The task will only be executed if the condition is met (i.e. the payload does indeed refer to the dev branch).

Creating the dev webhook 1

Define a secret if necessary, which will allow Ctfreak to validate that calls to the webhook come from GitHub, and validate.

Once the webhook is created, we retrieve its URL for GitHub configuration.

Creating the dev webhook 2

In your GitHub repository settings, add the webhook we just created, specifying:

  • Its URL and secret
  • That the expected format is JSON
  • That the webhook should be called ONLY for push events

Configuring the dev webhook in github

Proceed in the same way (create the webhook, reference it in GitHub) for the deployment task (be sure to reference the master branch in the execution condition):

Creating the master webhook

Pipeline testing

Now that everything is set up, let’s see what happens when we do a git push on the dev branch:

After calling the GitPushDev webhook, the build task was executed successfully.

By clicking on the eye-shaped icon, we can see the execution logs.

Let’s do the same test with a git push on the master branch:

This time, the GitPushMaster webhook was called as expected.

And the added bonus here is the receipt of a Discord notification confirming that the deployment was successful.

Conclusion

You now have an operational CI/CD pipeline that is flexible enough to handle even the most complex build workflows.

By opting for this self-hosted solution and placing the core of the build process in a shell script rather than in a continuous integration software like Jenkins, GitHub Actions, etc., you will benefit from several advantages:

  • You will not be constrained by the limitations of the latter (older version of Git, certain specificities of your build not supported, limit of the number of daily builds reached, etc.).
  • If your build server is unavailable, you can still perform an emergency deployment from a developer’s workstation (trust me, it can be useful).

Top comments (0)