Poor man's CI/CD with GitHub Webhooks

Background

In my 5th semester of college (mid 2021), I was in-charge of the Web Operations team at E-Cell IITM. Our responsibility was to make web applications for events, workshops and competitions. The tech stack was primarily Node.js with Express, React and MongoDB for the database. If you're a full-stack developer, you might know this as the MERN stack.

The environment was fast-paced. We were building a lot of things and breaking more of them in the process. It was like a startup. The highest number of users for an application was only 6.5k. And that's the reason we could experiment a lot of stuff. Following best practices was something we tried a lot to inculcate by adding tools like linters, formatters, pre-commit hooks, etc. but again, we were all freshers; making and breaking things and somewhere down the line we would just push something with just a commit message of fix stuff.

We have an in-premise server in the institute cluster for hosting the MERN applications. It is just an Ubuntu box with pm2 installed for running Node.js apps. A lot of our time was spent in deploying very, very minor changes. Fix-a-typo-in-a-static-page-level minor. The process was generally this:

1. Connect to institute's VPN if you're not in Insti's network
2. SSH into the server
3. git pull
4. npm run build if it's a React application
5. pm2 restart frontend-server

Easy and simple. Manual and boring. Perfect characteristics of something that needs to be automated. Automation is the mother of invention when it comes to software development and so I went looking for answers.

Self-hosted GitHub Actions

The first and the most obvious solution was to use GitHub Actions. I've never used them before and was pretty excited to learn and use it. My excitement was soon doubled when I got to know that you can self-host them. I was able to learn it and setup it for our backend Node.js monolith in a day. But something felt off.

1. It required installing a lot of stuff to a rather humble machine. It also demanded a lot of memory relative to what our applications were using. To give you an idea, our main backend service uses 105 MB at maximum whereas the runner used around 1 GB in average.
2. The action was such that it checked out the whole repository every time. The repository was, in a way, large.¹
3. All our applications stayed in the home directory ($HOME). I could not figure out how to checkout the repository in the same place where it was and got around it by adding post-job scripts.
4. We weren't looking at a lot of scale so this level of sophistication felt overkill.

All we wanted to do was to run a few scripts on a git push. Surely, there was a minimal way of doing this?

The hack which seemed like the solution

Webhooks are an elegant way to communicate from server to client. They are one of the simpler ways to implement event-driven architecture. Think of webhooks as a third-party server sending an HTTP POST request to your own server. This doesn't require you to keep asking that server for data or keep an open connection. GitHub offers Webhooks for a lot of repository and organisation level events.

Here's how they work:

1. You provide a URL and a secret to GitHub. GitHub stores the hash of that secret.
2. You choose the events you want a webhook for.
3. Whenever that event happens, GitHub sends a JSON or XML payload via a HTTP POST request to the URL you provided in step 1. It also sends the signature which is derived from the secret you provided.
4. In your server, you check the signature with your original signature. If it's correct, you handle the payload however you want to.

The payload being talked about contains a lot of information related to the event.

And there was it. A simple HTTP POST containing relevant data. The idea was to commit and push with a commit message starting with deploy. The commit message, committer, files changed, etc. would be included in the payload for the push event. And here's how it panned out in our case:

1. Provide a URL like https://ecell.iitm.ac.in/api/gh-webhook
2. Choose to send the webhook only on push events.
3. When we receive a webhook, check if the commit message starts with deploy, the committer is one of the admins and some code is changed.
4. Run a simple script:

   git pull
   npm run build
   pm2 restart

Step by step instructions

Let's try to replicate this with an example repository and localhost as the server. Find the repository here: webhook-example.

Adding the webhook to the repository

1. Choose a repository you want to work on. You can fork the example repository. Make sure to rename .template.env to .env and change the secret string if you want to.
2. Go to Settings > Webhooks > Add webhook. Or just visit https://github.com/<username>/<repo-name>/settings/hooks/new.
3. We will start the server and expose it via a tunnel. I'm using cloudflared but you can use anything else if you want to.
4. As you can see, we have chosen application/json as the content type cause we want the payload in JSON. For the secret, you can put any random string but make sure you remember/note what you put there. For the example repository you also have to update your .env file. Details in the next section. > Note about SSL: We've only disabled it because it's just a demo. Enable this without fail in your real applications.

Listening to the webhook in your server

1. The index.js is the entry point for our server. It's a simple static site server in Node.js which serves a React build. We mount the webhook listener in the /webhook route. Notice how we used this route in the URL we provided to GitHub in the previous step.

   // Mount the webhook listener
   app.use('/webhook', webhookRouter);

1. In webhook.js, we first parse the .env file which contains our secret.

   dotenv.config();
   const SECRET = process.env.GH_WEBHOOK_SECRET;

1. Next we add 2 middlewares. The first middleware stores the raw request buffer. We do this because parsing damages the integrity of the signature.

   bodyParser.json({
    verify: (req, res, buf) => {
    req.rawBody = buf;
        },
   })

1. The second middleware verifies the signature sent with our secret that only we and GitHub knows. This way we know the request is actually a valid one from GitHub. This is why SSL is so important because without that you're exposing this request to MITM attacks.

   const body = Buffer.from(req.rawBody, 'utf8');
   const ghSign = req.get('x-hub-signature-256');
   const ourSign = 'sha256=' + crypto.createHmac('sha256', SECRET).update(body).digest('hex');
   if (ghSign !== ourSign) {
      throw new Error();
   }

1. And finally, we listen to it in the POST handler.

   webhookRouter.post('/', async (req, res) => {
        console.debug(req.body)
   }

At this point, if you push some changes, you can see the payload in your terminal where the server is running:

   {
       "ref": "refs/heads/master",
       "repository": {
           "name": "webhook-example",
           "full_name": "abhijit-hota/webhook-example"
       },
       "pusher": {
           "name": "abhijit-hota",
           "email": "abhihota025@gmail.com"
       },
       "head_commit": {
           "id": "6c386ca6d6f3df80ea7b0abd59c3b9a8c3983726",
           "message": "chore: add comments and increase legibility",
           "timestamp": "2022-10-05T22:19:21+05:30",
           "author": {
               "name": "abhijit-hota",
               "email": "abhihota025@gmail.com",
               "username": "abhijit-hota"
           },
           "added": [],
           "removed": [],
           "modified": [
               ".template.env",
               "index.js",
               "webhook.js"
           ]
       }
   }

I've removed a lot of stuff from the payload and have just shown the relevant ones. Now all that left is to run commands based on the data we get. More on this in the next section.

Building and executing commands

We're only going to see how to solve it for Linux but it shouldn't be that hard to build queries for Windows systems in a similar way, if you're using Windows at all for deployments.

1. We check for the commit message and the pusher. You can of course, choose your own logic here:

   const { head_commit: commit, pusher } = req.body;

   if (!commit.message.startsWith('deploy') || pusher.name !== 'abhijit-hota') {
       return res.send('Deployment skipped.');
   }

1. We create an array of commands to run. And we add to it as we go over the changes. The code is pretty self-explanatory and I've added comments too.

   const commands = ["cd "];
   const changes = [...commit.modified, ...commit.added, ...commit.removed];

   const requiresReinstall = changes.any((change) => change.startsWith('package'));
   if (requiresReinstall) {
       commands.push('npm install');
   }

   const hasFrontendChanges = changes.any((change) => change.startsWith('frontend'));
   if (hasFrontendChanges) {
       commands.push('cd frontend'); 
       const requiresReinstallInFrontend = changes.any((change) => change.startsWith('frontend/package'));
       if (requiresReinstallInFrontend) {
           commands.push('npm install');
       }
       commands.push('npm run build');
   }
   commands.push('npm start');

1. Join the whole command string with && and run it.

   const cmd = commands.join(' && ');
   console.debug("Command to run:", cmd);
   exec(cmd); // exec from child_process

1. Let's put it to test! We commit with the message deploy: test and we see the message:

   Command to run: cd /home/kreta/work/misc/webhook-example && git pull --all && git checkout main && npm start

npm start in the package.json starts the server in localhost:8080. And sure enough:

It's a hack at the end of the day

1. The performance cost is way less than GH Action runners. We run a separate Node.js server for this which only takes up like 50 MB of RAM. I'm sure we can remove all the fluff from there and make it leaner but it works for now.
2. For multi-repo systems, you can create organisation-level webhooks and change the command according to the repository field in the payload.
3. "But Abhijit, there's no CI at all!". Just add an npm test somewhere and log the reports, email, etc. if you care about it.

Addressing the obvious thoughts in your head, yes, it can break and there's no way of knowing if the command is faulty. But you can setup logging, alerts etc. It's your system. It's code. It cannot get more IaC than this. I set this up on the actual server almost 2 years ago and it hasn't been changed ever since. As you can see, this is a setup once and forget kinda system.

But if your infra is changing a lot of times and requires really good monitoring, please don't use this. This is a hack and I'm not very fond of hacks. Except for this one. This one, I'm proud of.

First principles are good

The simplest solutions are the best ones. If we go backwards from a solution to the problem, we can find better ways of thinking. Some hacks and workarounds can sometimes go against the practice of using the right tools and can feel like reinventing the wheel. But if the trade-off is worth it, maybe it is the best solution for it after all. Remember to scale with common sense.

1. The repository was larger than needed because it had a whole Bootstrap theme template. ↩