Automatic deployment from Github to your server with no third party app

When it comes to setting up a deployment pipeline I believe the number of solutions in the wild are countless. If we are on AWS we can use Code Deploy, Heroku, Zeit Now and Netlify provide their own solutions and of course one of the popular ones is to delegate the task to your CI/CD server (Travis, Circle CI etc) to handle it. If you are using Docker containers the best orchestrator tools to use are AWS ECS, Swarm and Kubernetes. Most probably if you are working with a bigger team you have Dev-Ops team mates to handle it and you might take your infrastructure as granted! 😐 However, if like me you joined a startup and the deployment process was manual (SSH to server, git pull, etc) and you wrote some bash scripts to do it for them you might want to embrace your inner nerd 🤓 and level up your deployment game. In this case, running your bash scripts automatically upon merging a Pull Request on Github would make everyone happy and that's not rocket science so let's do it!

Goal

To automatically deploy your code after merging pull requests to branches dev and master.

Things we use

• Node JS
• Bash Scripts
• Github Webhooks
• SSH command line

Getting Started

Let's say we have two versions of our website that we need to automatically deploy. One is called stage and represents the latest merged commits. These changes are often buggy and not reliable so we want only the internal team have access to it. "stage" represents our dev branch in Github. The second version of the website is called "prod" and will rerpresent the master branch in Github. This branch is (hopefully) stable and has passed the QA team and is considered safe for end-users. This version is the one that everyone outside the company know as our website URL.

Step 1: Clone your Git repositories

If you already don't have your github repositories cloned in the server you have to do it.

Make two directories called : prod and stage.

mkdir live stage
git clone git@github.com:p0o/your_repo.git
cp -a your_repo/. ./prod/your_repo
cp -a your_repo/. ./stage/your_repo
rm -rf ./your_repo

Make sure to add the extra . after your_repo, this a special cp syntax that allows copying hidden files and folders inside your folder as well (we need it to copy .git folder as well).

Wild Assumption: I assume you are familiar with the basics of managing a server and your can run your websites in the URL you want with a proper SSL certificate. I use Nginx for this purpose but I'm not gonna explain these steps in the post. You might search if you are not sure.

Step 2: Make bash scripts

We need to have two bash scripts to handle the deployment for each of these cases. If you need to build your files and Let's create a directory in our server's home directory and start from there. I call this directory scripts:

cd ~/
mkdir scripts
cd scripts

Okay let's proceed with making the bash files:

touch ./deploy_stage
touch ./deploy_prod

Give them execution permission:

chmod +x ./deploy_stage
chmod +x ./deploy_prod

(Thanks to darksmile92 for pointing this out)

I will put the sample code for one of them, the other one is just a different folder and might have different environment variables according to your project dependencies.

#!/bin/bash
echo "Deploying stage your_repo"

cd ~/stage/your_repo \
&& git checkout dev \
&& git pull \
&& npm i \
&& npm run build \
&& (pm2 stop your_repo_stage || true) \
&& echo 'Installing:  done.' \
&& (pm2 delete your_repo_stage || true) \
&& NODE_ENV=development pm2 --name your_repo_stage start npm -- start \
&& echo "your_repo deployed successfully"

This bash script will basically fetch the latest code from github, install dependencies, build the script (if required) and run it using PM2. If you are not familiar, PM2 is a very useful process management tool and you can easily install it using NPM.

Also it's good to mention that I chained my whole process with logical AND (&&) because I wan to exit the execution in case one of the processes failed.

Step 3: Write code to handle the webhook events

In order to get notified anytime something happens in Github we have to subscribe to their Webhook API which essentially means giving some URLs to github so they would send some information to it. These URLs have to be public and they can run scripts that would lead to deployment of your code so having them accessible to anyone except Github servers would have serious security implications (e.g Denial Of Service attack).

Github is using a SH1 HMAC signature to verify the JSON object it's sending you. We will have this signature hash in the X-Hub-Signature value of header. Since taking care of all this is a bit complicated, we can use github-webhook-handler package which is created exactly for the same purpose.

We also need to run our bash script files from node. We can do it using native functions but I prefer to use shellJs for the sake of simplicity.

Okay enough ranting, here is the code you need:

const http = require('http');
const createHandler = require('github-webhook-handler');
const shell = require('shelljs');

// We avoid to hardcode the secret in the code, you should provide it with an ENV variable before running this script
const { MY_SECRET } = process.env;
// You might use the same script for multiple repositories, this is only one of them
const REPO_NAME = 'my_repo';
// port is default on 6767
const PORT = process.env.PORT || 6767;

var handler = createHandler({ path: '/', secret: MY_SECRET })

http.createServer(function (req, res) {
  handler(req, res, function (err) {
    res.statusCode = 404
    res.end('no such location')
  })
}).listen(PORT);

handler.on('error', function (err) {
  console.error('Error:', err.message)
})

handler.on('pull_request', function (event) {
  const repository = event.payload.repository.name;
  const action = event.payload.action;

  console.log('Received a Pull Request for %s to %s', repository, action);
  // the action of closed on pull_request event means either it is merged or declined
  if (repository === REPO_NAME && action === 'closed') {
    // we should deploy now
    shell.cd('..');
    shell.exec('~/scripts/deploy_stage');
  }
});

Save it in a folder somewhere in your server and install the dependencies:

npm init
npm i github-webhook-handler shelljs --save

And then just run it with environment variable forever using PM2:

MY_SECRET=MyGithubWebhookSecret pm2 --name github-deployer start node -- ./index.js

That's all!

Step 4: Configure github webhook

Now we just need to go to Github and introduce our webhook to github. But pay attention that in the previous step we ran the webhook on the port 6767 with no HTTPS. So you need to setup nginx and give it a proper domain with HTTPS. You can just put it on a path in your main domain but explaining that process is not in the scope of this article. Furtunately there are multiple articles in the web for you to look for.

Go to the Setting tab of your repository and click on the Webhooks. In the right side of the page, click on the "Add Webhook" button.

Enter the URL you introduced in your Nginx for the Node JS app we ran. Let's say it is https://yourdomain.com/webhook

Choose application/json for the content-type and enter the secret we used to run our service with. In my example it was "MyGithubWebhookSecret" up there.

In the section "Which events would you like to trigger this webhook?" click on "Let me select individual events" and find Pull Requests and check it:

Make sure everything else is unchecked and click "Add Webhook" to save it. We are all set now 🦸

Step 5: Test and verify

Use PM2 to monitor the logs for the node js app we made just now. Enter:

pm2 log github_deployer

Now you can see if any changes happens! Go to your repository and change something in a new branch. Submit a pull request and merge it. You should see your bash script in the log would perform a deployment and after that your changes should be reflected to the website. If you had any error, you can see it here in the log and well... do something about it 😂

Conclusion

Even though I think the proposed solution in this article is fairly straightforward, it is not the best solution out there for this particular problem. Even my personal blog is using Zeit Now Github Integration to get deployed! However other solutions rely on third party apps and sometimes not accessible to certain teams according to their resources. In my case, the deployment scripts were already there, repositories were not using docker and I had very limited time to spend on this problem. Advance with it if you also happen to be in the same boat!

This article is originally published on my blog with the title Automatic deployment From Github to your server feel free to check it out for more posts! 👀
You can also find me on twitter => @p0oker