DEV Community

Sylvain Lesage
Sylvain Lesage

Posted on • Updated on

Using webhooks to update a self-hosted Jekyll blog

I host the blog of a community project on my personal server. It's a static website generated with Jekyll, and the contents are versionned in a git repository.

In this blog post, I explain how I made the blog automatically update on every new commit. It's the same idea as Github Pages, but for self-hosting.

TL;DR

  • create a script to clone the git repository, then build and deploy the website
  • publish a webhooks endpoint on the server
  • configure the GitHub repository to send a webhook on every new commit

Deploy script

The website is hosted on a Debian server, and the files are served from the /var/www/atlas.tecnologia.bo/ directory. To update it from command line, first install the Jekyll requirements, then create and launch the bash script:

$ /opt/deploy_atlas.tecnologia.bo.sh

that contains:

#!/usr/bin/env bash

# Clone git repository
rm -rf /tmp/atlas.tecnologia.bo
git clone https://github.com/RipeAtlasBolivia/atlas.tecnologia.bo.git /tmp/atlas.tecnologia.bo

# Generate blog
cd /tmp/atlas.tecnologia.bo
bundle install
bundle exec jekyll build --source /tmp/atlas.tecnologia.bo/ --destination /var/www/atlas.tecnologia.bo/

# Clean
rm -rf /tmp/atlas.tecnologia.bo

Webhook daemon

Instead of manually launching the script, let a webhook daemon do the job every time it receives an HTTP request on an endpoint.

First install the webhook package:

$ sudo apt install webhook

Verify the service is running:

$ sudo service webhook status
● webhook.service - Small server for creating HTTP endpoints (hooks)
   Loaded: loaded (/etc/systemd/system/webhook.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-08-12 08:25:24 UTC; 2 weeks 0 days ago
     Docs: https://github.com/adnanh/webhook/
 Main PID: 419 (webhook)
    Tasks: 6 (limit: 4697)
   Memory: 4.8M
   CGroup: /system.slice/webhook.service
           └─419 /usr/bin/webhook -verbose -hooks /etc/webhook.conf

We configure the webhook daemon, creating the file /etc/webhook.conf with the following content (see the webhook documentation for more details):

[
  {
    "id": "atlas",
    "execute-command": "/opt/deploy_atlas.tecnologia.bo.sh",
    "command-working-directory": "/tmp/",
    "trigger-rule": {
      "match": {
        "type": "payload-hash-sha1",
        "secret": "xxxxxxxxx",
        "parameter": {
          "source": "header",
          "name": "X-Hub-Signature"
        }
      }
    }
  }
]

This creates a new hook called atlas, that will be triggered when a GET or POST request is received on http://localhost:9000/hooks/atlas. When triggered, it will check that a valid X-Hub-Signature HTTP header has been provided, and then launch the /opt/deploy_atlas.tecnologia.bo.sh script from the /tmp working directory.

Try it:

  • compute the X-Hub-Signature corresponding to the xxxxxxxxx secret
$ echo -n "" | openssl sha1 -hmac "xxxxxxxxx"
(stdin)= d46d87941e6f285be78ff0f1c8ea32620577b9ef
  • request the endpoint:
$ curl -X POST -H "X-Hub-Signature: sha1=d46d87941e6f285be78ff0f1c8ea32620577b9ef" -v http://localhost:9000/hooks/atlas
...
> POST /hooks/atlas HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.64.0
> Accept: */*
> X-Hub-Signature: sha1=d46d87941e6f285be78ff0f1c8ea32620577b9ef
>
< HTTP/1.1 200 OK
< Date: Mon, 26 Aug 2019 13:45:09 GMT
< Content-Length: 0
<
  • check the logs:
$ grep webhook /var/log/syslog
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 Started POST /hooks/atlas
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] incoming HTTP request from [::1]:58800
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] atlas got matched
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] atlas hook triggered successfully
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 Completed 200 OK in 670.302µs
Aug 26 14:06:40 webhook[10552]: [webhook] 2019/08/26 14:06:40 [d2ae30] executing /opt/deploy_atlas.tecnologia.bo.sh (/opt/deploy_atlas.tecnologia.bo.sh) with arguments ["/opt/deploy_atlas.tecnologia.bo.sh"] and environment [] using /tmp/ as cwd
Aug 26 14:06:46 webhook[10552]: [webhook] 2019/08/26 14:06:46 [d2ae30] command output: Cloning into '/tmp/atlas.tecnologia.bo'...
...
Aug 26 14:06:46 webhook[10552]: Bundle complete! 4 Gemfile dependencies, 24 gems now installed.
Aug 26 14:06:46 webhook[10552]: Use `bundle info [gemname]` to see where a bundled gem is installed.
Aug 26 14:06:46 webhook[10552]: Configuration file: /tmp/atlas.tecnologia.bo/_config.yml
Aug 26 14:06:46 webhook[10552]:             Source: /tmp/atlas.tecnologia.bo/
Aug 26 14:06:46 webhook[10552]:        Destination: /var/www/atlas.tecnologia.bo/
Aug 26 14:06:46 webhook[10552]:  Incremental build: disabled. Enable with --incremental
Aug 26 14:06:46 webhook[10552]:       Generating...
Aug 26 14:06:46 webhook[10552]:        Jekyll Feed: Generating feed for posts
Aug 26 14:06:46 webhook[10552]:                     done in 0.45 seconds.
Aug 26 14:06:46 webhook[10552]:  Auto-regeneration: disabled. Use --watch to enable.
Aug 26 14:06:46 webhook[10552]: [webhook] 2019/08/26 14:06:46 [d2ae30] finished handling atlas

Note: the 200 HTTP code only means the hook has been found for the atlas identifier and the X-Hub-Signature is valid. It says nothing about the deploy script success or failure.

In order to publish the endpoint from a Apache webserver, configure a reverse proxy in your Apache configuration:

ProxyPass /webhook/ http://localhost:9000/hooks/
ProxyPassReverse /webhook/ http://localhost:9000/hooks/

GitHub configuration

Finally, in order to trigger a webhook on every new push, go to the GitHub project settings, then "Webhooks", and create a new webhook with the following parameters:

  • Payload URL: https://mydomain/webhook/atlas
  • Content type: application/json
  • Secret: xxxxxxxxx
  • Which events would you like to trigger this webhook?: Just the push event.

Alternative configuration with GitLab

If your git repository is hosted on a GitLab instance, you must change the headers check in the /etc/webhook.conf file:

[
  {
    "id": "atlas",
    "execute-command": "/opt/deploy_atlas.tecnologia.bo.sh",
    "command-working-directory": "/tmp/",
    "trigger-rule": {
      "match": {
        "type": "value",
        "value": "xxxxxxxxx",
        "parameter": {
          "source": "header",
          "name": "X-Gitlab-Token"
        }
      }
    }
  }
]

Note: the X-Gitlab-Token header provided in the request will contain the value xxxxxxxxx, not its HMAC as it occurs with GitHub. So, to simulate the request, adapt the curl command to:

$ curl -X POST -H "X-Gitlab-Token: xxxxxxxxx" -v http://localhost:9000/hooks/atlas

Then, the GitLab project configuration is similar to GitHub: go to the project "Settings", "Integrations", and add a webhook with:

  • URL: https://mydomain/webhook/atlas
  • Secret Token: xxxxxxxxx
  • Trigger: [x] Push events

References

Top comments (7)

Collapse
 
elmor3no profile image
elMor3no

Another question ;)

How im able to use the same webhook services for diferent request

Im try something like:

[
{
"id": "atlas",
"execute-command": "/scripts/myscript1.sh",
"command-working-directory": "/tmp/",
"trigger-rule": {
"match": {
"type": "value",
"value": "secrect1",
"parameter": {
"source": "header",
"name": "X-Gitlab-Token",
}
}
}
}
]
[
{
"id": "atlas2",
"execute-command": "/scripts/myscript1.sh",
"command-working-directory": "/tmp/",
"trigger-rule": {
"match": {
"type": "value",
"value": "secrect2",
"parameter": {
"source": "header",
"name": "X-Gitlab-Token",
}
}
}
}

]

Im try in diferent ways but all the time i get:

[webhook] 2019/12/17 00:06:22 Started POST /hooks/atlas2
[webhook] 2019/12/17 00:06:22 Completed 404 Not Found in 30.368µs
[webhook] 2019/12/17 00:06:28 Started POST /hooks/atlas
[webhook] 2019/12/17 00:06:28 Completed 404 Not Found in 34.086µs

If i use only one all works fine:

[
{
"id": "atlas",
"execute-command": "/scripts/myscript1.sh",
"command-working-directory": "/tmp/",
"trigger-rule": {
"match": {
"type": "value",
"value": "secrect1",
"parameter": {
"source": "header",
"name": "X-Gitlab-Token",
}
}
}
}
]
[webhook] 2019/12/17 00:02:37 Started POST /hooks/atlas
[webhook] 2019/12/17 00:02:37 Completed 200 OK in 164.625µs

Collapse
 
severo profile image
Sylvain Lesage

I think it's because of your configuration JSON format: instead of

[
  {...1}
]
[
  {...2}
]

you should have

[
  {...1},
  {...2}
]

You could use a linter like jsonlint.com/?json= or prettier.io/ to check your configuration file.

Collapse
 
elmor3no profile image
elMor3no

Thanks....

Thats solve the issue..

Collapse
 
hrvoj3e profile image
hrvoj3e

Nice. Thanks for this.

But, is this secure?
The hook scripts in execute-command run as root when using weebhook as a service (systemd).

Collapse
 
severo profile image
Sylvain Lesage

Sure, you might want to edit /etc/systemd/system/webhook.service to specify the user, for example a dedicated webhook user created for these tasks only:

[Service]
User=webhook
Collapse
 
elmor3no profile image
elMor3no

Nice...

How i can debug the execution of the command...

Im not able to check why anything happends evens when i get a access sucessful...

Collapse
 
severo profile image
Sylvain Lesage

You can access the logs produced by the command with:

grep webhook /var/log/syslog

I updated the blog post to add this information.

Note that you need access to the server to see these logs. From a client point of view, you will only see a 200 HTTP code if the command has been launched successfully, but this doesn't mean the command itself has worked as expected.