Webhook Testing with Blackbird

#api #webhooks #softwaredevelopment #testing

If you've built integrations with 3rd party applications such as Stripe, PayPal, or GitHub chances are you've had to worry about handling webhooks. As a software developer it can be tricky to test webhook handling, because it relies on the 3rd party app to send a request to your API server - and in development your server is usually running at localhost.

In this post, I'll show how you can use Blackbird to quickly create a public URL for testing webhooks and show how it can also help debug webhook requests to your server.

Example: GitHub Commit Tracker

In this example, I'll be setting up a server that handles webhook requests from GitHub. Let's pretend I'm building a simple application that tracks anytime someone pushes a commit to a repo. You can follow along by cloning the Commit Tracker App written in Go.

git clone https://github.com/matty-v/commit-tracker
cd commit-tracker
go run cmd/commit-tracker/main.go

Test out the webhook endpoint by sending a POST request:

curl --request POST -d '{
  "ref":"refs/heads/main",
  "commits":
  [
    {
      "id":"abc123",
      "message":"Updated a file",
      "timestamp":"2024-09-18T00:00:00-06:00",
      "author":
      {
        "email":"test@example.com",
        "name":"Some Committer"
      }
    }
  ]
}' localhost/v1/webhook

It works locally, but the real test will be if it can handle an actual webhook from GitHub. This is where Blackbird comes in.

Setting up Blackbird

Follow the steps to install the Blackbird CLI on your laptop
Login to Blackbird: blackbird login - this will also provision a personal remote environment that you can use for testing

Testing the webhook with Blackbird code run

Instead of using go run I'll issue the following Blackbird command:

blackbird code run commit-tracker -d Dockerfile -c . -l 80

✔ image successfully built
✔ connected to Matt's organization blackbird environment 0
✔ temporary instance created
+----------------+------+--------+----------------+---------------------------------------------------------------------------------------------+------------+
| NAME           | TYPE | STATUS | APIKEY HEADERS | URL                                                                                         | CREATED BY |
+----------------+------+--------+----------------+---------------------------------------------------------------------------------------------+------------+
| commit-tracker | Code | Ready  |                | https://default-blackbird-matts-organization-5b603-0.blackbird-relay.a8r.io/commit-tracker/ | Matt Voget |
+----------------+------+--------+----------------+---------------------------------------------------------------------------------------------+------------+
✔ type <ctrl>-C to end...

Here is what Blackbird did behind the scenes:

Containerized my app and is running it in Docker on my local machine, exposing port 80
Generated a public URL in my personal remote environment
Actively routing traffic from the public URL directly to the local container

Now if I send that same curl command to test the webhook using the Blackbird provided URL I can see that my local server is responding the same way!

curl --request POST -d '{
  "ref":"refs/heads/main",
  "commits":
  [
    {
      "id":"abc123",
      "message":"Updated a file",
      "timestamp":"2024-09-18T00:00:00-06:00",
      "author":
      {
        "email":"test@example.com",
        "name":"Some Committer"
      }
    }
  ]
}' https://<my-env>.blackbird-relay.a8r.io/commit-tracker/v1/webhook

Be sure to replace <my-env> with the specific name of your environment as provided by Blackbird.

Let's go over to GitHub now to setup and test the webhook from the repo whose commits I wish to track. I can do this by navigating to Settings > Webhooks > add webhook

The Payload URL is the endpoint where GitHub will deliver the webhook request. Use the URL provided by Blackbird (e.g. https://<my-env>.blackbird-relay.a8r.io/commit-tracker/v1/webhook). Also be sure to check the "send me everything" option to get commit data.

With the webhook created, you can now test it out by making commits on the repo and see that your locally running code is handling it!

2024-09-18T21:02:30Z INF commit-tracker server is starting up
2024-09-18T21:02:30Z INF starting commit-tracker server, address: :80
2024-09-18T21:03:06Z INF POST /v1/webhook 200 729.608µs
[{{matt.voget@gmail.com Matt Voget} eec15... Update README.md 2024-09-18T12:05:32-06:00}]

When I'm done testing, I can ctrl-c on the command line to end the Blackbird code run session and kill the public URL.

Debugging webhooks with Blackbird

Let's pretend that I wasn't perfect and accidentally introduced a bug in my webhook handler code. I can use Blackbird to attach a debugger to my code to step through breakpoints as the webhook is being received.

To follow along, you can uncomment the //decoder.DisallowUnknownFields() line in handlers.go in the commit tracker code:

// HandleHandleWebhook handles parsing input to pass to the HandleWebhook operation and sends responses back to the client
func (h *APIHandler) HandleHandleWebhook(w http.ResponseWriter, r *http.Request) {
  var err error
  reqBody := GitHubWebhook{}
  decoder := json.NewDecoder(r.Body)
  decoder.DisallowUnknownFields()

This will make the handler reject webhook requests where the payload does not match the GitHubWebhook schema in the commit-tracker code. Perhaps you develop and test against a mocked GitHub client sending webhooks without all the payload fields. You may find your code working in dev and test, but failing when it hits production!

Here's how to use Blackbird to debug this problem:

blackbird code debug commit-tracker -d Dockerfile -c . -l 80

Once more, Blackbird containerized my code, but this time it followed instructions in the Dockerfile to build the image with debug symbols and expose port 2345 for a debugger to attach to.

In VSCode, I defined a launch.json debugger config with instructions to attach at that port:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Connect to server",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "port": 2345,
            "host": "127.0.0.1"
        }
    ]
}

Now that I have attached the debugger, I can once more deliver a webhook from GitHub to the same public URL: https://<my-env>.blackbird-relay.a8r.io/commit-tracker/v1/webhook

This time, I can hit breakpoints and step through my code to see what's happening

Not only can I test webhooks with Blackbird, but I can also debug any issues directly from my IDE.

Notes on Security

Unlike other tools out there, Blackbird does not use tunneling to open traffic from the public URL to your localhost, nor will it modify your local machine's host network or DNS. It instead will leverage Docker's local container networking to establish a secure connection to the remote host - think of it like a mini-VPN. Out of the box, this significantly decreases the attack surface area for malicious actors who get access to the public URL.

Also, Blackbird has an ability to pass a custom-defined API key to blackbird code run and blackbird code debug commands. This will protect your public URL from being used by anyone who does not pass that key as a header in a request. To find out more, read the docs around securing Blackbird instances.

Wrapping Up

In this guide I showed how you can use the Blackbird CLI to test webhooks by using the blackbird code run and blackbird code debug commands. Try it yourself by creating a Blackbird account and downloading the CLI today!