Article originally published on Medium https://didil.medium.com/inhooks-3983b68274e1
Lightweight Incoming Webhooks Gateway
Introduction
When subscribing to Webhooks from third parties, developers usually need their services to handle failures, retries, delayed processing, buffering, etc. Inhooks aims to be an open source, lightweight, incoming webhooks gateway solution to handle these requirements in a simple but scalable way. Written in Go and runnable as a single binary or docker container. It only requires a Redis database for storage/queueing.
Architecture
Inhooks Architecture
High level overview
Inhooks listens to HTTP webhooks and saves the messages to Redis. A processing module retrieves the messages and sends them reliably to the defined targets.
Features
- Receive HTTP Webhooks and save them to a Redis database
- Fanout messages to multiple HTTP targets
- Fast, concurrent processing
- Supports delayed processing
- Supports retries on failure with configurable number of attempts, interval and constant or exponential backoff
- … more features planned
Demo
Let’s now see inhooks in action. For this simple demo, we will run inhooks on a cloud virtual machine and we will send it Github push event Webhooks.
We have previously installed Redis on the virtual machine. For this demo we’re running Redis on the same server but connecting to any remote Redis instance should work.
Inhooks Setup
Let’s first fetch and extract the inhooks binary tarball:
$ wget -qO - https://github.com/didil/inhooks/releases/download/v0.1.5/inhooks_0.1.5_linux_amd64.tar.gz | tar -xzvf -
CHANGELOG.md
LICENSE
README.md
inhooks
Next we create the inhooks config file inhooks.yml in the same directory:
#inhooks.yml
flows:
- id: github-webhook
source:
id: gh
slug: github
type: http
sinks:
- id: requestbin-simple
type: http
url: https://en87n9uqqlch.x.pipedream.net
- id: requestbin-advanced
type: http
url: https://ene9c6929em87.x.pipedream.net
delay: 60s # delay processing by 60 seconds
retryInterval: 5m # on error, retry after 5 minutes
retryExpMultiplier: 2 # exponential backoff with a multiplier of 2
maxAttempts: 10 # try up to 10 times
With this configuration, inhooks will ingest http POST requests sent to the path /api/v1/ingest/github and
- Send a request to the first requestbin endpoint immediately.
- Send a request to the second requestbin endpoint after a 60s delay. The second sink (which is what we call the output targets in inhooks) is also configured to retry with exponential backoff starting at 5 minutes with 10 maximum attempts.
Github Webhooks Setup
We add the ingest url endpoint http://35.232.6.40:4567/api/v1/ingest/github to our github test repository’s webhooks configuration and we select the content type as application/json.
Github Webhooks Config
Running the server
We run the server and set a few environment variables. Here we set APP_ENV to development to get human-readable log messages, but in a production environment we would usually set it to production.
$ APP_ENV=development PORT=4567 REDIS_URL=redis://localhost:6379 REDIS_INHOOKS_DB_NAME=demo ./inhooks
2023-05-19T14:30:07.378Z INFO api/main.go:45 loading inhooks config {"inhooksConfigFile": "inhooks.yml"}
2023-05-19T14:30:07.378Z INFO services/inhooks_config_service.go:109 loaded flow {"id": "github-webhook", "sourceID": "gh", "sourceSlug": "github", "sourceType": "http"}
2023-05-19T14:30:07.378Z INFO services/inhooks_config_service.go:117 flow sink {"id": "requestbin-simple", "type": "http", "url": "https://en87n9uqqlch.x.pipedream.net", "delay": "0s"}
2023-05-19T14:30:07.378Z INFO services/inhooks_config_service.go:117 flow sink {"id": "requestbin-advanced", "type": "http", "url": "https://ene9c6929em87.x.pipedream.net", "delay": "1m0s"}
2023-05-19T14:30:07.380Z INFO api/main.go:87 listening ... {"addr": ":4567"}
2023-05-19T14:30:07.380Z INFO api/main.go:124 starting supervisor ...
The server has started successfully. We can now trigger a git push on our test Github repository and see the results in the logs:
2023-05-19T14:31:53.689Z INFO handlers/ingest.go:20 new ingest request {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github"}
2023-05-19T14:31:53.691Z INFO handlers/ingest.go:53 message queued {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github", "flowID": "github-webhook", "sourceID": "gh", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "queue": "ready"}
2023-05-19T14:31:53.691Z INFO handlers/ingest.go:53 message queued {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github", "flowID": "github-webhook", "sourceID": "gh", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "queue": "scheduled", "nextAttemptAfter": "2023-05-19T14:32:53.690Z"}
2023-05-19T14:31:53.691Z INFO supervisor/ready.go:65 processing message {"flowID": "github-webhook", "sinkID": "requestbin-simple", "sinkType": "http", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001", "attempt#": 1}
2023-05-19T14:31:53.691Z INFO handlers/ingest.go:57 ingest request succeeded {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github", "flowID": "github-webhook", "sourceID": "gh"}
2023-05-19T14:31:53.870Z INFO supervisor/ready.go:77 message processed ok {"flowID": "github-webhook", "sinkID": "requestbin-simple", "sinkType": "http", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}
2023-05-19T14:31:53.871Z INFO supervisor/ready.go:83 message processed ok - finalized {"flowID": "github-webhook", "sinkID": "requestbin-simple", "sinkType": "http", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}
What happened above is :
- An ingest request was received.
- The message was queued to redis for both the requestbin-simple as well as the requestbin-advanced sinks.
- For the requestbin-simple sink, the message was sent to requestbin and the request succeeded.
- For the requestbin-advanced sink, the message was scheduled to be processed after 60s as specified in the config.
After a couple of minutes we can see in the logs:
2023-05-19T14:33:07.389Z INFO supervisor/ready.go:65 processing message {"flowID": "github-webhook", "sinkID": "requestbin-advanced", "sinkType": "http", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001", "attempt#": 1}
2023-05-19T14:33:07.550Z INFO supervisor/ready.go:77 message processed ok {"flowID": "github-webhook", "sinkID": "requestbin-advanced", "sinkType": "http", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}
2023-05-19T14:33:07.550Z INFO supervisor/ready.go:83 message processed ok - finalized {"flowID": "github-webhook", "sinkID": "requestbin-advanced", "sinkType": "http", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}
The message for the requestbin-advanced sink was processed and the request succeeded.
Let’s now have a look at the results on the RequestBin side.
For our first target:
requestbin-simple results
And the second target:
requestbin-advanced results
The http headers and body look good. Seems like it worked 🎆 !
Conclusion
I hope you will find inhooks useful. This project is still very young but I believe it has a lot of potential and there are many useful features that could be added. If your backend services receive webhooks, please give inhooks a try in your dev/staging environment and share any feedback to https://github.com/didil/inhooks/issues.
Top comments (0)