We have often used Webhook a lot during our interactions with the systems unknowingly but do we know what exactly a Webhook is and how it is better than other system design patterns like short/long polling?
A Webhook is an HTTP/S call that gets triggered between the two systems where the source system triggers a specific event that leads to the notification of the targeted application. A very common example is when you swipe your credit card at the merchant for a Point of Sale (POS) transaction, you receive a notification in your phone about the transaction that you made.
Just think of the same scenario of using a Point of Sale (POS) transaction using the concept of a short/long polling pattern where the mobile application or client (in this above example) has to constantly poll the status of your bank account frequently to see any latest updates for any transaction. This leads to the wastage of computing power costing huge money to the company.
Benefits of using Webhooks
- Realtime communication
- Loose-coupled architecture
- No compute resource wastage unlike in the case of short/long polling
- Encryption of the communication in transit over HTTPS
- Tokenized, only authorized system to invoke the Webhook
In this blog, I have designed a Webhook system using the AWS serverless services: Highly Scalable, Cost-effective, Resilient, and Highly Available system.
Architecture
Note: You can also use your own Auth system to secure this Webhook Endpoint using a serverless AWS Cognito or SAML-based system (not shown in the architecture) at the API Gateway
How does this serverless Webhook work?
The DNS (Domain Name System) for the Webhook Endpoint is configured with the AWS Route 53 service with an alias record pointing to the endpoint of the API Gateway.
The API Gateway is a regional service from AWS and serves as the entry point for incoming requests efficiently routing them to the AWS SQS (Simple Queue Service). The API Gateway has a default rate limit of 10,000 requests per second (RPS) and a default burst limit of 5,000 requests.
The SQS Queue used here is FIFO (First In First Out) based as the ordering of every request is critical to the Webhook system and not just that the SQS FIFO provides content-based deduplication out of the box, which means if a request from the producer system is triggered twice by mistake within 5-minute span then it will be considered as duplicate and will not reach to the queue. The SQS will act as a reliable buffer, decoupling components and ensuring no data loss during traffic spikes or service failures, allowing the system to retry.
All the messages received in the SQS FIFO will be processed in the same order by the AWS Lambda which can scale out by itself up to 1,000 concurrent executions of Lambda every 10 seconds. The AWS Lambda will be used for the actual processing of the Webhook business logic here. When handling potentially high volumes of requests, Lambda functions further enhance scalability by allowing for event-driven, serverless execution of code, automatically scaling up or down based on demand.
All the logs generated by it will be streamed to the AWS CloudWatch Logs for monitoring and debugging purposes.
If a message is not processed timely or if the SQS queue receives a bad message then it will be moved to the Dead Letter Queue (DLQ) for re-processing of them.
The AWS Certificate Manager will be used to manage the SSL certificates and ensure that they are valid and rotated before their expiration date ensuring that all HTTPS requests are encrypted in transit
This combination of API Gateway, SQS, and Lambda enables robust, cost-effective resilient, and scalable architectures capable of handling varying workloads and maintaining high availability under challenging conditions.
Conclusion
In conclusion, implementing a scalable Webhook system using AWS serverless services offers numerous advantages in terms of real-time communication, loose-coupled architecture, and efficient resource utilization. By leveraging AWS API Gateway, SQS, and Lambda, we can design a robust, cost-effective, resilient, and highly available architecture capable of handling varying workloads and maintaining high availability under challenging conditions. By adopting this architecture, organizations can streamline their workflows, improve system reliability, and reduce operational costs associated with traditional polling-based approaches. Embracing serverless technologies on AWS empowers developers to focus on building and innovating, while AWS handles the underlying infrastructure management, enabling scalable and resilient solutions for diverse use cases.
Top comments (0)