AWS messaging and events - Decoupling applications with an event bus

Few days back, I was listening to a session from James and this note is summary of my notes from his session. All credit goes to him for such an outstanding session.

What is Event Bus?

Standard Wikipedia definition says “An Eventbus is a mechanism that allows different components to communicate with each other without knowing about each other. ... Components can also listen to Events on an Eventbus, without knowing who sent the Events” .

Event driven architecture pattern is a distributed asynchronous architecture pattern which helps to create highly scalable reactive applications. The main idea is delivering and processing events asynchronously without depending on producers and consumers.

Advantages are:

• Decoupling of producers and consumers

• Filters and routes events

What is monolithic micro service Integration

Let's look into the Integration architecture. Good point is that here, we are using microservice approach but bad part is that it has become monolithic architecture approach.

Now, lets look more into the following and understand how fallout management will happen. Let’s say we have some error happened into Fulfilment microservice and now we need to communicate back to payment microservice to rollback/credit the payment and then report the same to Invoice, Forecasting microservice.

This architecture still have all our standard microservice architecture advantages

• Loose Coupling
• Scalability
• Extensibility
• Complexity reduction by multiple microservices
• High Availability
• Fault Tolerance

Integration Patterns with Event Bus

So, the above monolithic architecture, shown above can be more decoupled with following architecture where Order Service (Producer) and other services (consumers) are separated by Event Bus. So, all downstream services read from Event Bus and we have rules to filter accordingly. So, order service need to talk only to Event Bus and team of Order Service does not need to be bothered about other dependent services.

Integration pattern with AWS EventBridge

This is a revolutionary service which helps in serverless world to create decoupled services. This is a serverless event bus service which is managed by AWS. It can integrate with Our own native application through AWS SDK and AWS services.

It is always recommended to use SQS between EventBridge and Lambda to provide durability.

This service is originated from Amazon Cloudwatch events which processes loads of events from several AWS services.

Another interesting point is that, we have SQS, SNS and then EventBridge to Integrate with. So, confusion is which one to use and when to use.

• SQS is a queue where producer pushes the event into queue and consumers reads from Queue. Messages in queue are durably persistently.
• The above pattern is quite similar with SNS where publishers sent message with a topic and consumers read as per their selected topic.

Main difference between SNS and EventBridge are routing capability of service and type of consumers.

• SNS is great for large set fan-out and having thousands of consumers
• Whereas, EventBridge is more suitable when we need rich routing capability. It is also possible to combine with high set of consumer services coupled with this rich routing capability ones.
• Also, EventBridge can have many(Producers) to many (Consumers) relationship which is quite unique.

EventBridge has retry capability also, if we have a downstream system which is not available for some reason then EventBridge will keep on retrying for 24 hrs. We then can have DLQ attached with EventBridge attached where message can be moved if the message is not delivered after multiple attempt.

Default Event Bus always comes with account and we need to create other type of Event Bus if required. Once we have these Bus define then we define rules and they can become very complex sometime depending on routing requirements. Rule can define, “Event with attribute of type Payment need to go to Payment Service”.

What is EventBridge Rules

In EventBridge, an event is simply a JSON structure. Events generated from AWS services always contain a number of descriptive fields and are identifiable by the source attribute prefix “aws”. A typical event structure for a custom application looks like this:

AWS SDK EventBridge PutEvent API sends custom events to Amazon EventBridge so that they can be matched to rules.

Request Syntax:

{
   "Entries": [ 
      { 
         "Detail": "string",
         "DetailType": "string",
         "EventBusName": "string",
         "Resources": [ "string" ],
         "Source": "string",
         "Time": number,
         "TraceHeader": "string"
      }
   ]
}

Response Structure from EventBridge

• If the action is successful, the service sends back an HTTP 200 response.

{
   "Entries": [ 
      { 
         "ErrorCode": "string",
         "ErrorMessage": "string",
         "EventId": "string"
      }
   ],
   "FailedEntryCount": number
}

EventBridge examines the incoming event and compares it against this rule. The rule specifies a source value of custom.myown and, as this exists in the event, the pattern matches. It then routes the event to the rule’s targets:

The example above shows a static, exact match pattern – the attribute is either present, or it’s not.

Here is another architecture pattern which shows how multiple consumers can be triggered based on rules.

Scalable webhook

Following examples shows a scalable architecture, where API Endpoint is connected with EventBriddge. EventBridge is then routing specific events to SQS which is triggering lambda when message arrived in SQS. So even if Lambda is down due to some application code issue then SQS will be able to persistently retain message for a fixed time. So here even though API gateway has 29 sec sync time out, it is always a good practice to have send something from API gateway and get some response (may be a dummy one) and then in backend we do async processing. So here in this architecture, API gateway is getting the response from EventBridge and the behind the scene EventBridge is doing all the async work to get job done.

Responding to AWS Step Functions workflows

Let’s refer to above ECommerce application microservice application architecture and take Payment microservice. Payment service can have very complex logic sometime depending of payment type Cheque, Credit Card etc. Sometime Cheque payment can take several days to process due to Bank clearance etc and there may be requirement of reversing the payment when payment fails. Key point is that no pricing impact on AWS EventBridge as any events to EventBridge from AWS Services are free of charge to process.

Here, lambda function does something depending on state function result is successful or not.

Cloudtrail-integration

CloudTrail is enabled on your AWS account when you create the account. For an ongoing record of events in your AWS account, including events for EventBridge, create a trail. A trail enables CloudTrail to deliver log files to an Amazon S3 bucket.

Here, is the extension of above architecture pattern where cloudtrail will capture all the events and then pass-on to EventBridge. Here the lambda function is being triggered only when our source of event is “aws.s3” and bucketName are “mysales-Jan”, “mysales-Feb” and “mysales-March”.

Key point here is that our bucket names are hardcoded and this means whenever a new buckets are created and if we have to use same lambda then rule needs to be changed. So we can make a dynamic rule as shared below by using bucket prefix.

Multiple buckets with multiple Lambda functions

In the above standard S3 and Lambda integration, a single Lambda function can only be invoked by bucket name prefix in the rule. When you need to invoke multiple functions with the same or overlapping prefixes or suffixes, the EventBridge integration can handle this.

EventBridge allows up to five targets per rule, so you can specify up to five separate Lambda functions to receive the event. All five functions are invoked in parallel when the event pattern matches. To use this, add the targets in the rule – no change to the event pattern is required.

Following example shows, three buckets and three Lambda functions, all subscribing to the same event pattern.

Decoupling large no applications with EventBridge

Following architecture pattern uses events to decouple each service used to process incoming S3 objects. It can also use one or more buckets as event sources, which you can change dynamically as needed. Most importantly, it can be easier to introduce changes and new functionality, since the application is no longer deployed as a mono-repo. Metadata is being sent to Elasticsearch service for Indexing and searching ability.

References

• All credit of this article goes to James Beswick.