AWS EventBridge Replays

AWS recently announced native replay functionality for EventBridge! In my opinion, replayability and event archiving in event-driven architectures are often overlooked and a resilient, cost-effective solution is often needlessly complicated - so this is fantastic news! 🚀

I was eager to try it out so I decided to build a small demo - come join me!

Project setup

npm init -y, install Serverless framework, serverless-pseudo-parameters and the AWS SDK

npm install serverless serverless-pseudo-parameters --dev

Let's start by creating a fake order producer that we later can pretend is some important service that lives somewhere else. The "createOrder" service will put an event on our message bus containing an order based on the orderId we pass.

// createOrder.js
const AWS = require('aws-sdk');

const eventBridge = new AWS.EventBridge({ region: 'eu-north-1' });

const handler = async (event) => {
  const order = {
    id: event.orderId,
  };

  return eventBridge.putEvents({
    Entries: [{
      EventBusName: 'orders',
      Source: 'elk.orders',
      DetailType: 'order',
      Detail: JSON.stringify(order)
    }]
  }).promise();
};

module.exports.handler = handler;

next, let's create our "orderCreated" function which will listen for any order events on the bus and work its magic on them.

// orderCreated.js
const handler = async (event) => {
  console.log(`handled order ${event.detail.id}`);
};

module.exports.handler = handler;

and finally, let's create our serverless.yml where we'll define all the infrastructure resources our service will need.

# serverless.yml
service: order-service

plugins:
  - serverless-pseudo-parameters

provider:
  name: aws
  region: eu-north-1
  runtime: nodejs12.x
  iamRoleStatements:
    - Effect: Allow
      Action: events:PutEvents
      Resource: ${self:custom.eventBusArn}

custom:
  eventBusArn: arn:aws:events:${self:provider.region}:#{AWS::AccountId}:event-bus/orders

functions:
  createOrder:
    handler: createOrder.handler
  orderCreated:
    handler: orderCreated.handler
    events:
      - eventBridge:
          eventBus: ${self:custom.eventBusArn}
          pattern:
            source:
              - elk.orders

resources:
  Resources:
    EventBus:
      Type: AWS::Events::EventBus
      Properties:
        Name: orders
    EventArchive:
      Type: AWS::Events::Archive
      Properties:
        ArchiveName: orders-archive
        SourceArn: ${self:custom.eventBusArn}

There's quite a lot going on here, so let's break it down a bit, shall we? We shall.

We're specifying that the orderCreated function should be triggered by all events on the "orders" EventBus. The EventBus itself is defined as a CloudFormation resource.
However, at the time of writing this, there's a bug in Serverless framework that prevents us from referencing the ARN of the CF resource in the event definition so we reference the "calculated" ARN in a custom property, eventBusArn, instead.

In addition to the EventBus resource, there's an Archive resource as well. This is the resource that defines our archive, in which all events put on the bus will be archived (and thus available to be replayed). We also set up the Lambda EventBridge trigger and specified that we're interested in all events coming from the "elk.orders" source.

We've also added an IAM role statement that allows our service to put events on the bus since we're hosting our event producer in the same service.

Alright, let's try it out.

From your nearest terminal, deploy the service by running npx sls deploy. We can create our first event by invoking the producer with an order

npx sls invoke --function createOrder -d '{ "orderId": "1" }'

if we now look at the logs of our orderCreated function, we should see that that it's received and "processed" the order.

npx sls logs --function orderCreated
[...]
START RequestId: d8069d8e-ed1a-4837-8e3f-b44f12e23773 Version: $LATEST
2020-11-09 00:18:41.268 (+01:00)    d8069d8e-ed1a-4837-8e3f-b44f12e23773    INFO    handled order 1
END RequestId: d8069d8e-ed1a-4837-8e3f-b44f12e23773

Great, it works - let's destroy something! If our orderCreated Lambda were to fail, the events would automatically be retried with an automatic exponential backoff - but what happens if we don't actually "fail", we just fail to process the event correctly? Unfortunately, it would be lost.

Let's simulate something like that happening by changing the name of our order's id field from id to orderId in the event.

// createOrder.js
[...]
const handler = async (event) => {
  const order = {
    orderId: event.orderId,
  };
[...]

deploy our service again, and then create another event, order 2

npx sls invoke --function createOrder -d '{ "orderId": "2" }'

The logs from the orderCreated function will now say

INFO    handled order undefined

Alright, I think we can figure out what the problem is, let's fix the bug

// orderCreated.js
const handler = async (event) => {
  console.log(`handled order ${event.detail.orderId}`);
};

module.exports.handler = handler;

and deploy again.

Replays to the rescue

We botched the handling of that second order but luckily we created an archive for our events when we created the event bus.

@mhlabs/evb-cli is a neat little CLI tool to manage EventBridge, so let's use that to trigger our replay.

npm i @mhlabs/evb-cli -g

Run the tool and specify the "orders" bus, evb replay -b orders, then adjust the time span as needed.

It'll take a minute or so for EventBridge to fetch and re-send your events from the archive, but in a moment we should be able to see that the second order has now been processed by our orderCreated handler! 🎉

INFO    handled order 2

Isn't that amazing? With just a handful of lines of code, we've set up an EventBus that durably stores all events for as long as we need them from which we can replay events sent within a given time span to any consumers that need to (re)process the events as well as a service that reads and writes to the bus.

Do keep in mind that when designing systems capable of replaying events, it's crucial to make sure your services are idempotent. If (when) the same order event is sent for the second time, your customer likely didn't buy two of the same Die Hard 2 DVD off your site at the same time.

The code for this demo can be found here.

PS.

Did you know EventBridge can automatically generate schemas and code bindings for Typescript, Python or Java? Using those would've prevented that whole id->orderId fiasco from before... Oh well.

If you enjoyed this guide and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS and developer productivity!