Amazon EventBridge API Destinations with parametrized endpoint URL and Mailchimp integration

I love ❤️ working with Amazon EventBridge service. This is not a big surprise as it is beloved by the community. However, for me, it's not just the pure technical excellence of it.

Whenever I think about EventBridge I hear in my mind famous words from the TRON movie. They were even reused at the very beginning of the sequel TRON: Legacy.

I tried to picture clusters of information as they moved through the computer.
What do they look like? Ships? Motorcycles?
Were the circuits like freeways?
I kept dreaming of a world I thought I'd never see.
And then, one day...
I got in.

Even though those words were said about computer hardware not message bus per se, in my opinion they apply perfectly to it.

Whenever I work with EventBridge I have this sci-fi excitement that origins from that movie.

I hope you will get that vibe too!

What you will learn from the article

This article is not just about Mailchimp integration. You should read it if you want to learn how to:

• setup Amazon EventBridge API Destinations to send REST requests to any 3rd party endpoint on the Internet
• use HTTP Path Parameters to trigger different endpoints based on message content - this is huge as it is almost not described in the documentation, and there are no examples explaining that functionality.
• setup and deploy sample project using Serverless Framework to learn by doing in your AWS account
• integrate it with Mailchimp to tag users based on messages on your Event Bus
• and finally, what to do when something doesn't work (how to debug API Destinations integrations)?

Business case

This tutorial is based on my experiences while implementing API Destinations integration with Mailchimp.

I have a small e-commerce platform that sells online courses. Whenever a customer buys any of the products I want to mark that information in my Email Service Provider (ESP). My ESP of choice is Mailchimp and I want to give a specific tag associated with the product to my customer, so I will not send him promotional emails about products that he already has. Unfortunately, my e-commerce platform is unable to tag users in Mailchimp, so I had to implement that myself.

Of course, I utilized serverless and as always it was fun & a lot of learning 😃

High-level architecture

Take a look at high-level architecture of the solution.

Getting Orders from e-commerce platform is beyond the scope of this article. We're going to focus on EventBridge and integration with Mailichimp by using API Destinations. The assumption is that you know how to send a new event to a Custom Event Bus.

What is API Destinations?

API Destinations is relatively new functionality of Amazon EventBridge which allows us to integrate with 3rd party REST endpoints on the Internet directly without any Lambda functions!

All you need to do is to configure the service instead of writing your own code. As you're going to see, there is plenty of CloudFormation. However, you will get much better result compared to your own implementation.

This update follows the essential AWS principle stated by Chriss Munns during re:Invent 2019 use functions to transform, not transport.

And that explains the whole idea behind it. Let's leave the mundane task of calling 3rd party endpoint (data transport) to AWS. We trade code (custom implementation) for configuration of API destinations. In return, we get a reliable solution that:

• always works
• is someone's else problem / responsibility
• is being developed by best engineers there are. 😃

Implementation

In real life there is no place for clicking and defining infrastructure manually. Professional projects require automation, repeatability, reliability, and reusability.

Therefore, in this article instead of AWS Console screenshots I will present code excerpts following Infrastructure as Code principle.

If you're impatient, you can check this sample project on GitHub, that contains all the sources.

Solution components

Event Bus

To configure API Destinations we need our custom Event Bus, which is a backbone of our solution. It is simple to define, using CloudFormation.

    EventBus:
      Type: AWS::Events::EventBus
      Properties:
        Name: ${self:service}-${self:provider.stage}

Both variables ${self:service} and ${self:provider.stage} come from Serverless Framework (SF). Let me explain, for those of you who are not fluent with SF. The first one resolves to the name of the project, and the second one to the name of a stage to which the service is deployed. That way there is no name clashing when different stages of the same project are deployed to the same region (or at all, as some resources are global i.e. S3 buckets, IAM Roles). Personally, I use dev, test and prod stage names in my projects.

Connection

Next we need AWS::Events::Connection object that contains authentication credentials (such as username and password) to the 3rd party endpoint.

    MailchimpConnection:
      Type: AWS::Events::Connection
      Properties:
        Name: MailchimpConnection-${self:provider.stage}
        AuthorizationType: BASIC
        AuthParameters:
            BasicAuthParameters:
                Username: "randomUser"
                Password: ${ssm:mailchimp-api-key~true}

To login to Mailchimp we will use BASIC authorization type. In case of the Username parameter Mailchimp allows any string and is only interested in the Password part. From Mailchimp perspective Password value is an Api Key that can be generated under Account->Extras-> API Keys.

Basic Authorization Type configuration will result in the authorization header added to the request created by API Destinations when calling Mailchimp REST endpoint. It looks something like that:

authorization   Basic cmFuZG9tVXNlcjpZlPkmY2hpbXBIcGlQYXNzd5CyZA==

Value of the Password parameter is set to ${ssm:mailchimp-api-key~true} which is again Serverless Framework specific. It retrieves secret value from Systems Manager Parameter Store (ssm prefix) during deployment. Parameter is of name mailchimp-api-key and ~true means it must be decrypted. More on this in Serverless Framework documentation.

ApiDestination

What could be strange for you is that we didn't define endpoint URL in the connection object. This is done in AWS::Events::ApiDestination.

    MailchimpApiDestination:
      Type: AWS::Events::ApiDestination
      Properties:
        Name: mailchimp-tag-${self:provider.stage}
        ConnectionArn: !GetAtt MailchimpConnection.Arn
        InvocationEndpoint: ${self:custom.mailchimp.endpoint}/lists/${self:custom.mailchimp.list}/segments/*/members
        HttpMethod: POST
        InvocationRateLimitPerSecond: 20

Here we have three interesting elements:

• InvocationEndpoint
• HttpMethod
• InvocationRateLimitPerSecond

The InvocationEndpoint contains two custom variables that contain my specific Mailchimp URL and mailing list Id. This particular URL allows adding a user to the tag (named as segment in Mailchimp). However, what's fascinating from API Destinations perspective is the * character in URL. It works as a placeholder which will be replaced by the HTTP Path Parameter value, taken from the Order event. We will define that parameter later.

Right now I want you to fully grasp the meaning of that. The endpoint URL can be parametrized based on event payload sent to the EventBus. This is great, as allows us to reuse single ApiDestination configuration to send data to complex endpoints.

The InvocationRateLimitPerSecond is another great treat of API Destinations. It allows us to limit rate of requests fired by API Destinations to the 3rd party. Obviously, it's a crucial thing to consider during integrations, as not all systems are as scalable as AWS serverless solutions. Here we get that for free and without any additional coding.

I hope this incredibly usefull setting will be added to other contstructs in EventBridge family. I could imagine rate limit added to existing Lambda targets or Simple Email Service targets (which dear AWS would be awesome to have 🤞) 😉

Rule

The last EventBridge specific element of the puzzle is AWS::Events::Rule that includes so called Target. The Rule object acts as a router or dispatcher. It tells our custom EventBus which events send where. The Target defines the where part and ties together Rule with new API Destinations functionality.

    ApiDestinationDeliveryRule:
      Type: AWS::Events::Rule
      Properties:
        EventBusName: !Ref EventBus
        Name: SendOrderEventsToApiDestinations
        EventPattern:
          detail-type:
            - ORDER_COURSE_SERVERLESS
            - ORDER_COURSE_DATALAKE
        State: "ENABLED"
        Targets:
          - Id: MailchimpApiDestination
            Arn: !GetAtt MailchimpApiDestination.Arn
            RoleArn: !GetAtt ApiDestinationsTargetRole.Arn
            InputTransformer:
              InputPathsMap:
                mcTagId: $.detail.mcTagId
                email: $.detail.email
              InputTemplate: >
                {
                  "email_address": <email>
                }
            HttpParameters:
              PathParameterValues:
                - $.detail.mcTagId

In the EventPattern section it is defined to which events this rule applies to. In my case, it is limited only to Order events.

Now let's talk about targets. A single Rule can define many of them, here we have just one, named MailchimpApiDestination. Target refers to defined above ApiDestination, it requires a dedicated IAM Role.

It also defines InputTransformer, that allows us to modify event payload and tailor it to the 3rd party API specification. In my case, it's Mailchimp's Add member to segment method, which expect single value under email_address variable in JSON.

Event values can be access by $.detail and used in the InputTemplate section to construct expected object structure.

The last section allows us to parametrize endpoint URL defined in ApiDestination object with values from the event. PathParameterValues takes an array of values, which means that we can define more than one placeholder in the URL. They are resolved in order, the first element in the array refers to the first * in the URL, and so on.

IAM Role

We also need to define a Role that can be assumed by a events.amazonaws.com service. This role must grant events:InvokeApiDestination on MailchimpApiDestination resource. The full definition of the role can be found in the sample project on GitHub in serverless.yml file.

Sample solution deployment

I have prepared a sample project using Serverless Framework, so you can deploy it on your AWS account and play with it.

You need to clone the code, and install dependencies:

git clone https://github.com/serverlesspolska/eventbridge-api-destinations-mailchimp.git
cd eventbridge-api-destinations-mailchimp
npm i

This requires node.js and npm be installed on your machine.

As not everyone uses Mailchimp, in the project I defined a second target that sends REST requests to webhook.site service. This is a free, easy to use web application that will work as our 3rd party endpoint.

Please go to the webhook.site and copy Your unique URL. Paste that URL into serverless.yml config file in line 38.

Now we're ready to deploy the project to the dev stage using command:

sls deploy

(This assumes that you have your default AWS profile defined under ~/.aws/credentials.)

After successful deployment (can take several minutes) you can invoke a Lambda function, that will send a sample Order event to the EventBus.

sls invoke -f sendOrderEvent -l

As a result, you should see the new request on the webhook.site website. (If don't refresh the page)

That means that API Destinations just called the endpoint responding to the new message on the EventBus.

How to debug EventBridge API Destinations?

If you misconfigured something, or perhaps your credentials are wrong you will not get any error messages, there are no logs that you can read and try to determine what happened. All you can do is to check 3rd party system and see if the request came to it - that's why we used webhook.site.

What can we do to find out what's the root cause if the request doesn't hit the endpoint at all?

Fortunately, API Destinations allows the use of Dead Letter Queues (DLQ). For each Target defined under AWS::Events::Rule we can specify SQS queue that will contain messages that couldn't be successfully processed by the Target API Destination. Each message that ends up in the DLQ is annotated with information explaining the problem.

Here are two samples of error messages that I managed to get.
This one was returned when I set wrong Mailchimp API Key.

This one was caused by the wrong IAM Role definition.

Summary

I hope you found that article informative, and now you know how to use Amazon EventBridge API Destinations in your projects. Should you have any questions ask them below or on GitHub project site.

I'm convinced that this is a great functionality, even if it took me several hours to understand and implement it correctly for the first time. In the long run, it will save me much more on implementing and debugging Lambda functions that call 3rd party endpoints. They seem to be trivial to implement, but in reality they are not (How would you implement InvocationRateLimitPerSecond functionality in Lambda in a way that you don't pay for idle time?) and I will not miss them. Remember use functions to transform, not transport.

What is really cool and worth stressing out is that this functionality is fully developed and comes with all the parts. Most importantly there is full CloudFormation support which I must sadly admit was not always the case. Kudos to the CloudFormation team! 🙏

Amazon EventBridge is a central point of many serverless solutions, and I'm glad it is still being developed. API Destinations is a great addition to that service and hopefully indicator of many more to come.