Amazon Eventbridge Pipes was launched on the last re: invent 2022. Amazon EventBridge Pipes allows you to glue different AWS services and optionally filter or enrich the structure of the events passed from consumers to producers.
You can use an SQS queue as a source of the Amazon EventBridge Pipe and an AWS Step Function as a target.
I attached a DLQ to the SQS source, but reading the documentation it was not clear to me when the message was going to be sent to the DLQ, so I decide to give it a try myself.
When the Amazon EventBridge Pipe is created with a State Machine as a target, it is required to set the type of Invocation as FIRE_AND_FORGET or REQUEST_RESPONSE.
Standard workflows
For Standard workflows, the only type of Invocation allowed is the FIRE_AND_FORGET which means that EventBridge doesn’t wait for a response before proceeding. I set up a Standard Workflow with a Lambda which throw an error when it was invoked.
The Standard Workflow was failing, but the Amazon EventBridge Pipe was not aware of the executions and counted each one of the invocations as successful. I polled the messages of the DLQ over various minutes through the AWS Console, but the DLQ did not have any messages.
Express workflows
For the test with an Express workflow, I set up the workflow as invocation type REQUEST_RESPONSE. The Amazon EventBridge Pipe this time was aware of the FailedExecutions and it did not take too much time until the messages started appearing in the DLQ. This means that the failed messages were able to be re-driven to the original source queue.
Summary
A message will never arrive in the DLQ attached to an SQS if we use SQS as a source of the Amazon EventBridge Pipes and a Standard Workflow as a target.
Hopefully in the future, the EventBridge team will make this possible. But nowadays this does not help in the evolution to an asynchronous world.
Top comments (0)