EDA Messaging: SQS Event Source to Lambda Function Demo

This is the blog post to the demo application on EDA Point-to-point messaging with an Amazon Simple Queue Service (SQS) queue as the event source to an AWS Lambda function. This post examines some of the features, capabilities, and limitations of SQS queues and how a Lambda function gets messages from queues.

Amazon SQS

Amazon SQS is a fully managed queue service for buffering messages. A queue provides a point-to-point or one-to-one integration with a consumer. The queue buffers the messages it receives for a specified retention period. When the consumer is a Lambda function with the queue configured as its event source, an internal event source mapping resource will perform the polling and invoke the Lambda function with a batch of messages when available.

Setting Visibility Timeout on the queue

How much time to allocate for consumers to complete processing and deleting the messages so the messages remain unavailable to any other consumer? In general, if that amount of time can be estimated, set it to the maximum of the estimate. Otherwise, keep extending the timeout while it is still being processed. Maximum visibility timeout is 12 hours, default is 30 seconds.

Adding Dead-letter Queue for troubleshooting

If a message is unprocessed and visibility times out, it comes back to the original queue. The default number of times a message can be received is 10 times. To send undeliverable and unprocessed messages to another buffer for troubleshooting, add a dead-letter queue on the original message queue. Configure with a maximum retention period of 14 days.

Standard Queue Type

At-least-once delivery

Standard queues do At-least once delivery so it is possible to have duplicate messages delivered. So, to help prevent messages from being polled and processed again, the consumer Lambda function should delete those messages from the queue after processing. However, even after deleting those messages, it is still possible that another copy of that message exists and gets sent to a consumer.

Considering Idempotency

Because duplicate messages can be received by consumers, it’s important to consider the idempotency of the operations performed.

What are the side effects caused by this lambda function? If this lambda function is invoked with another copy of the message, will it have the same impact on the outcome as if it were invoked with that same message exactly once?

What will make this message unique, so that a duplicate message is caught, and that an actual different message can go through to processing?

AWS Lambda

AWS Lambda is a fully managed compute service that executes and monitors your Lambda functions and logs their outputs to CloudWatch Logs. The service runs your functions only when needed and scales automatically. The lambda function consists of the code, dependencies, and configurations.

Lambda function invocation types

With synchronous invocation, the caller waits for a response until it times out. The response will be either a success or failure directly from the function. If the response is an error, the caller making the synchronous invocation would have to catch that.

With asynchronous invocation of a Lambda function, the caller sends the request and immediately receives a successful response. This caller continues to the next task without waiting for the processor to complete processing. Destinations can be added to asynchronously invoked Lambda functions. In this way, when processing completes, either successfully or in error, there is a place to store that information for monitoring, further processing, or troubleshooting.

With event source mapping. For this lambda function, an internal event source mapping resource is used to poll messages from the SQS queue, batch queue records together, and invoke the function with those messages. In the AWS Console, this is adding the SQS as a trigger to run the Lambda function:

Demo application in CDK Python

GitHub repo:
https://github.com/evefonwu/sqs-event-source-to-lambda

The codebase consists of the following:

Sender with a Function URL to send a message to the SQS queue.
Original message queue that will receive the message.
Dead-letter queue added to the original message queue.
Processor function configured with the original message queue as an event source.
Processor function logs messages received and deletes those from queue.

Once deployed, CDK will output the sender Function URL.

Access the URL to send a message to the queue.

The receiving processor function will log the SQS message records to CloudWatch Logs.

This is an example of the Lambda function log stream: