Tracking failed SQS messages

_{^{Photo by Paweł Czerwiński on Unsplash}}

Tracking the message flow through the system is indispensable ability in large distributed applications.

Amazon provides limited tools out of the box. There are better dedicated services for this purpose, but sometimes we are left with what we have in AWS.

In this article I will show you how you can track the logs of Lambda function.

The function was invoked from CloudWatch scheduled event and had dead-letter queue (DLQ) assigned.
As a result the failing message landed in DLQ.

1) Get SQS DLQ message - navigate in AWS Cloud Console to SQS service, then find your DLQ and select "Queue Actions -> View/Delete Messages".

You can then see all messages. Select one message and you should see something similar as below:

{
"version":"0",
"id":"29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2",
"detail-type":"Scheduled Event",
"source":"aws.events",
"account":"763369520800",
"time":"2019-09-19T23:30:15Z",
"region":"eu-west-1",
"resources":["arn:aws:events:eu-west-1:763369520800:rule/trigger-lambda-function"],
"detail":{}
}

2) Get the id value (29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2) and lookup in CloudWatch Log Insights for your Lambda function, e.g. use query:

fields @timestamp, @message
| sort @timestamp desc
| limit 20
| filter id='29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2'

As a result you would get a bunch of messages like:

@ingestionTime 1568936016992
@log 763369520800:/aws/lambda/my-lambda-function
@logStream 2019/09/19/[$LATEST]45f87b84a2fd4fb0a79df2c81c57cc06
@message 2019-09-19T23:33:30.528Z b276b9b4-bc43-4876-8052-0f94a38720a7 event: {"version":"0","id":"29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2","detail-type":"Scheduled Event","source":"aws.events","account":"763369520800","time":"2019-09-19T23:30:15Z","region":"eu-west-1","resources":["arn:aws:events:eu-west-1:763369520800:rule/trigger-lambda-function"],"detail":{}}
@requestId b276b9b4-bc43-4876-8052-0f94a38720a7
@timestamp 1568936010528
account 763369520800
detail-type Scheduled Event
id 29d8bf9d-94c0-0e45-b68b-b9aeb4c891c2
region eu-west-1
resources.0arn:aws:events:eu-west-1:763369520800:rule/trigger-lambda-function
source aws.events
time 2019-09-19T23:30:15Z

Pay attention to @logStream value. Now you can lookup this stream in CloudWatch:

The stream can contain plenty of logs so navigating through all of them and searching interesting ones can be tedious work.

Instead of fetching Lambda logs in AWS Cloud Console there are better tools.

You can check saw and the other article which I dedicated to using this tool.