Amazon CloudWatch Logs Insights Query Snippets as Code

#aws #cloudwatch #cdk #serverless

Amazon CloudWatch is a suite of observability-related products. From log filtering to alerting, the service can do it all!

In this blog post, I would like us to focus on a feature of the Amazon CloudWatch Logs Insights that I feel is deeply underutilized - the predefined queries.

This blog post assumes the reader is familiar with AWS CloudFormation. The code presented is written using AWS CDK. If you are not familiar with it, fear not! In the example architecture GitHub repository, you can find the same architecture deployed using AWS SAM.

Amazon CloudWatch Logs Insights refresher

Living under the Amazon CloudWatch umbrella, the Amazon CloudWatch Logs Insights allow you to query logs of your application using a declarative query language.

If your understanding of the query language is deep enough, you will be able to create elaborate diagrams or even business metrics using this service.

The thing to remember here is that, in my opinion, the AWS CloudWatch Logs Insights service is essential in the context of day-to-day development on AWS.

The problem

Given my passion for everything AWS (and serverless), I find myself interacting with the Amazon CloudWatch Logs Insights quite frequently.

During my time spent with the service, I discovered the "Queries" tab in the service console.

The "Queries" tab contains various predefined queries one might use to filter through application logs. What I think some people do not realize is that it is possible to save custom queries.

Imagine being woken up in the middle of the night by PagerDuty. You login into the AWS console, search for CloudWatch Logs Insights, and, using the predefined query, dig into application logs within seconds. All of this without having to remember the query language.

Sounds fantastic, right? I think so as well. Sadly, this reality is often not possible because in my experience, most people underutilize the "save query" feature.

Maybe it is a problem of adoption – perhaps teams are using different tools for searching their application logs. I'm not sure.

Suppose your team uses Amazon CloudWatch Logs Insights for log filtering. In that case, I believe that having a predefined set of queries tailored to your application ready to go will make every developer's life easier.

Let us explore how one might create predefined queries using (AWS CDK). This method works great if your infrastructure is defined using code already (if it is not the case, please consider doing so).

Queries as Code

Since, to my best knowledge, there is no "predefined Amazon CloudWatch Logs Insights query" AWS CloudFormation resource available, we have to turn our attention to either AWS CloudFormation custom resources or AWS CloudFormation modules.

I'm going to use AWS CloudFormation custom resources since they work well with AWS CDK.

The first step is to identify the API call responsible for creating, updating, and deleting the query. After a bit of googling, I was able to pinpoint the operations I need to get myself familiar with – PutQueryDefinition and the DeleteQueryDefinition.

The second step is to create a custom AWS CloudFormation resource that uses those APIs. Luckily for us AWS CDK exposes great abstractions that will help us greatly in this department – enter the custom-resource package.

The following is a modified excerpt from the code found in the example architecture GitHub repository.



import * as cdk from "@aws-cdk/core";
import * as customResources from "@aws-cdk/custom-resources";
import * as iam from "@aws-cdk/aws-iam";

export class YourStackName extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // This is the name that will be visible in the AWS CloudWatch Logs Insights "Queries" tab.
    const queryName = "EXAMPLE_QUERY_NAME";
    // Remember to format the query for readability purposes!
    const byAPIGWRequestIdQuery = `fields @timestamp, @logStream, @message
    | sort @timestamp desc
    | filter @requestId = "PASTE_REQUEST_ID_HERE"`;

    new customResources.AwsCustomResource(this, "insightsQuery", {
      policy: customResources.AwsCustomResourcePolicy.fromStatements([
        new iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          actions: ["logs:PutQueryDefinition", "logs:DeleteQueryDefinition"],
          resources: [`arn:aws:logs:${cdk.Aws.REGION}:${cdk.Aws.ACCOUNT_ID}:*`]
        })
      ]),
      onCreate: {
        action: "putQueryDefinition",
        service: "CloudWatchLogs",
        parameters: {
          name: queryName,
          queryString: byAPIGWRequestIdQuery
        },
        physicalResourceId:
          customResources.PhysicalResourceId.fromResponse("queryDefinitionId")
      },
      onUpdate: {
        action: "putQueryDefinition",
        service: "CloudWatchLogs",
        parameters: {
          name: queryName,
          queryString: byAPIGWRequestIdQuery,
          queryDefinitionId: new customResources.PhysicalResourceIdReference()
        },
        physicalResourceId:
          customResources.PhysicalResourceId.fromResponse("queryDefinitionId")
      },
      onDelete: {
        action: "deleteQueryDefinition",
        service: "CloudWatchLogs",
        parameters: {
          queryDefinitionId: new customResources.PhysicalResourceIdReference()
        }
      }
    });
  }
}

That might seem like a lot of code, but in reality, we repeat many things in-between the onCreate, onUpdate, and onDelete calls.

After deployment, open the AWS CloudWatch Logs Insights console and navigate to the "Queries" tab. The newly defined query will be visible there.

Looking at the code, I think the most crucial part is the IAM policy definition. Please remember to always scope IAM policies down as much as possible. If you find more granular resources ARN, I could use here, please let me know!

As for the customResource class itself, please consult the custom-resource module documentation – it is excellent!

Closing words

There you have it – how to create AWS CloudWatch Logs Insights queries and incorporate that flow into your IaC pipeline.

I would argue that the effort required to create those queries is relatively low, but the potential gains from having them available are relatively high. I hope that you or your team find some value in this blog post.

Thank you for your time.