Implement AWS Lambda Advanced Logging Controls using Infrastructure as Code

Some time ago, AWS release advanced logging controls for AWS Lambda. This gives developers and operators greater control over how function logs are captured, processed, and consumed.

There are three new capabilities introduced:

1. You can capture Lambda function logs in JSON structured format without having to use your own logging libraries. JSON structured logs make it easier to search, filter, and analyze large volumes of log entries.

2. You can control the log level granularity of Lambda function logs without making any code changes, enabling more effective debugging and troubleshooting.

3. You can also set which Amazon CloudWatch log group Lambda sends logs to, making it easier to aggregate and manage logs at scale.

In this article I will show you how you can implement the Advanced Logging Controls using Infrastructure as Code. I will show you an example using Serverless Framework.

Why is this good to implement?

Imagine you have an application with multiple Lambdas, and now you have the capability to query logs from various sources simultaneously. This provides you with increased control over the capture, processing, and consumption of the logs of your Functions. For instance, you can seamlessly query this data in CloudWatch Log Insights by selecting your aggregated log group, encompassing all the Lambdas you've added for logging.

Let's implement this using Serverless Framework. For this we'll be using the serverless-logging-config plugin.

Three steps need to be completed:

1. Add the plugin:

plugins:
  - serverless-logging-config

1. Customize the settings for the plugin:

custom:
  serverless-logging-config:
    enableJson: true # [Optional] if enabled, set the LogFormat to JSON
    logGroupName: AggregatedLogGroup # [Optional] if set, all functions will send logs this log group
    applicationLogLevel: INFO # [Optional] valid values are DEBUG, ERROR, FATAL, INFO, TRACE and WARN
    systemLogLevel: INFO # [Optional] valid values are DEBUG, INFO and WARN
    useDefaultLogGroup: # [Optional] these functions would keep logging to their default log group
      - function1
      - function2

Note that everything under the useDefaultLogGroup will not be included in your aggregated log group.

1. Create the custom AggregatedLogGroup where your logs will be stored:

You final serverless.yml file should look something like this:

service: advanced-lambda-logging
frameworkVersion: '3'

plugins:
  - serverless-logging-config

provider:
  name: aws
  runtime: nodejs18.x

custom:
  serverless-logging-config:
    enableJson: true
    logGroupName: AggregatedLogGroup
    applicationLogLevel: INFO
    systemLogLevel: INFO

resources:
  Resources:
    AggregatedLogGroup:
      Type: 'AWS::Logs::LogGroup'
      Properties:
        LogGroupName: AggregatedLogGroup
        RetentionInDays: 90

functions:
  add:
    handler: src/todo/add.handler
    name: 'todo-app-add'
    events:
      - httpApi:
          path: /
          method: post
  fetch:
    handler: src/todo/fetch.handler
    name: 'todo-app-fetch'
    events:
      - httpApi:
          path: /todo/{id}
          method: get
  update:
    handler: src/todo/update.handler
    name: 'todo-app-update'
    events:
      - httpApi:
          path: /todo/{id}
          method: put
  remove:
    handler: src/todo/remove.handler
    name: 'todo-app-remove'
    events:
      - httpApi:
          path: /todo/{id}
          method: delete

Access the same Log Group from multiple Serverless files

A common practice is that you may have multiple Serverless files and sometimes you want to access the same Log Group in one or more of these Serverless files. This is how you can achieve that:

In one of your Serverless file, you create the Log Group:

resources:
  Resources:
    AggregatedLogGroup:
      Type: 'AWS::Logs::LogGroup'
      Properties:
        LogGroupName: AggregatedLogGroup
        RetentionInDays: 90

Then just reference this Log Group in your other Serverless files:

Note: just make sure the Log Group created and deployed before you reference it and that the name is exactly the same. In this case it's AggregatedLogGroup.

...

custom:
  serverless-logging-config:
    enableJson: true
    logGroupName: AggregatedLogGroup
    applicationLogLevel: INFO
    systemLogLevel: INFO

...

Conclusion

I have begun implementing advanced logging controls for AWS Lambda in my projects, and I've already observed a significant improvement in the quality of using our logs. This has positively impacted the processing efficiency and overall consumption of the logs. Have you tried this out? If so, what has been your experience with it?