DEV Community

Cover image for AWS CloudWatch Contributor Insights. Is it useful?
Paweł Piwosz for AWS Community Builders

Posted on • Updated on

AWS CloudWatch Contributor Insights. Is it useful?

Check logs with Contributor Insights

Right! So we have logs. We can use CloudWatch Logs Insights to query it, we can observe it through CloudWatch logs, and we can do many, many things with them now. Finally, we have them!

AWS gives us another tool, called Contributor Insights. This service can be used to analyze logs, create time series and display this data. That's quite powerful!

First look on Contributor Insights

Go to CloudWatch service, Contributor insights. Click Create rule. In the dropdown Log group(s) select our API logs. For the first time, select Sample rule, API Gateway and Calls by route and HTTP method (CLF). In fact, it doesn't matter if you select JSON or CLF (in this case).

First sample rule

Keep rest of configuration as it is.

Log format: CLF (or JSON if you selected it earlier)  
Contribution: ip  
Aggregation: Count
Enter fullscreen mode Exit fullscreen mode

Click Next and give the rule name. I named mine as APIGW_byCount.

After you click Create rule, you have to wait a few minutes (AWS claims 5) to see any data.

The result

This will build the analysis and aggregation. On the screen you see four different IPs (are they IPs, really? We'll see).

The example above is autogenerated.

Let's see, if we can to create custom one.

Go to editor, click Syntax to change the screen to text editor.

Switch editor type

Our example looks like this:

    "Schema": {
        "Name": "CloudWatchLogRule",
        "Version": 1
    "LogGroupNames": [
    "LogFormat": "JSON",
    "Contribution": {
        "Keys": [
        "Filters": []
    "AggregateOn": "Count"
Enter fullscreen mode Exit fullscreen mode

Here you can start playing with your query.

Let's visualise what we queried

It is time to visualise our work. Click Actions, View in CloudWatch Metrics and select Unique contributors.

Render metrics

In time I received a few more requests from different IPs, and below you can see the visualisation.

Rendered metrics

Let's make it useful

Well... What we see now is... not really useful, right? Right. This visualisation is not really about IPs. It is some mix of many elements. You know what? Forget this. Let's do it right.

What we need to do, is work on the samples provided by AWS and rewrite them, or create our own. Or, and this is what we will do, we will rewrite the logs which API is generating. Well, we should do it, anyway.

Ok. Let's go to the API service, select the proper API, again, got to Stages, Prod and select the Logs/Tracing tab.

Update API logs

Click Enable Access Logging, and use Insert Example for JSON. You will have default log (which actually is different than default default log ;) ). Let's add one more field, like in example below.

{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp", "caller":"$context.identity.caller", "user":"$context.identity.user","requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath", "status":"$context.status","protocol":"$context.protocol", "responseLength":"$context.responseLength","traceId":"$context.xrayTraceId" }
Enter fullscreen mode Exit fullscreen mode

On the end of the string you can find traceId field.

I also slightly modified the Log Destination ARN in order to differentiate the behavior between the logs. It will be more readable for you.

Ok, now let's be sure, that the syntax for rule is like this:

    "AggregateOn": "Count",
    "Contribution": {
        "Filters": [],
        "Keys": [
    "LogFormat": "JSON",
    "LogGroupNames": [
    "Schema": {
        "Name": "CloudWatchLogRule",
        "Version": 1
Enter fullscreen mode Exit fullscreen mode

Be also sure, you use the proper (new one!) log group (remember, yours will be different than mine!)

First of all, you can see totally different log. Shorter, more clearly structurized.

API logs

And how it looks in Contributor Insights? Well, great!

API logs visualization

SAM template

It is time to modify the template.

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: simple Lambda

    Type: AWS::Logs::LogGroup

    Type: AWS::Serverless::Api
      StageName: Prod
      Description: 'Prod stage'
      TracingEnabled: true
      - HttpMethod: '*'
        LoggingLevel: INFO
        ResourcePath: '/*'
        MetricsEnabled: true
        DataTraceEnabled: true
        DestinationArn: !GetAtt accessLogGroup.Arn
        Format: >-
          '{ "requestId":"$context.requestId", 
          "ip": "$context.identity.sourceIp", 
          "traceId":"$context.xrayTraceId" }'

    Type: AWS::Serverless::Function
      Handler: simplefunction.handler
      CodeUri: lambdafunction/
      Runtime: python3.8
        - CloudWatchLambdaInsightsExecutionRolePolicy
      AutoPublishAlias: live
      Description: Simple demo function
      MemorySize: 128
      Timeout: 10
      Tracing: Active
        - !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14"
          Type: Api
              !Ref lambdaDemoApi
            Path: /
            Method: get
Enter fullscreen mode Exit fullscreen mode

As you can see, I added one resource - AccessLogGroup and AccessLogSettings in Properties of lambdaDemoApi.

Now you can go crazy. There is many possibilities to build API Logs and to visualize it. Now you know :)

Cover image by Hebi B. from Pixabay

Top comments (0)