DEV Community 👩‍💻👨‍💻

DEV Community 👩‍💻👨‍💻 is a community of 963,274 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Cover image for AWS CloudWatch Contributor Insights. Is it useful?
Paweł Piwosz for AWS Community Builders

Posted on • Updated on

AWS CloudWatch Contributor Insights. Is it useful?

Check logs with Contributor Insights

Right! So we have logs. We can use CloudWatch Logs Insights to query it, we can observe it through CloudWatch logs, and we can do many, many things with them now. Finally, we have them!

AWS gives us another tool, called Contributor Insights. This service can be used to analyze logs, create time series and display this data. That's quite powerful!

First look on Contributor Insights

Go to CloudWatch service, Contributor insights. Click Create rule. In the dropdown Log group(s) select our API logs. For the first time, select Sample rule, API Gateway and Calls by route and HTTP method (CLF). In fact, it doesn't matter if you select JSON or CLF (in this case).

First sample rule

Keep rest of configuration as it is.

Log format: CLF (or JSON if you selected it earlier)  
Contribution: ip  
Aggregation: Count
Enter fullscreen mode Exit fullscreen mode

Click Next and give the rule name. I named mine as APIGW_byCount.

After you click Create rule, you have to wait a few minutes (AWS claims 5) to see any data.

The result

This will build the analysis and aggregation. On the screen you see four different IPs (are they IPs, really? We'll see).

The example above is autogenerated.

Let's see, if we can to create custom one.

Go to editor, click Syntax to change the screen to text editor.

Switch editor type

Our example looks like this:

{
    "Schema": {
        "Name": "CloudWatchLogRule",
        "Version": 1
    },
    "LogGroupNames": [
        "API-Gateway-Execution-Logs_9fcv1s573l/Prod"
    ],
    "LogFormat": "JSON",
    "Contribution": {
        "Keys": [
            "$.ip"
        ],
        "Filters": []
    },
    "AggregateOn": "Count"
}
Enter fullscreen mode Exit fullscreen mode

Here you can start playing with your query.

Let's visualise what we queried

It is time to visualise our work. Click Actions, View in CloudWatch Metrics and select Unique contributors.

Render metrics

In time I received a few more requests from different IPs, and below you can see the visualisation.

Rendered metrics

Let's make it useful

Well... What we see now is... not really useful, right? Right. This visualisation is not really about IPs. It is some mix of many elements. You know what? Forget this. Let's do it right.

What we need to do, is work on the samples provided by AWS and rewrite them, or create our own. Or, and this is what we will do, we will rewrite the logs which API is generating. Well, we should do it, anyway.

Ok. Let's go to the API service, select the proper API, again, got to Stages, Prod and select the Logs/Tracing tab.

Update API logs

Click Enable Access Logging, and use Insert Example for JSON. You will have default log (which actually is different than default default log ;) ). Let's add one more field, like in example below.

{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp", "caller":"$context.identity.caller", "user":"$context.identity.user","requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath", "status":"$context.status","protocol":"$context.protocol", "responseLength":"$context.responseLength","traceId":"$context.xrayTraceId" }
Enter fullscreen mode Exit fullscreen mode

On the end of the string you can find traceId field.

I also slightly modified the Log Destination ARN in order to differentiate the behavior between the logs. It will be more readable for you.

Ok, now let's be sure, that the syntax for rule is like this:

{
    "AggregateOn": "Count",
    "Contribution": {
        "Filters": [],
        "Keys": [
            "$.ip"
        ]
    },
    "LogFormat": "JSON",
    "LogGroupNames": [
        "API-Gateway-Execution-Logs_9fcv1s573l"
    ],
    "Schema": {
        "Name": "CloudWatchLogRule",
        "Version": 1
    }
}
Enter fullscreen mode Exit fullscreen mode

Be also sure, you use the proper (new one!) log group (remember, yours will be different than mine!)

First of all, you can see totally different log. Shorter, more clearly structurized.

API logs

And how it looks in Contributor Insights? Well, great!

API logs visualization

SAM template

It is time to modify the template.

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: simple Lambda

Resources:
  accessLogGroup:
    Type: AWS::Logs::LogGroup

  lambdaDemoApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Description: 'Prod stage'
      TracingEnabled: true
      MethodSettings:
      - HttpMethod: '*'
        LoggingLevel: INFO
        ResourcePath: '/*'
        MetricsEnabled: true
        DataTraceEnabled: true
      AccessLogSetting:
        DestinationArn: !GetAtt accessLogGroup.Arn
        Format: >-
          '{ "requestId":"$context.requestId", 
          "ip": "$context.identity.sourceIp", 
          "caller":"$context.identity.caller", 
          "user":"$context.identity.user", 
          "requestTime":"$context.requestTime", 
          "httpMethod":"$context.httpMethod", 
          "resourcePath":"$context.resourcePath", 
          "status":"$context.status", 
          "protocol":"$context.protocol", 
          "responseLength":"$context.responseLength", 
          "traceId":"$context.xrayTraceId" }'

  lambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: simplefunction.handler
      CodeUri: lambdafunction/
      Runtime: python3.8
      Policies:
        - CloudWatchLambdaInsightsExecutionRolePolicy
      AutoPublishAlias: live
      Description: Simple demo function
      MemorySize: 128
      Timeout: 10
      Tracing: Active
      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14"
      Events:
        simpleApi:
          Type: Api
          Properties:
            RestApiId: 
              !Ref lambdaDemoApi
            Path: /
            Method: get
Enter fullscreen mode Exit fullscreen mode

As you can see, I added one resource - AccessLogGroup and AccessLogSettings in Properties of lambdaDemoApi.


Now you can go crazy. There is many possibilities to build API Logs and to visualize it. Now you know :)


Cover image by Hebi B. from Pixabay

Top comments (0)

🏗 We built a 100% open source community software called Forem.

You can contribute to the codebase or host your own.