_{Photo by Bjørn Christian Tørrissen}

With various serverless and opensource options available, it's easier to spin up a concept into some kind of working model with comparatively quicker than earlier times. With this advancement it becomes essential to be able to effectively monitoring the different components and solution.
What makes monitoring easier than having single logcentral to be able to get a view of required metrics.

Functionbeat for Monitoring

Yes, as an engineer, I would like to setup a dashboard with all required information. And functionbeat helps in achieving the same with as simple as writing a lambda to collect the logs from different sources. Functionbeat is one of Elastic's beat family allowing you to be able to stream logs from Kinesis, SQS, Cloudwatch (as of today) to single logcentral.

Collect the Cloudwatch Logs

What we are focusing here is, functionbeat to read each row of cloudwatch logs and stream it to elasticsearch. With functionbeat deployed as serverless lambda to AWS, you should be able to achieve above.

In this post, we focus on collecting logs from cloudwatch group and shipping it to elasticsearch.
lambda -> Cloudwatch Logs -> Functionbeat -> Elasticsearch

Setup a lambda with cloudwatch monitoring

A lambda function with any business function generating logs and metrics to cloudwatch.

Setup Functionbeat lambda

1. Download functionbeat distribution
   https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-installation.html

2. Before deploying cloudwatch, ensure you have AWS profile set-up. Verify that the aws account and region are set correctly using
   aws configure

3. Create S3 bucket from AWS console for the functionbeat to be uploaded to. The reference to created S3 bucket is provided as
   

  functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
  functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy"

4. For the functionbeat to be deployed, lambda should have required permissions to the AWS resources. Create IAM role with access permissions to
ec2, es, s3
For detailed IAM cloudformation code refer to sample

5. Configure functionbeat in functionbeat.yaml and setup lambda to add triggers to cloudwatch group defined in #1. functionbeat-cloudwatch is the name of lambda function and type representing the trigger type with actual values listed under triggers configuration. We could configure multiple cloudwatch log group triggers as comma separated values.

  functionbeat.provider.aws.functions:
    - name: fb-cloudwatch
      enabled: true
      type: cloudwatch_logs

      role: arn:aws:iam::xxawsaccountidxx:role/lambda.functionbeat.role

      triggers:
      - log_group_name: /aws/lambda/sample-lambda-events

If you would want to deploy the lambda as part of private cloud set-up, look at configuring virtual_private_cloud with subnet details.

Now, lets look at configure the output of these logs to be shipped to. It only requires to configure the elasticsearch host and port to be configured. Provided the AWS connectivity to elasticsearch is already established, hook into the host. Else, use the local elasticsearch distribution to verify.

  output.elasticsearch:
    hosts: ["localhost:9200"]
    protocol: http

If the elasticsearch setup requires credentials, the same can be configured under output.elasticsearch with username and password. To verify the setup so far, we could simply run the command

./functionbeat test config -e

1. Finally, we are ready to deploy functionbeat to AWS. Deploy command uploads the functionbeat to s3 bucket functionbeat-deploy created as part of #3. Use the command:

./functionbeat -v -e -d "*" deploy fb-cloudwatch

This would create cloudformation stack in AWS creating required lambda and resources. Once successfully deployed, you should be able to navigate to cloudwatch and look at the functionbeat fb-cloudwatch logs. You could enable debug logs on functionbeat to be able to view detailed logs, add config logging.level: debug to functionbeat.yaml.

With any additional changes, you can update the functionbeat using update command without requiring to recreate the whole function.
./functionbeat -v -e -d "*" update fb-cloudwatch

Improvements

As part of the lambda function collecting and streaming logs, we could pick the required fields from log source. This could be configured as processors to the function. Say, if its required to drop field noMuchUse from log group and enrich with host metadata, set up processor

  processors:
    - include_fields:
      fields: ["id"]
    - drop_fields: 
      fields: ["id.noMuchUse"]
    - add_host_metadata:
      netinfo.enabled: false

References:
Functionbeat
Github