Photo by Bjørn Christian Tørrissen
With various serverless and opensource options available, it's easier to spin up a concept into some kind of working model with comparatively quicker than earlier times. With this advancement it becomes essential to be able to effectively monitoring the different components and solution.
What makes monitoring easier than having single logcentral to be able to get a view of required metrics.
Yes, as an engineer, I would like to setup a dashboard with all required information. And functionbeat helps in achieving the same with as simple as writing a lambda to collect the logs from different sources. Functionbeat is one of Elastic's beat family allowing you to be able to stream logs from Kinesis, SQS, Cloudwatch (as of today) to single logcentral.
What we are focusing here is, functionbeat to read each row of cloudwatch logs and stream it to elasticsearch. With functionbeat deployed as serverless lambda to AWS, you should be able to achieve above.
In this post, we focus on collecting logs from cloudwatch group and shipping it to elasticsearch.
lambda -> Cloudwatch Logs -> Functionbeat -> Elasticsearch
A lambda function with any business function generating logs and metrics to cloudwatch.
Download functionbeat distribution
Before deploying cloudwatch, ensure you have AWS profile set-up. Verify that the aws account and region are set correctly using
Create S3 bucket from AWS console for the functionbeat to be uploaded to. The reference to created S3 bucket is provided as
4. For the functionbeat to be deployed, lambda should have required permissions to the AWS resources. Create IAM role with access permissions to
ec2, es, s3
For detailed IAM cloudformation code refer to sample
5. Configure functionbeat in
functionbeat.yaml and setup lambda to add triggers to cloudwatch group defined in #1.
functionbeat-cloudwatch is the name of lambda function and
type representing the trigger type with actual values listed under
triggers configuration. We could configure multiple cloudwatch log group triggers as comma separated values.
- name: fb-cloudwatch
- log_group_name: /aws/lambda/sample-lambda-events
If you would want to deploy the lambda as part of private cloud set-up, look at configuring
virtual_private_cloud with subnet details.
Now, lets look at configure the output of these logs to be shipped to. It only requires to configure the elasticsearch host and port to be configured. Provided the AWS connectivity to elasticsearch is already established, hook into the host. Else, use the local elasticsearch distribution to verify.
If the elasticsearch setup requires credentials, the same can be configured under
password. To verify the setup so far, we could simply run the command
./functionbeat test config -e
- Finally, we are ready to deploy functionbeat to AWS. Deploy command uploads the functionbeat to s3 bucket
functionbeat-deploycreated as part of #3. Use the command:
./functionbeat -v -e -d "*" deploy fb-cloudwatch
This would create cloudformation stack in AWS creating required lambda and resources. Once successfully deployed, you should be able to navigate to cloudwatch and look at the functionbeat
fb-cloudwatch logs. You could enable debug logs on functionbeat to be able to view detailed logs, add config
logging.level: debug to
With any additional changes, you can update the functionbeat using
update command without requiring to recreate the whole function.
./functionbeat -v -e -d "*" update fb-cloudwatch
As part of the lambda function collecting and streaming logs, we could pick the required fields from log source. This could be configured as
processors to the function. Say, if its required to drop field
noMuchUse from log group and enrich with host metadata, set up processor