“ I have checked the documents of AWS to export cloudwatch logs to s3 bucket automatically. So I found that it can be possible using lambda functions with eventbridge rules. Pricing of S3, lambda and trigger set is very cheap in comparison to storing the logs in cloudwatch.”
Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging. With Lambda, you can run code for virtually any type of application or backend service. To learn more, read the AWS Lambda.
Amazon EventBridge is a serverless event bus service that you can use to connect your applications with data from a variety of sources. EventBridge delivers a stream of real-time data from your applications, software as a service (SaaS) applications, and AWS services to targets such as AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.
In this post, you will get to know how to automate the export of cloudwatch logs to s3 bucket using lambda with eventbridge trigger. Here I have used a lambda function with eventbridge trigger on it so that the cloudwatch logs get exported to s3 bucket as per schedule on trigger when run every time which automates the process.
You’ll need an Amazon RDS Database for this post. Getting started with RDS provides instructions on how to create a RDS. For this blog, I assume that I have a rds database.
The blog post consists of the following phases:
- Create of Option Group and Parameter Group with Required Values to Get the Logs in Cloudwatch
- Create a Bucket in S3 with Custom Bucket Policy and IAM Role with Required Permission Policy
- Create Lambda Function with Eventbridge Trigger
- Output of Logs in S3 Bucket and Log Stream in Cloudwatch
Phase 1: Create of Option Group and Parameter Group with Required Values to Get the Logs in Cloudwatch
- Open the option group console, create an option group with add option for query parameter. Open the parameter group, create a parameter group with values for parameters for log enabled to cloudwatch. Also modify the rds with the created option and parameter group. Also can check the log groups in cloudwatch console.
Phase 2: Create a Bucket in S3 with Custom Bucket Policy and IAM Role with Required Permission Policy
- Open the S3 console, create a bucket with a custom bucket policy. Open the IAM console, create an IAM role for lambda service with AWS managed permission policy.
- Open the lambda console, create a lambda function with environment variables and required custom parameter values. Open the eventbridge console, create a rule for target as lambda function to run every 5 min to export the cloudwatch logs to s3 bucket.
Delete S3 Bucket, IAM Role, Lambda Function, EventBridge Rule, RDS Database, Cloudwatch Log groups.
I review the pricing and estimated cost of this example.
Cost of RDS = $0.10
Cost of Cloudwatch = $0.0
Cost of Lambda = $0.0
Cost of Simple Storage Service = $0.009
Total Cost = $0.109
In this post, I showed “how to automate the export of cloudwatch logs to s3 bucket using lambda with eventbridge trigger”.
Thanks for reading!
Connect with me: Linkedin