“ I have checked the documents of AWS to export cloudwatch logs to s3 bucket automatically. So I found that it can be possible using lambda functions with eventbridge rules. Pricing of S3, lambda and trigger set is very cheap in comparison to storing the logs in cloudwatch.”

Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging. With Lambda, you can run code for virtually any type of application or backend service. To learn more, read the AWS Lambda.

Amazon EventBridge is a serverless event bus service that you can use to connect your applications with data from a variety of sources. EventBridge delivers a stream of real-time data from your applications, software as a service (SaaS) applications, and AWS services to targets such as AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.

In this post, you will get to know how to automate the export of cloudwatch logs to s3 bucket using lambda with eventbridge trigger. Here I have used a lambda function with eventbridge trigger on it so that the cloudwatch logs get exported to s3 bucket as per schedule on trigger when run every time which automates the process.

Prerequisites

You’ll need an Amazon RDS Database for this post. Getting started with RDS provides instructions on how to create a RDS. For this blog, I assume that I have a rds database.

Architecture Overview

The architecture diagram shows the overall deployment architecture with data flow, amazon RDS, cloudwatch logs, s3 bucket, eventbridge rule, lambda function.

Solution overview

The blog post consists of the following phases:

1. Create of Option Group and Parameter Group with Required Values to Get the Logs in Cloudwatch
2. Create a Bucket in S3 with Custom Bucket Policy and IAM Role with Required Permission Policy
3. Create Lambda Function with Eventbridge Trigger
4. Output of Logs in S3 Bucket and Log Stream in Cloudwatch

I have a rds database as below →

Phase 1: Create of Option Group and Parameter Group with Required Values to Get the Logs in Cloudwatch

1. Open the option group console, create an option group with add option for query parameter. Open the parameter group, create a parameter group with values for parameters for log enabled to cloudwatch. Also modify the rds with the created option and parameter group. Also can check the log groups in cloudwatch console.

Phase 2: Create a Bucket in S3 with Custom Bucket Policy and IAM Role with Required Permission Policy

1. Open the S3 console, create a bucket with a custom bucket policy. Open the IAM console, create an IAM role for lambda service with AWS managed permission policy.

Phase 3: Create Lambda Function with Eventbridge Trigger

1. Open the lambda console, create a lambda function with environment variables and required custom parameter values. Open the eventbridge console, create a rule for target as lambda function to run every 5 min to export the cloudwatch logs to s3 bucket.

Phase 4: Output of Logs in S3 Bucket and Log Stream in Cloudwatch

Clean-up

Delete S3 Bucket, IAM Role, Lambda Function, EventBridge Rule, RDS Database, Cloudwatch Log groups.

Pricing

I review the pricing and estimated cost of this example.

Cost of RDS = $0.10

Cost of Cloudwatch = $0.0

Cost of Lambda = $0.0

Cost of Simple Storage Service = $0.009

Total Cost = $0.109

Summary

In this post, I showed “how to automate the export of cloudwatch logs to s3 bucket using lambda with eventbridge trigger”.

For more details on Amazon EventBridge Rules, Checkout Get started Amazon EventBridge Rules, open the Amazon EventBridge console. To learn more, read the Amazon EventBridge documentation.

For more details on AWS Lambda, Checkout Get started AWS Lambda, open the AWS Lambda console. To learn more, read the AWS Lambda documentation.

Thanks for reading!

Connect with me: Linkedin