Introduction :
Nowadays, One of the most fundamental security measures when working on a multi-account AWS environment is the ability to consolidate, manage, and analyze logs coming from various AWS Services in multiple accounts and multiple AWS Regions in one single place.
Solution Overview :
Based on the necessity to create a single Dashboard from where our Security Team can analyze all the logs coming from multiple AWS Application Accounts. We ended up with the following solution :
Using the above solution we were able to stream CloudWatch logs for a specific set of AWS Services (for example: AWS WAF) from multiple Application Accounts to a Centralized Log Archive Account.
Solution Setup :
let's start by setting up the required resources in our Centralized Log Account, then we will move to the Application Account
Centralized Log Account :
We are going to create the following resources :
1- A Centralized Kinesis Firehose Stream & S3 Bucket :
Using the AWS Console, create your Kinesis Firehose Stream by setting the Source as Direct Put and the Destination as Amazon S3.
You can choose an already existing Bucket as a destination otherwise you can create a new one.
For the rest you can keep the default values, otherwise you can follow the AWS Documentation for any further customizations : Link
2- IAM Role for CloudWatch Logs Destination :
To enable the CloudWatch Logs Destination which We are going to create in the next step to send data to the Kinesis Firehose Stream We already setup in the previous step, We need to create a IAM Role "CWLtoKinesisFirehoseRole" with the following permissions:
{
"Statement":[
{
"Effect":"Allow",
"Action":["firehose:*"],
"Resource":["arn:aws:firehose:region:LogAccountID:*"]
}
]
}
With the following Trust Relationships to allow the CloudWatch service to assume it :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
3- CloudWatch Logs Destination :
The CloudWatch Logs Destination will work as the Access Point for your remote AWS Accounts to stream their logs to your centralized Kinesis Firehose in the Log Account.
The CloudWatch Logs Destination is a regional resource but can stream data to a Kinesis Firehose Stream in a different region, So you can create multiple CloudWatch Logs Destinations in different regions targeting your Centralized Kinesis Firehose Stream "This is what we've done in our case".
The necessity to create Multiple CloudWatch Logs Destinations in different regions is based on the regions from where you want to stream logs in your Application Accounts.
The Application Accounts should stream CloudWatch logs to the Centralized Log Account using the CloudWatch Logs Destination in the same region.
We are going to create it using the CLI as there is no way to do so using the AWS Console.
CLI Command :
aws logs put-destination --destination-name "demoFirehoseCrossAccount" --target-arn "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT" --role-arn "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole"
target-arn and role-arn are refering to the Kinesis Firehose stream and the IAM Role we've created in the previous steps.
Command Result :
{
"destination": {
"destinationName": "demoFirehoseCrossAccount",
"targetArn": "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT",
"roleArn": "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole",
"arn": "arn:aws:logs:region:LogAccountID:destination:demoFirehoseCrossAccount",
"creationTime": 1675210769461
}
}
Save the command result somewhere as we are going to need the Destination ARN during resources creation in the Application Account.
4- CloudWatch Logs Destination Policy :
Each CloudWatch Logs Destination should have a policy attached to it.
Using the Destination Policy, you can decide which Remote Accounts can stream their logs through this CloudWatch Logs Destination.
Create a policy.json file with the following content, to allow remote Account/s to create CloudWatch Subscription Filters targeting the CloudWatch Logs Destination indicated in the Resource section of the Policy :
{
"Version" : "2012-10-17",
"Statement" : [
{
"Sid" : "",
"Effect" : "Allow",
"Principal" : {
"AWS" : "ApplicationAccountID"
},
"Action" : "logs:PutSubscriptionFilter",
"Resource" : "arn:aws:logs:region:logAccountID:destination:demoFirehoseCrossAccount"
}
]
}
CloudWatch Subscription Filters are what we are going to create in the Application Accounts
The AWS Principal should be the Account ID/IDs of the Application Accounts and Resource is the ARN **of the **CloudWatch Logs Destination created in the previous step.
CLI Command :
aws logs put-destination-policy --destination-name "DEMO-FIREHOSE-CROSSACCOUNT" --access-policy "file://policy.json"
Application Account :
In your Application Accounts all you need to create is a CloudWatch Kinesis Firehose Subscription Filter for the CloudWatch Log Group you want to stream to your Centralized Log Account :
as a Destination choose Cross-Account and then insert the CloudWatch Logs Destination ARN for the one created in the Centralized Log Account :
Finally :
Now, We've all the required resources to stream CloudWatch Logs from multiple Application Accounts to a Centralized Log Account.
If you've the necessity to elaborate your data before storing it in S3, Kinesis Data Firehose can invoke a Lambda function to transform incoming source data and deliver the transformed data to destination LINK.
Top comments (0)