DEV Community

Cover image for Cross-Account log data sharing using Kinesis Data Firehose
Peter Eskandar for AWS Community Builders

Posted on

Cross-Account log data sharing using Kinesis Data Firehose

Introduction :

Nowadays, One of the most fundamental security measures when working on a multi-account AWS environment is the ability to consolidate, manage, and analyze logs coming from various AWS Services in multiple accounts and multiple AWS Regions in one single place.


Solution Overview :

Based on the necessity to create a single Dashboard from where our Security Team can analyze all the logs coming from multiple AWS Application Accounts. We ended up with the following solution :

Image description

Using the above solution we were able to stream CloudWatch logs for a specific set of AWS Services (for example: AWS WAF) from multiple Application Accounts to a Centralized Log Archive Account.


Solution Setup :

let's start by setting up the required resources in our Centralized Log Account, then we will move to the Application Account

Centralized Log Account :

We are going to create the following resources :

1- A Centralized Kinesis Firehose Stream & S3 Bucket :

Using the AWS Console, create your Kinesis Firehose Stream by setting the Source as Direct Put and the Destination as Amazon S3.

Image description

You can choose an already existing Bucket as a destination otherwise you can create a new one.

For the rest you can keep the default values, otherwise you can follow the AWS Documentation for any further customizations : Link


2- IAM Role for CloudWatch Logs Destination :

To enable the CloudWatch Logs Destination which We are going to create in the next step to send data to the Kinesis Firehose Stream We already setup in the previous step, We need to create a IAM Role "CWLtoKinesisFirehoseRole" with the following permissions:



{
    "Statement":[
      {
        "Effect":"Allow",
        "Action":["firehose:*"],
        "Resource":["arn:aws:firehose:region:LogAccountID:*"]
      }
    ]
}


Enter fullscreen mode Exit fullscreen mode

With the following Trust Relationships to allow the CloudWatch service to assume it :



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}


Enter fullscreen mode Exit fullscreen mode

3- CloudWatch Logs Destination :

The CloudWatch Logs Destination will work as the Access Point for your remote AWS Accounts to stream their logs to your centralized Kinesis Firehose in the Log Account.

The CloudWatch Logs Destination is a regional resource but can stream data to a Kinesis Firehose Stream in a different region, So you can create multiple CloudWatch Logs Destinations in different regions targeting your Centralized Kinesis Firehose Stream "This is what we've done in our case".

The necessity to create Multiple CloudWatch Logs Destinations in different regions is based on the regions from where you want to stream logs in your Application Accounts.

The Application Accounts should stream CloudWatch logs to the Centralized Log Account using the CloudWatch Logs Destination in the same region.

We are going to create it using the CLI as there is no way to do so using the AWS Console.

CLI Command :



aws logs put-destination --destination-name "demoFirehoseCrossAccount" --target-arn "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT" --role-arn "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole"


Enter fullscreen mode Exit fullscreen mode

target-arn and role-arn are refering to the Kinesis Firehose stream and the IAM Role we've created in the previous steps.

Command Result :



        {
            "destination": {
                "destinationName": "demoFirehoseCrossAccount",
                "targetArn": "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT",
                "roleArn": "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole",
                "arn": "arn:aws:logs:region:LogAccountID:destination:demoFirehoseCrossAccount",
                "creationTime": 1675210769461
            }
        }


Enter fullscreen mode Exit fullscreen mode

Save the command result somewhere as we are going to need the Destination ARN during resources creation in the Application Account.


4- CloudWatch Logs Destination Policy :

Each CloudWatch Logs Destination should have a policy attached to it.

Using the Destination Policy, you can decide which Remote Accounts can stream their logs through this CloudWatch Logs Destination.

Create a policy.json file with the following content, to allow remote Account/s to create CloudWatch Subscription Filters targeting the CloudWatch Logs Destination indicated in the Resource section of the Policy :



{ 
        "Version" : "2012-10-17", 
        "Statement" : [ 
            { 
                "Sid" : "", 
                "Effect" : "Allow",
                "Principal" : {
                   "AWS" : "ApplicationAccountID"
                },
                "Action" : "logs:PutSubscriptionFilter", 
                "Resource" : "arn:aws:logs:region:logAccountID:destination:demoFirehoseCrossAccount"
            } 
        ] 
    }


Enter fullscreen mode Exit fullscreen mode

CloudWatch Subscription Filters are what we are going to create in the Application Accounts

The AWS Principal should be the Account ID/IDs of the Application Accounts and Resource is the ARN **of the **CloudWatch Logs Destination created in the previous step.

CLI Command :



aws logs put-destination-policy --destination-name "DEMO-FIREHOSE-CROSSACCOUNT" --access-policy "file://policy.json"


Enter fullscreen mode Exit fullscreen mode

Application Account :

In your Application Accounts all you need to create is a CloudWatch Kinesis Firehose Subscription Filter for the CloudWatch Log Group you want to stream to your Centralized Log Account :

Image description

as a Destination choose Cross-Account and then insert the CloudWatch Logs Destination ARN for the one created in the Centralized Log Account :

Image description


Finally :

Now, We've all the required resources to stream CloudWatch Logs from multiple Application Accounts to a Centralized Log Account.

If you've the necessity to elaborate your data before storing it in S3, Kinesis Data Firehose can invoke a Lambda function to transform incoming source data and deliver the transformed data to destination LINK.

Image description


Helpful Links :

Top comments (0)