Receiving Slack notifications when CloudTrail logging gets turned off

When someone turns off CloudTrail logs, it should be a warning sign, which we want to know when it happens. Luckily we can use EventBridge to route CloudTrail events to various targets and take the necessary steps to remediate the issue.

1. Problem statement

Amazon CloudTrail is the surveillance camera for our accounts. It records every API call that any users or roles make. If we have multiple accounts set up in AWS Organizations, we can create a central trail in the management account. We can then enable logging to all accounts and all regions. Or, if we use Control Tower to set up the account structure, we don't need to do anything because it will automatically create the trail.

It's needless to say that we should always have CloudTrail logging turned on. If someone turns off logging, we won't see who did what and when in our accounts. If anybody with the relevant permissions does anything, we will miss important information for our investigations.

This way, we want to know when someone turns off CloudTrail. We can also restart logging automatically if relevant to the business use case.

2. Solution overview

In this solution we will create an EventBridge rule that listens to the StopLogging CloudTrail event. When we have a multi-account setup, it should be in the management account that contains the trail.

After logging has stopped, EventBridge will route a message to a dedicated Slack channel with the user's name, account number and the time of the event. Optionally, it will also invoke a Lambda function, which will re-enable logging in CloudTrail.

This post won't discuss how to create Slack channels and apps and how to enable incoming webhooks. I won't go into detail about creating Lambda functions and EventBridge rules either. Instead, I'll highlight some key points we want to consider when implementing the architecture.

I assume the solution's components (trail, Lambda function, EventBridge rule) are in the same region.

3. EventBridge

As discussed in the last paragraph, we'll create an EventBridge rule that listens to the StopLogging CloudTrail event.

Some key points to consider are the following.

3.1. Use the default bus

We should create the rule in the default bus because AWS services send their events there.

3.2. Creating a connection

The Slack channel webhook is an external API, so we should first create an API destination. Before doing that, we'll need a connection.

We can create a connection named TestConnection in the Integration/API destinations section on the EventBridge page. The new connection can look like this:

Slack uses a bot user to forward webhook notifications to the associated channel. The webhook endpoint authentication occurs with a token, which we must specify in the connection setup. The authorization type is API Key, the header's name is Authorization, and the value should be in Bearer TOKEN format. We can get the token on the Slack app page from the Features/Oauth & Permissions section.

The status of the connections should now be Authorized. Deauthorized indicates that the connection setup was unsuccessful.

3.3. Creating an API destination

Next, we can create the API destination, and let's creatively call it Test.

The destination endpoint is the webhook URL, and we should select POST as the method. The TestConnection we created above is available now, and we can choose it from the dropdown.

If everything goes well, the API destination's status will be Active.

We have now prepared EventBridge to make authorized calls to the Slack channel's incoming webhook.

3.4. Creating a rule

We are ready to create the rule in EventBridge!

Event pattern

The event pattern should look like this:

{
  "source": ["aws.cloudtrail"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["cloudtrail.amazonaws.com"],
    "eventName": ["StopLogging"]
  }
}

The source and eventSource properties are pointing at CloudTrail because we want to catch the StopLogging event. CloudTrail emits this event when someone turns off logging.

Slack target

Let's add the first target to the rule. The target type is EventBridge API destination for the Slack webhook endpoint, and we should select the one we created above. It's called Test in this example, and the corresponding connection (TestConnection) will auto-populate the relevant fields.

An API destination is a separate resource, so EventBridge will need permission to invoke it. EventBridge will create a new IAM role if we don't already have one. Its permission policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "events:InvokeApiDestination"
      ],
      "Resource": [
        "arn:aws:events:us-east-1:123456789012:api-destination/Test/*"
      ]
    }
  ]
}

Input transformer

Slack's incoming webhook expects an object with a text property. We should define what we want to see in the channel in this field. In this case, it's a plain string without fancy formatting: USER_NAME/ROLE_NAME turned off CloudTrail logging in account 123456789012 at 2023-08-07T11:31:21Z.

For this to happen, we should configure the target input and create a custom output in the Additional settings section on the Select target(s) page. The configurator uses JSON path, so the Input path field will be very similar to what we see in Step Functions.

If we want EventBridge to send the above message, we can write something like this:

{
  "account": "$.account",
  "dateTime": "$.detail.eventTime",
  "user": "$.detail.userIdentity.principalId"
}

The values come from the CloudTrail event object EventBridge receives.

In the following textbox, the Template will define the message format:

{
  "text": "<user> turned off CloudTrail logging in account <account> at <dateTime>"
}

The <KEY> element refers to the relevant property key defined in the input path object.

Lambda function target

If we want to restart logging automatically, we can add a second target that can be a Lambda function (see next section). It's an AWS Service target type, and there shouldn't be any problem finding it from the dropdown (provided that we have previously created it).

EventBridge will automatically add the lambda:InvokeFunction permission to the function's resource-based policy. The statement will allow the EventBridge rule to invoke the function.

4. Automating logging restart

Let's see some minimal Node.js code for the function that we can use to restart CloudTrail logging.

import { CloudTrailClient, StartLoggingCommand } from "@aws-sdk/client-cloudtrail";

const client = new CloudTrailClient();

export const handler = async (event) => {
  const trail = event.detail.requestParameters.name;

  const input = {
    Name: trail,
  };
  const command = new StartLoggingCommand(input);

  try {
    const response = await client.send(command);
    return response;
  } catch (error) {
    console.error('Error: ', error.message);
    throw error;
  }
};

The only thing worth mentioning here is that we want to know the trail where we want to start the logging. We can get this information from the event object and store it in the trail variable. It will be the same event as discussed in the input path section.

To make the code work, we should add the cloudtrail:StartLogging permission to the function's execution role.

5. It should work!

We can now try the solution.

If we stop logging in CloudTrail, we should receive a message within a few seconds in the Slack channel specified in the webhook settings. If we add the second trigger to automate logging restart, we should see a green Logging indicator in the General details section on the CloudTrail page.

6. Considerations

Of course, Slack is not the only way to receive notifications. EventBridge integrates with many target services, both AWS and 3rd parties. For example, we can set up email or phone text message alerts in SNS, or add a different target if business needs require.

7. Summary

CloudTrail logging should be a key element in any company's security management. If we turn off logging, we won't have a history of what happened in our accounts. To meet security and compliance requirements, it might be a good idea to create notifications and trigger automated fixes in case someone turns off trail logging.

One option is to send Slack notifications from EventBridge and use a Lambda function to re-enable logging automatically.

8. Further reading

Creating a trail - How to create a CloudTrail trail

Sending messages using Incoming Webhooks - Enabling Slack webhooks

Getting started with Lambda - How to create a Lambda function

Creating Amazon EventBridge rules that react to events - Everything about EventBridge rules AWS style

API destination partners - Slack - Some guidelines for setting up Slack as an API destination in EventBridge

Amazon EventBridge input transformation - Lots of information on input transformation