Dickson for AWS Community Builders

Posted on Jan 22, 2023 • Updated on Mar 19

AWS Config Auto Remediation for Configuring S3 Lifecycle Rule

#aws #config #ssm #s3

Amazon Simple Storage Service (Amazon S3) is a popular storage solution offering high availability, scalability and durability. Enterprises around the world have put petabytes of data on S3 for various use cases, ranging from big data analysis to critical data backup. With the increasing number of objects stored in S3 buckets, storage cost becomes a major concern.

To ensure the objects are stored cost-effectively, S3 lifecycle configuration can be configured in every bucket so that all objects are automatically transitioned, archived or expired after a certain time. In this article, we will go through a solution that leverages AWS Config and AWS Systems Manager (SSM). This guarantees all S3 buckets to have at least one active lifecycle rule.

Note: Lifecycle configuration cannot be set on buckets with MFA delete enabled.

The Config rule evaluates the lifecycle rule configuration of all S3 buckets. If a bucket is found non-compliant, AWS Config applies the remediation using a SSM Automation document.

Creating an AWS Config rule

In the AWS Config console, select Rules in the navigation pane and click Add rule. If you cannot find the button, set up the AWS Config first.
Among the AWS managed rules, choose the s3-lifecycle-policy-check rule and click Next.

Note: There is another similar rule called s3-version-lifecycle-policy-check which checks on versioned S3 buckets.
Leave all the values as default and click Next. Optionally, you can specify the parameters such as targetTransitionDays and targetTransitionStorageClass.
Review all the configurations and click Add rule.

The rule is now created and shows up on the console. If there is any noncompliant resource (i.e. S3 bucket with no lifecycle policy configured), you should see the count on the compliance column.

Creating a custom SSM Automation document

Since there is no existing automation for configuring S3 lifecycle rule, we have to write our own document. The automation will execute both the PutBucketLifecycleConfiguration and GetBucketLifecycleConfiguration APIs.

description: |
  ### Document Name - ConfigureS3BucketLifecycleRule

  ## What does this document do?
  This document is used to create or modify the lifecycle rule configuration for an Amazon S3 bucket.

  ## Input Parameters
  * BucketName: (Required) Name of the S3 bucket (not the ARN).
  * TransitionDays: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    * Default: 90
  * TransitionStorageClass: (Optional) Storage class to which the object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * NoncurrentTransitionDays: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    * Default: 30
  * NoncurrentTransitionStorageClass: (Optional) Storage class to which the noncurrent object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * AutomationAssumeRole: (Required) ARN of the role that allows Automation to perform the actions.

  ## Output Parameters
  * GetBucketLifecycleConfiguration.Output - JSON formatted response from the GetBucketLifecycleConfiguration API call
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
outputs:
  - GetBucketLifecycleConfiguration.Output
parameters:
  BucketName:
    type: String
    description: (Required) Name of the S3 bucket (not the ARN).
    allowedPattern: (?=^.{3,63}$)(?!^(\d+\.)+\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$)
  TransitionDays:
    type: Integer
    description: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    default: 90
  TransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  NoncurrentTransitionDays:
    type: Integer
    description: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    default: 30
  NoncurrentTransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the noncurrent object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  AutomationAssumeRole:
    type: String
    description: (Required) ARN of the role that allows Automation to perform the actions.
    allowedPattern: ^arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/[\w+=,.@-]+
mainSteps:
  - name: PutBucketLifecycleConfiguration
    action: "aws:executeAwsApi"
    description: |
      ## PutBucketLifecycleConfiguration
      Creates or modifies the lifecycle configuration for a S3 Bucket.
    isEnd: false
    inputs:
      Service: s3
      Api: PutBucketLifecycleConfiguration
      Bucket: "{{BucketName}}"
      LifecycleConfiguration:
        Rules:
          - Filter:
              Prefix: ""
            ID: "Default Lifecycle Rule"
            Status: Enabled
            Transitions:
              - Days: "{{ TransitionDays }}"
                StorageClass: "{{ TransitionStorageClass }}"
            NoncurrentVersionTransitions:
              - NoncurrentDays: "{{ NoncurrentTransitionDays }}"
                StorageClass: "{{ NoncurrentTransitionStorageClass }}"
    isCritical: true
    maxAttempts: 2
    timeoutSeconds: 600
  - name: GetBucketLifecycleConfiguration
    action: "aws:executeScript"
    description: |
      ## GetBucketLifecycleConfiguration
      Retrieves the S3 lifecycle configuration for a S3 Bucket.
      ## Outputs
      * Output: JSON formatted response from the GetBucketLifecycleConfiguration API call.
    timeoutSeconds: 600
    isCritical: true
    isEnd: true
    inputs:
      Runtime: python3.6
      Handler: validate_s3_bucket_lifecycle_configuration
      InputPayload:
        Bucket: "{{BucketName}}"
        TransitionDays: "{{ TransitionDays }}"
        TransitionStorageClass: "{{ TransitionStorageClass }}"
        NoncurrentTransitionDays: "{{ NoncurrentTransitionDays }}"
        NoncurrentTransitionStorageClass: "{{ NoncurrentTransitionStorageClass }}"
      Script: |-
        import boto3

        def validate_s3_bucket_lifecycle_configuration(event, context):
            s3_client = boto3.client("s3")
            bucket = event["Bucket"]
            transition_days = event["TransitionDays"]
            transition_storage_class = event["TransitionStorageClass"]
            noncurrent_transition_days = event["NoncurrentTransitionDays"]
            noncurrent_transition_storage_class = event["NoncurrentTransitionStorageClass"]

            output = s3_client.get_bucket_lifecycle_configuration(Bucket=bucket)
            updated_rules = output["Rules"]

            if any(
                any(updated_transition["Days"] == transition_days
                    and updated_transition["StorageClass"] == transition_storage_class
                    for updated_transition in updated_rule["Transitions"])
                and any(updated_noncurrent_transition["NoncurrentDays"] == noncurrent_transition_days
                        and updated_noncurrent_transition["StorageClass"] == noncurrent_transition_storage_class
                        for updated_noncurrent_transition in updated_rule["NoncurrentVersionTransitions"])
                and updated_rule["Status"] == "Enabled"
                for updated_rule in updated_rules
            ):
                return {
                    "output":
                    {
                        "message": "Bucket lifecycle configuration successfully set.",
                        "configuration": updated_rules
                    }
                }
            else:
                info = "CONFIGURATION VALUES DO NOT MATCH WITH PARAMETERS PROVIDED VALUES TransitionDays: {}, TransitionStorageClass: {}, NoncurrentTransitionDays: {}, NoncurrentTransitionStorageClass: {}".format(
                    transition_days,
                    transition_storage_class,
                    noncurrent_transition_days,
                    noncurrent_transition_storage_class
                )
                raise Exception(info)
    outputs:
      - Name: Output
        Selector: $.Payload.output
        Type: StringMap

The same document can also be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/ConfigureS3BucketLifecycleRule.yaml.

In the AWS Systems Manager console, select Documents in the navigation pane, click Create document and choose Automation.
For Name, enter ConfigureS3BucketLifecycleRule. Switch to the Editor tab, click Edit and replace the content with the above document. Click Create automation.

Creating an IAM role for the SSM automation

In the IAM console, select Roles in the navigation pane and click Create role.
For Use case, choose Systems Manager. Click Next.
Choose AmazonSSMAutomationRole and click Next.
For Role name, enter s3-configure-bucket-lifecycle-rule. Click Create role.
Click View role.
Click Add permissions and choose Create inline policy.

Switch to the JSON tab, replace the content with the below document. Click Review policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutLifecycleConfiguration",
                "s3:GetLifecycleConfiguration"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

For Name, enter s3-configure-bucket-lifecycle-rule. Click Create policy.

Setting up an automatic remediation for the AWS Config rule

On the Rules page of the AWS Config console, choose the s3-lifecycle-policy-check rule and click View details.
Click Actions and choose Manage remediation.
For Select remediation method, choose Automatic remediation. For Remediation action details, choose ConfigureS3BucketLifecycleRule. For Resource ID parameter, choose BucketName. For AutomationAssumeRole, enter arn:aws:iam::<<aws-account-id>>:role/s3-configure-bucket-lifecycle-rule. Leave other values as default and click Save changes. Optionally, you can specify the parameters such as TransitionDays and TransitionStorageClass.

The remediation is now configured. If there was any noncompliant resource before, it will become compliant after some time. In the future, all new S3 buckets will also have the lifecycle rule configured upon creation.

Advanced usage

The entire solution can be deployed with an AWS CloudFormation template, which can be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/main.yaml.

For enterprise use, this template can be deployed into multiple accounts easily using an AWS CloudFormation StackSet.

Note: If the solution is deployed in multiple regions, it is advised to separate the IAM role creation from the template as IAM resources are global constructs.

DEV Community

AWS Config Auto Remediation for Configuring S3 Lifecycle Rule

Creating an AWS Config rule

Creating a custom SSM Automation document

Creating an IAM role for the SSM automation

Setting up an automatic remediation for the AWS Config rule

Advanced usage

References

Top comments (0)

Read next

Ansible vs Terraform

Spring Boot 3 application on AWS Lambda - Part 2 Introduction to AWS Serverless Java Container

Learning AWS Day by Day — Day 24 — S3 Introduction — Part 2

My Past 1 Year+ with AWS Cognito