Amazon Simple Storage Service (Amazon S3) is a popular storage solution offering high availability, scalability and durability. Enterprises around the world have put petabytes of data on S3 for various use cases, ranging from big data analysis to critical data backup. With the increasing number of objects stored in S3 buckets, storage cost becomes a major concern.
To ensure the objects are stored cost-effectively, S3 lifecycle configuration can be configured in every bucket so that all objects are automatically transitioned, archived or expired after a certain time. In this article, we will go through a solution that leverages AWS Config and AWS Systems Manager (SSM). This guarantees all S3 buckets to have at least one active lifecycle rule.
Note: Lifecycle configuration cannot be set on buckets with MFA delete enabled.
The Config rule evaluates the lifecycle rule configuration of all S3 buckets. If a bucket is found non-compliant, AWS Config applies the remediation using a SSM Automation document.
Creating an AWS Config rule
In the AWS Config console, select Rules in the navigation pane and click Add rule. If you cannot find the button, set up the AWS Config first.
-
Among the AWS managed rules, choose the
s3-lifecycle-policy-checkrule and click Next.
Note: There is another similar rule called
s3-version-lifecycle-policy-checkwhich checks on versioned S3 buckets. Leave all the values as default and click Next. Optionally, you can specify the parameters such as
targetTransitionDaysandtargetTransitionStorageClass.
Review all the configurations and click Add rule.
The rule is now created and shows up on the console. If there is any noncompliant resource (i.e. S3 bucket with no lifecycle policy configured), you should see the count on the compliance column.
Creating a custom SSM Automation document
Since there is no existing automation for configuring S3 lifecycle rule, we have to write our own document. The automation will execute both the PutBucketLifecycleConfiguration and GetBucketLifecycleConfiguration APIs.
description: |
### Document Name - ConfigureS3BucketLifecycleRule
## What does this document do?
This document is used to create or modify the lifecycle rule configuration for an Amazon S3 bucket.
## Input Parameters
* BucketName: (Required) Name of the S3 bucket (not the ARN).
* TransitionDays: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
* Default: 90
* TransitionStorageClass: (Optional) Storage class to which the object is transitioned.
* Default: "INTELLIGENT_TIERING"
* NoncurrentTransitionDays: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
* Default: 30
* NoncurrentTransitionStorageClass: (Optional) Storage class to which the noncurrent object is transitioned.
* Default: "INTELLIGENT_TIERING"
* AutomationAssumeRole: (Required) ARN of the role that allows Automation to perform the actions.
## Output Parameters
* GetBucketLifecycleConfiguration.Output - JSON formatted response from the GetBucketLifecycleConfiguration API call
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
outputs:
- GetBucketLifecycleConfiguration.Output
parameters:
BucketName:
type: String
description: (Required) Name of the S3 bucket (not the ARN).
allowedPattern: (?=^.{3,63}$)(?!^(\d+\.)+\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$)
TransitionDays:
type: Integer
description: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
default: 90
TransitionStorageClass:
type: String
description: (Optional) Storage class to which the object is transitioned.
default: INTELLIGENT_TIERING
allowedValues:
- STANDARD_IA
- INTELLIGENT_TIERING
- ONEZONE_IA
- GLACIER
- GLACIER_IR
- DEEP_ARCHIVE
NoncurrentTransitionDays:
type: Integer
description: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
default: 30
NoncurrentTransitionStorageClass:
type: String
description: (Optional) Storage class to which the noncurrent object is transitioned.
default: INTELLIGENT_TIERING
allowedValues:
- STANDARD_IA
- INTELLIGENT_TIERING
- ONEZONE_IA
- GLACIER
- GLACIER_IR
- DEEP_ARCHIVE
AutomationAssumeRole:
type: String
description: (Required) ARN of the role that allows Automation to perform the actions.
allowedPattern: ^arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/[\w+=,.@-]+
mainSteps:
- name: PutBucketLifecycleConfiguration
action: "aws:executeAwsApi"
description: |
## PutBucketLifecycleConfiguration
Creates or modifies the lifecycle configuration for a S3 Bucket.
isEnd: false
inputs:
Service: s3
Api: PutBucketLifecycleConfiguration
Bucket: "{{BucketName}}"
LifecycleConfiguration:
Rules:
- Filter:
Prefix: ""
ID: "Default Lifecycle Rule"
Status: Enabled
Transitions:
- Days: "{{ TransitionDays }}"
StorageClass: "{{ TransitionStorageClass }}"
NoncurrentVersionTransitions:
- NoncurrentDays: "{{ NoncurrentTransitionDays }}"
StorageClass: "{{ NoncurrentTransitionStorageClass }}"
isCritical: true
maxAttempts: 2
timeoutSeconds: 600
- name: GetBucketLifecycleConfiguration
action: "aws:executeScript"
description: |
## GetBucketLifecycleConfiguration
Retrieves the S3 lifecycle configuration for a S3 Bucket.
## Outputs
* Output: JSON formatted response from the GetBucketLifecycleConfiguration API call.
timeoutSeconds: 600
isCritical: true
isEnd: true
inputs:
Runtime: python3.6
Handler: validate_s3_bucket_lifecycle_configuration
InputPayload:
Bucket: "{{BucketName}}"
TransitionDays: "{{ TransitionDays }}"
TransitionStorageClass: "{{ TransitionStorageClass }}"
NoncurrentTransitionDays: "{{ NoncurrentTransitionDays }}"
NoncurrentTransitionStorageClass: "{{ NoncurrentTransitionStorageClass }}"
Script: |-
import boto3
def validate_s3_bucket_lifecycle_configuration(event, context):
s3_client = boto3.client("s3")
bucket = event["Bucket"]
transition_days = event["TransitionDays"]
transition_storage_class = event["TransitionStorageClass"]
noncurrent_transition_days = event["NoncurrentTransitionDays"]
noncurrent_transition_storage_class = event["NoncurrentTransitionStorageClass"]
output = s3_client.get_bucket_lifecycle_configuration(Bucket=bucket)
updated_rules = output["Rules"]
if any(
any(updated_transition["Days"] == transition_days
and updated_transition["StorageClass"] == transition_storage_class
for updated_transition in updated_rule["Transitions"])
and any(updated_noncurrent_transition["NoncurrentDays"] == noncurrent_transition_days
and updated_noncurrent_transition["StorageClass"] == noncurrent_transition_storage_class
for updated_noncurrent_transition in updated_rule["NoncurrentVersionTransitions"])
and updated_rule["Status"] == "Enabled"
for updated_rule in updated_rules
):
return {
"output":
{
"message": "Bucket lifecycle configuration successfully set.",
"configuration": updated_rules
}
}
else:
info = "CONFIGURATION VALUES DO NOT MATCH WITH PARAMETERS PROVIDED VALUES TransitionDays: {}, TransitionStorageClass: {}, NoncurrentTransitionDays: {}, NoncurrentTransitionStorageClass: {}".format(
transition_days,
transition_storage_class,
noncurrent_transition_days,
noncurrent_transition_storage_class
)
raise Exception(info)
outputs:
- Name: Output
Selector: $.Payload.output
Type: StringMap
The same document can also be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/ConfigureS3BucketLifecycleRule.yaml.
In the AWS Systems Manager console, select Documents in the navigation pane, click Create document and choose Automation.
For Name, enter
ConfigureS3BucketLifecycleRule. Switch to the Editor tab, click Edit and replace the content with the above document. Click Create automation.
Creating an IAM role for the SSM automation
In the IAM console, select Roles in the navigation pane and click Create role.
For Use case, choose Systems Manager. Click Next.
Choose
AmazonSSMAutomationRoleand click Next.
For Role name, enter
s3-configure-bucket-lifecycle-rule. Click Create role.
Click View role.
Click Add permissions and choose Create inline policy.
Switch to the JSON tab, replace the content with the below document. Click Review policy.```json
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:PutLifecycleConfiguration",
"s3:GetLifecycleConfiguration"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
```
- For Name, enter
s3-configure-bucket-lifecycle-rule. Click Create policy.
Setting up an automatic remediation for the AWS Config rule
On the Rules page of the AWS Config console, choose the
s3-lifecycle-policy-checkrule and click View details.
Click Actions and choose Manage remediation.
For Select remediation method, choose Automatic remediation. For Remediation action details, choose ConfigureS3BucketLifecycleRule. For Resource ID parameter, choose BucketName. For
AutomationAssumeRole, enterarn:aws:iam::<<aws-account-id>>:role/s3-configure-bucket-lifecycle-rule. Leave other values as default and click Save changes. Optionally, you can specify the parameters such asTransitionDaysandTransitionStorageClass.
The remediation is now configured. If there was any noncompliant resource before, it will become compliant after some time. In the future, all new S3 buckets will also have the lifecycle rule configured upon creation.
Advanced usage
The entire solution can be deployed with an AWS CloudFormation template, which can be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/main.yaml.
For enterprise use, this template can be deployed into multiple accounts easily using an AWS CloudFormation StackSet.
Note: If the solution is deployed in multiple regions, it is advised to separate the IAM role creation from the template as IAM resources are global constructs.
References
- https://docs.aws.amazon.com/config/latest/developerguide/s3-lifecycle-policy-check.html
- https://docs.aws.amazon.com/systems-manager/latest/userguide/create-ssm-console.html
- https://docs.aws.amazon.com/config/latest/developerguide/remediation.html#setup-autoremediation
- https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console
Top comments (0)