Our scenario?
We have an existing S3 bucket in an AWS account (A) and we have to somehow move the existing data onto AWS account (B) because of internal requirements or compliance decisions. Before the data landing onto the destination S3 bucket, it must be encrypted with an AWS KMS key which belongs to the destination account.
For this post, I am going to use S3 Batch Replication and show steps by steps from AWS console.
When to use S3 Batch Replication?
As per AWS official documentation, here are the main points:
- Existing objects
- Objects that failed to replicate previously
- Objects that were already replicated
- Replicas that were created from a replication rule
The 1st point is related to our scenario.
Prerequisites
- Enable bucket versioning on both source and destination S3 buckets.
- The service, which is Amazon S3, must have permissions to perform the tasks.
- Destination account (B) must allow Source acccount (A) to use its AWS KMS encryption key.
Preparation
Enable Versioning
To be applied on both Source and Destination AWS accounts
- Go to your S3 bucket and locate Properties tab from AWS console.
- From Bucket Versioning, ensure the value is
Enabled.
Source Account A
Create a new S3 bucket
Let's quickly create a new bucket for storing replication reports or you can use any existing ones.
aws s3api create-bucket \
--bucket <S3_BUCKET_REPORT_A> \
--region ap-southeast-1 \
--create-bucket-configuration LocationConstraint=ap-southeast-1
IAM role for S3 replication
Let's quickly create an IAM role for our later usage of S3 replication rule.
aws iam create-role \
--role-name <IAM_ROLE_REPLICATION_A> \
--assume-role-policy-document file://trust.json
aws iam put-role-policy \
--role-name <IAM_ROLE_REPLICATION_A> \
--policy-name replication-policy \
--policy-document file://policy.json
Here is the IAM policy (policy.json) to be used.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:InitiateReplication",
"s3:GetReplicationConfiguration",
"s3:PutInventoryConfiguration",
"s3:ListBucket",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectTagging",
"s3:GetObjectVersionForReplication",
"s3:GetObjectVersionAcl",
"s3:GetObjectVersionTagging"
],
"Resource": [
"arn:aws:s3:::<S3_BUCKET_A>",
"arn:aws:s3:::<S3_BUCKET_A>/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ReplicateObject",
"s3:ReplicateTags",
"s3:ReplicateDelete",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectVersionAcl",
"s3:PutObjectTagging",
"s3:PutObjectVersionTagging",
"s3:GetObjectVersionForReplication",
"s3:ObjectOwnerOverrideToBucketOwner"
],
"Resource": [
"arn:aws:s3:::<S3_BUCKET_B>",
"arn:aws:s3:::<S3_BUCKET_B>/*"
]
},
{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<S3_BUCKET_REPORT_A>/*"
},
{
"Effect": "Allow",
"Action": "kms:*",
"Resource": "*"
}
]
}
Here is the trust relationship (trust.json) to be used.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"batchoperations.s3.amazonaws.com",
"s3.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
Destination Account B
Update S3 bucket policy
Ensure the S3 bucket policy to be updated.
{
"Version": "2012-10-17",
"Id": "PolicyForDestinationBucket",
"Statement": [
{
"Sid": "S3PolicyStmt-DO-NOT-MODIFY-1234567890123",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<AWS_ID_A>:root",
"arn:aws:iam::<AWS_ID_A>:role/service-role/<IAM_ROLE_REPLICATION_A>"
]
},
"Action": [
"s3:GetBucketVersioning",
"s3:GetObjectAcl",
"s3:ReplicateObject",
"s3:ReplicateDelete",
"s3:PutObjectAcl",
"s3:PutObjectVersionAcl",
"s3:PutBucketVersioning",
"s3:ObjectOwnerOverrideToBucketOwner",
"s3:Put*"
],
"Resource": [
"arn:aws:s3:::<S3_BUCKET_B>",
"arn:aws:s3:::<S3_BUCKET_B>/*"
]
}
]
}
Update KMS key policy
Ensure the KMS key policy to be updated.
{
"Version": "2012-10-17",
"Id": "my-key-policy",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AWS_ID_B>:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow an external account to use this KMS key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<AWS_ID_A>:root",
"arn:aws:iam::<AWS_ID_A>:role/service-role/<IAM_ROLE_REPLICATION_A>"
]
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
}
]
}
Create a replication rule
From the source S3 bucket, create an S3 replication rule.
Choose the rule scope to all objects and specify destination bucket with AWS account ID.
Choose the option to change the bucket ownership and specify the IAM role that we have created earlier.
Enter the ARN of the destination AWS KMS encryption key to be used.
For destination classes and replication options, keep them as optional if you don't have any related requirements. Then, choose option to replicate existing objects. This will direct you to a new page which is the next step below.
Create Batch Operations job
For Job run options, we choose Automatically run the job when it's ready or it is up to you.
In order to gain visibility of replication process, we enable reports generation option and store them onto S3 bucket.
For Permissions, you can reuse the earlier IAM Role.
Once you clicked the Save button, it will create a new Job ID and based on my job run option earlier, it automatically runs so we will wait for the result.
If it is
failed, you can check the job status or report for further details. In my case, my replication IAM role does not have enough permissions to generate reports to the specified bucket.
If it is
progresses, you will observe the % of objects that have been replicated or objects have been failed to replicate.
Top comments (1)
Pretty sure you could be more specific then "s3:Put*"