How do I copy S3 objects/files/etc from another AWS account?
How do I migrate an S3 bucket to another AWS account? [playbook below + extra preliminary step, create the bucket]
Well, you can do it manually following the notes in the AWS KB article I linked at the end of the article.
Honestly though, I am always confused by those KB articles. They are concise ans usually well written. I guess it's my dislexya ;)
So I have to read them a few times before I’m able to apply the steps correctly. That's not a bad thing really ... better than blindly copy/paste commands (which I NEVER DO, right).
Recently, I’ve been going the other way around: whenever there are steps to follow, I try to translate them into an Ansible playbook. This way I make sure I got it right, AND I get to reuse the playbook, which makes things a lot faster the second time around. Oh and as an added bonus, I can share the playbook and have someone else take care of the task!
Enough introductions, here’s what you came here for: an ansible playbook for syncing files between S3 buckets in different accounts:
# copy-bucket.yml
- hosts: localhost
vars:
source_bucket: my-source-bucket
dest_bucket: my-dest-bucket
dest_user_arn: arn:aws:iam::ACCOUNTID:user/USERNAME
dest_user_name: USERNAME
source_profile: src_profile
dest_profile: dest_profile
tasks:
- name: Attach bucket policy to source bucket
s3_bucket:
name: "{{ source_bucket }}"
policy: "{{ lookup('template','bucket-policy.json.j2') }}"
profile: "{{ source_profile }}"
- name: Attach an IAM policy to a user in dest account
iam_policy:
iam_type: user
iam_name: "{{ dest_user_name }}"
policy_name: "s3_move_access"
state: present
policy_json: "{{ lookup( 'template', 'user-policy.json.j2') }}"
profile: "{{ dest_profile }}"
register: user_policy
- name: Sync the files
shell: aws s3 sync s3://{{ source_bucket }}/ s3://{{ dest_bucket }}/ --profile {{ dest_profile }}
You will also need the following json templates
bucket-policy.json.j2
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::{{ source_bucket }}",
"arn:aws:s3:::{{ source_bucket }}/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::{{ dest_bucket }}",
"arn:aws:s3:::{{ dest_bucket }}/*"
]
}
]
}
user-policy.json.j2
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {"AWS": "{{ dest_user_arn }}"},
"Action": ["s3:ListBucket","s3:GetObject"],
"Resource": [
"arn:aws:s3:::{{ source_bucket }}/*",
"arn:aws:s3:::{{ source_bucket }}"
]
}
]
}
run: ansible-playbook -v copy-bucket.yml
make sure the profile names are setup on your machine correctly, and the IAM user is there.
Top comments (0)