Migrate Data into S3 using AWS Transfer Family

What is AWS Transfer Family?

AWS Transfer Family is a fully managed AWS service that you can use to transfer files into and out of Amazon Simple Storage Service (Amazon S3) storage or Amazon Elastic File System (Amazon EFS) file systems over the following protocols:

• Secure Shell (SSH)
• File Transfer Protocol (SFTP)
• File Transfer Protocol (SFTP)
• File Transfer Protocol Secure (FTPS)
• File Transfer Protocol (FTP)

Read more: AWS Transfer Family

Benefits Of AWS Transfer Family?

• It is a fully managed, serverless File Transfer Workflow service that makes it simple to set up, execute, automate, and monitor file processing utilizing AWS Transfer Family.
• With your data safely stored in Amazon S3, you can utilize native AWS services for processing, analytics, reporting, auditing, and archiving.
• There are no upfront costs, and you pay only for the use of the service.
• You don't need to modify your applications or run any file transfer protocol infrastructure.

How it works?

AWS Transfer Family Architecture

AWS Transfer Family (SFTP, FTP, FTPS) – Step by Step Implementation Process

Setup the prerequisites for AWS transfer for SFTP

 - S3 bucket
 - EC2 instances (Linux and Windows)

Create an AWS IAM role and policy

- Edit the policy to provide the S3 access

Create the SFTP server

 - Create users
 - Create and assign public keys

Test the file transfer from SFTP clients

- For Linux
- For Windows

What is S3 Bucket?

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Read more: Amazon Simple Storage Service

Create a S3 Bucket

- Open AWS Console Click on Services
- In the Storage Section, Choose S3

- Click Create Bucket and Create Your Bucket
- Give Bucket Name (Should be unique Name use 
small alphabets Without Space. hyphens and Periods}
- Choose your Region. (Where user wants to store their data)
 rest settings choose default.

Create a VPC

- Open AWS Console Click on services
- In the Networking & Content Delivery Section, Click on VPC

Create Subnets

Create Two EC2 Instances (Linux & Windows)

- Open AWS Console Click on services 
- In the Compute section, click on EC2
- Click Launch instance

Security Group rules for Linux Instance

Security Group rules for Windows Instance

Create an IAM Role

- Open AWS Console Click on services 
- In the Security, Identity & Compliance section, click on IAM 

Create IAM roles and policies from the IAM console (Choose AWS service as Transfer)

The policy to access the S3 bucket

Policy to access CloudWatch Logs

Create two roles and attached the policies with each role (Choose AWS service as Transfer)

Create the SFTP server, Users, Public Keys and Assign the keys

- Open AWS Console Click on services 
- In the Migration & Transfer section, click on AWS Transfer Family

Click on create Server

Choose Protocols

Choose an Identity Provider

Choose an Endpoint

Choose a Domain

Add User to the Server

Generate the public key from the Linux machine?

Log in to your EC2 machine connect via Putty

Generate the key with the below command

ssh-keygen -P "" -f key 

Transfer the files using WinSCP for Linux

Test the file transfer from the Linux SFTP client

Connect to the server

- sftp -i key rm-transfer-family-user@
s-c0ca408efe3d4c3e8.server.transfer.ap-southeast-1.amazonaws.com

- put AWS-Transfer-Family-Linux.txt rm-transfer-family-2022 
AWS-Transfer-Family-Linux.txt 
to/rm-transfer-family-2022/rm-transfer-family-user/AWS-Transfer-Family-Linux.txt 

Successfully transfer File from Server to Bucket

Now do the same using the Windows Machine

Generate Public key for Windows user

Two Public keys are added

Now transfer the files using WinSCP for Windows

Connection established using WinSCP for Windows

Test the file transfer from SFTP client for Windows

Successfully Tested the file transfer from SFTP client and added to the Bucket