Zahraa Jawad

Posted on Feb 25, 2023

AWS Data Migration Service (DMS) - MySQL to S3 Bucket services

#aws #database #community

The article contains two parts:

Part- 1
Some configuration for the migration:

Create an IAM role to allow access to the target
Create S3 bucket to be the target endpoint.

Part-2

Margate the MySQL Database to S3 bucket.

outline

Margate the MySQL Database to S3 bucket
Data migration service architecture
Creation of Data Migration Service (DMS)

Margate the MySQL Database to S3 bucket

AWS Database Migration Service (AWS DMS) is a managed migration and replication service that helps move your database and analytics workloads to AWS quickly, securely, and with minimal downtime and zero data loss. AWS DMS supports migration between 20-plus database and analytics engines, such as Oracle to Amazon Aurora MySQL-Compatible Edition, MySQL to Amazon Relational Database (RDS) for MySQL, Microsoft SQL Server to Amazon Aurora PostgreSQL-Compatible
Edition, MongoDB to Amazon DocumentDB (with MongoDB compatibility), Oracle to Amazon Redshift, and Amazon Simple Storage Service (S3).

Migration use cases:

• Move to managed databases:
To migrate data from legacy or on-premises databases to managed cloud services through a streamlined migration process.

• Remove licensing costs and accelerate business growth:
To update purpose built databases to innovate and build faster for any use case at scale at one tenth of the cost.

• Replicate ongoing changes:
Create redundancies of business-critical databases and data stores to minimize downtime and protect against any data loss.

• Improve integration with data lakes:
Build data lakes and perform real-time processing on change data from your data stores.

Data migration service architecture

The Data Migration Service architecture consists of some configurations:

• Replication instance
• Database Endpoint
• Database Migration Tasks

• Replication instance

AWS DMS uses a replication instance to connect to your source data store, read the source data, and format the data for consumption by the target data store. A replication instance also loads the data into the target data store. Most of this processing happens in memory.

The creation of DNS architecture replication instance is similar to how we choose an ec2 instance typically we give a very small number of parameters and then our replication instance would be up and running we choose the size of the instance like how we choose an easy instance size and very few parameters whether you want a single ac or multi-ac and then your replication instance would be up and running.

• Database Endpoint

This is the stage we specify the connection details of the source and target database.

- A data source: is an initial location where a database is created or where physical information is first digitized, however even the most refined data may serve as a source, as long as another process accesses and utilizes it.

- Target database: is the database to which you are moving the data or new changes.

• Database Migration Tasks:

An AWS Database Migration Service (AWS DMS) task is where all the work happens. You specify what tables (or views) and schemas to use for your migration and any special processing, such as logging requirements, control table data, and error handling.

A task can consist of three major phases:

o Migration of existing data (Full load)
o The application of cached changes
o Ongoing replication (Change Data Capture)

Before creating the data migration we need to :
1- Create an IAM role to allow access to the target:

To open the IAM console :

From the services of the your AWS account choose the IAM service : • Choose Roles, then Create role.

• Under Trusted entity type, select AWS service.
• From the Use cases for other AWS services dropdown list, choose DMS.
• then choose Next.

• On the Add permissions tab, choose the permission Amazons3FullAccess.
• Choose Next.

• For Role name, enter a unique name for your role, such as s3 migration-role.
• For Description, enter descriptive text .

The permission is add

• Then choose Create role.

• The Role is successfully created

And we can add another permissions to the role by :
• choose the role that we are created:
• Select Add permissions then Attach polices.

• Search for the polices we need (AmazonRDSFullAccess) and choose it by click the check box.
• After add the polices click on Add polices.

• Notice that all policies have been successfully attached to our role.

• Copy the ARN for the role that will be used in configuring the migration task.

2- Create the S3 bucket to migrate the data to it:

Open the S3 console:
• Choose the Create bucket.

• Do some configuration and then choose the Create bucket.

The bucket is created and after opening it notice that it is empty

Creation the AWS Database Migration Service

To begin creation of AWS database migration service:

Open the AWS Database Migration Service console:
• From (Services ) select the Database Migration Service.

 In the DMS the first step is create an instance replication :

• Click on Create replication instance

In the instance configuration:

• In storage:

• In connectivity and security:

• The replication instance is creating:

• And after some minutes it will be available

 The second step of AWS Database Migration Service is to create the endpoint:

In this configuration, you specify the source endpoint for the data and the target endpoint to which the data will be migrated.

• Click on the Endpoint, then click on Create endpoint.

For the endpoint type:
• First, choose Source endpoint.
• If the source endpoint is RDS database instance, select the checkbox.
• Select the RDS instance from which you want to migrate the data.

• additional endpoint setting, using wizard or editor.

• Add tag to the end point (optional)

To test the running of the end point, click on Run test.

The source endpoint is created .

Now to create the target endpoint, click again on Create endpoint.
• In the endpoint type, select Target endpoint.

In the target endpoint configuration:

To test the running of the end point, click on Run test.

The target endpoint is created .

 The third step of AWS Database Migration Service is to create the database migration task:

• Click in the left side on Database migration tasks.
• Click on Create task.

• In the creation of database migration task:

You can configure some setting for the migration task:

In migration task startup configuration:

• You can start the migration automatically on create or manually later
• Add tag (optional)
• Click on Create task .

• The migration task has been created, and is in the creating state.

• Then after refresh it is ready.
• Then after a few minutes, the migrate task is in the running state.