As part of a merger with another team, my team migrated 8 DynamoDB tables from one AWS account to another.

This was a challenging process, in this blog I'll be covering everything the AWS documentation doesn't tell you to look out for when migrating DynamoDB table across AWS Accounts.

Design

Initially, we were investigating the use of AWS Glue following this blog. AWS Glue was a service we were unfamiliar with and it would have resulted in us needing to write a different Python Glue script for each table. We ruled this our as a potential option based on this.

During our investigation into a suitable approach for this work, AWS released the new import to S3 functionality, we were able to leverage this alongside a table export to S3 to complete the data move.

While AWS gives us the steps needed to complete this move using the console, we are typically unable to do this due to restricted access to anything above our development environment. Plus the steps outlined in the blog show creating a table through the console, hardly a suitable process for any real-life project.

We came up with the following process to complete this work that would work with the level of access we could get:

1. Elevate into the IAM role with permissions to trigger an S3 export and write to the target account data transfer bucket. Permissions required can be found here.
2. Trigger export, specifying target account ID and bucket name.
3. Once complete this will generate a new folder in the data transfer bucket. This will look like AWSDynamoDB/<id>/data with being a generated ID.
4. This path needs to be specified in the import parameters used to trigger the import and create the new table.
5. After adding this ID to your code, begin deployment of the new database, this will trigger an import as part of the deployment. Eventually, you will have a created table with the data present.

Full credit to Elias Brange for this fantastic blog post on how to trigger imports and table creation using the AWS CDK.

The CloudFormation documentation for ImportSourceSpecification has now been released and is considered stable.

Challenges

Zero Feedback on Progress

After triggering an import or export, the AWS Console will just say "importing". One of the tables that we imported had 55 million rows and took nearly 1 hour and 30 minutes to import. All while we had a production outage to facilitate the data migration.

Some indication of progress or estimated time remaining would instil more confidence in this process.

In addition to this, the import time varied widely and there was no way to speed it up. As we took a considerable production outage to migrate the data we ideally would have wanted a faster process for larger tables.

DynamoDB Global Tables

As part of the move, we switched to global tables as we look to enable multi-region for our applications.

One issue we had was a deployment of a table failed to complete as the waitForReplicationToFinish property is defaulted to true in the AWS CDK. This deployment failed because the custom resource time out is limited to a maximum of 60 minutes.

This resulted in this error:

CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [*****].

This resulted in a stack being stuck in a DELETE_FAILED state. As the source table could not be deleted as it had acted as a replica provider in the last 24 hours.

Resource handler returned message: "Replica cannot be deleted because it has acted as a source region for new replica(s) being added to the table in the last 24 hours.

As the actual table wasn't deleted, we were still able to read and write data from the table in a DELETE_FAILED state.

To repair this stack we had to export the new table to S3, delete the CloudFormation stack and re-import the data from S3 with the waitForReplicationToFinish property set to get the stack in a healthy state.

This may have been a user error but should serve as good learning for anyone switching from a standard DynamoDB table to a global table.

Manual Process

This was a very manual process to complete, compared with using AWS DMS to migrate data for other AWS Database offerings. It required a lot of pre-work, and investigation and wasn't a quick process to complete in production.

I identified 9 steps that we needed to take for each migration:

1. Deploy the application branch to shut off writes to the source database
2. Verify this has been completed before proceeding
3. Trigger export of source database
4. Add import ID to new database branch
5. Verify export has been completed before proceeding
6. Deploy the new database branch with import ID
7. Verify import completes
8. Revert application branch to resume writes to the database
9. Verify new writes happening to DB

I would ideally have liked a managed solution from AWS to handle this migration without downtime or as much of a manual process.

Redundant CloudFormation Properties

To trigger the imports, we used CDK escape hatches to specify ImportSourceSpecification when creating our DynamoDB table.

cfnTable.addPropertyOverride('ImportSourceSpecification.S3BucketSource.S3Bucket', bucketName);
cfnTable.addPropertyOverride('ImportSourceSpecification.S3BucketSource.S3KeyPrefix', bucketPath);
cfnTable.addPropertyOverride('ImportSourceSpecification.InputCompressionType', 'GZIP');
cfnTable.addPropertyOverride('ImportSourceSpecification.InputFormat', 'DYNAMODB_JSON');

As per the CloudFormation documentation for DynamoDB tables the update policy of this is Replacement.

Ideally, we would have liked to remove this property from our CDK stacks after that initial table creation and successful completion of the import. We have a large number of tables and want to keep our code neat and tidy.

It's definitely introduced a bit of technical debt to these projects as if someone inadvertently removes these properties it will break the deployment of the table. We have a warning comment above these lines in every project, but again this is a far from optimal solution.

Conclusion

If you are looking to migrate data, the export to S3 functionality is a good solution. Just be extra cautious of the things I've outlined in this post.

I'd also recommend outlining a plan to "abort" a migration once you have started it if anything goes wrong. This process has a lot of moving parts and a lot of potential failure points.

I'd also love if AWS solved some of the issues outlined in this post, it wasn't the best experience compared to if I had of been migrating a DB supported by DMS. Always felt weird that DynamoDB wasn't supported by this service.

Have you migrated DynamoDB data in a different way than I have, if so I'd love to hear about your approach in the comments!