The problem

Some time ago, me and my team were asked to migrate several Elasticache databases (Redis) from EC2-Classic (yep, I know 🤷‍♂️) to VPC. All these Databases were being used in production services so, taking an snapshot and restoring it on a VPC was not a good idea due to the time that takes doing the whole process.
Using AWS Online Migration was not an option too, since Online migration is designed for data migration from hosted Redis on Amazon EC2 or on-premise self-hosted Redis to ElastiCache for Redis and not between ElastiCache for Redis clusters.
So we kept looking for alternatives until we found a nice gist with a script that almost fit our needs. The only fail there was that it wasn't work ok on encryption in transit clusters.

The solution

So we did some changes on the script to fit our use-case and et voila! We finaly got a working version to move the data.

Demo

From a terminal with access to both Elasticache clusters (very important!):

1. Clone the repo:

git clone git@github.com:joseapeinado/ElasticacheMigration.git

1. Prepare the virtual env and install dependencies

cd ElasticacheMigration
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

1. Profit!!!

SRC_CLUSTER=src_cluster.spznk9.ng.0001.use1.cache.amazonaws.com
DST_CLUSTER=dst_cluster.spznk9.ng.0001.use1.cache.amazonaws.com

python3 migrate-redis.py $SRC_CLUSTER --src_port=6379 $DST_CLUSTER --dst_port 6379 --src_db 2 --dst_db 2
Connecting to Redis instances...
Counting keys to migrate...
19222 keys: 100% |###############################################################################################################################################################################################################################################| Time: 0:00:05
('Keys disappeared on source during scan:', 0)
('Keys already existing on destination:', 19195)

Use cases

Copying just one Database to other existing cluster

The script copies data between clusters in a per-database basis, so it is ready to perform this migration out-of-the-box.

Copy all databases from one cluster to other

The only caveat here is the need of run the migration script once per database.

Copy database data from one DB to other DB within the same cluster.

Advantages doing this migration

The more obvious advantage I think is the time that could take using the script versus:

1. Stop writes on Redis cluster
2. Take a snapshot
3. Launch a new cluster from the snapshot
4. Resume writes on the new cluster

This process can take at least 15 minutes or so.

Contribute

Any improvements will be more than welcomed 😃
https://github.com/joseapeinado/ElasticacheMigration/

This is my very first tech post, so any advice would be appreciated.
And I hope the migration script help you like it helped me 🙂