DEV Community

JackTT
JackTT

Posted on

Migrate Redis to AWS ElastiCache

How to Migrate Redis from One Server to Another

Starting from Redis version 5.0.0, Redis supports the REPLICAOF command, which assists in migrating data by making the destination Redis server a replica of the source Redis server. You can perform this migration with just a few simple steps:

Step 1: Make the destination Redis server a replica of the source Redis server by running the following command:

REPLICAOF host port
Enter fullscreen mode Exit fullscreen mode

Step 2: Transform the destination Redis server into a master server by executing the following command:

REPLICAOF NO ONE
Enter fullscreen mode Exit fullscreen mode

That's it!

AWS ElastiCache does not support REPLICAOF

When migrating Redis, we encountered a struggle as AWS Elasticache restricts access to certain commands, including "MIGRATE" and "REPLICAOF".

Solution

I discovered a repository written in Go that helped us address this issue by running a one-time application.
It appeared promising, but it hadn't been updated for several years. I identified some problems:

  • It used LPOP, which resulted in the removal of list elements from the "From Redis Instance."
  • It ran in a single thread, causing significant delays in migrating large records.
  • It did not support certain key types, including Set and SortedSet.

I forked the repository and made the necessary fixes & improvements.

Usage

Step 1: Create config file
config.yaml

old_redis:
  host: localhost # IP redis server
  port: 6379 # Port redis server
  password: "" # Password of redis server, leave empty if there is no password

new_redis:
  host: localhost
  port: 6380
  password: ""

migration_databases: [0,1,2,3,4,5,6,7,8] # Databases list which needs to be migrated
concurrent_workers: 5
clear_before_migration: true
Enter fullscreen mode Exit fullscreen mode

Step 2: Run using Docker

docker run \
--rm \
-v /path/to/config.yaml:/data/config.yaml \
 huanttok/redis-migrator migrate \
 --config.file=/data/config.yaml \
--log.level=debug
Enter fullscreen mode Exit fullscreen mode

(To call localhost while using Docker, use host.docker.internal instead.

Source code

GitHub logo huantt / redis-migrator

This tool is designed to migrate Redis database keys from one server to another, addressing the pain point of missing 'REPLICAOF' on AWS ElastiCache

About

GitHub Super-Linter made-with-Go GitHub go.mod Go version Go Report Card Apache License GitHub release (latest by date)

The Redis Migrator is a Golang-based tool designed to migrate database keys from one Redis cluster to another.

Since Redis 5.0.0, you can use REPLICAOF command to replicate data from one redis to another.

This application is developed for the older version of Redis or AWS ElastiCache instances that do not support REPLICAOF command (AWS docs).

(This repo is forked from opstree/redis-migration. The original repo has been inactive for over 3 years, so I decided to continue its development without creating pull requests.)

Supported key types:

  • String
  • Hash
  • List
  • Set
  • Sorted Set

Usage

Installation

Using Go

go install github.com/huantt/redis-migrator@latest
Enter fullscreen mode Exit fullscreen mode

Using Docker

docker run \
--rm \
-v /path/to/migrate.yaml:/data/migrate.yaml \
 huanttok/redis-migrator migrate \
 --config.file=/data/migrate.yaml \
--log.level=debug
Enter fullscreen mode Exit fullscreen mode

To call localhost while using Docker, use host.docker.internal instead.

Configuration

For using redis-migrator, we have to create a configuration file and provide some needful information to it. An example…

Top comments (0)