Copy Redis Data to New Server

Sometimes we need to copy redis data to new instance

I know 3 simple ways to do it:

• continuous migration tools
• simple script to copy data
• redis master-slave replication

Other options, more complex:

• Redis HA & sentinel solution - it will require rebuild redis
• https://github.com/Netflix/dynomite - replication and routing

redis-migrate-tool

https://github.com/vipshop/redis-migrate-tool

It monitors changes in source server/cluster and apply them in destination server. Useful when we migrate app that actively writes to redis to use new redis instance. There are other similar tools, but this one works and easy to configure.

Check README for supported types, It's written in C, easier to run it with docker

docker pull ahmadgh21/redis-migrate-tool
docker run -it ahmadgh21/redis-migrate-tool bash
# inside container
mkdir /tmp
chmod 777 /tmp
apt-get update
apt install nano
nano /etc/redis-migrate-tool/rmt.conf

Config example

[source]
type: single
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers :
- <source IP>:6379 # <--- change this

[target]
type: single
servers:
- <destination IP>:6379 # <--- change this

[common]
listen: 0.0.0.0:8888

run migrate-tool inside container

/opt/redis-migrate-tool -c /etc/redis-migrate-tool/rmt.conf

Make own script

This script simply copy all keys from server A to server B (it doesn't support all data types, find it in code)

require 'redis'

puts "Connecting to redis://#{ARGV[0]} and redis://#{ARGV[1]}"

redis_src = Redis.new(url: "redis://#{ARGV[0]}")
redis_dest = Redis.new(url: "redis://#{ARGV[1]}")

dest_keys = redis_dest.keys("*")

redis_src.keys("*").each do |key|
  next if dest_keys.include?(key)
  type = redis_src.type(key)

  if type == "hash"
    value = redis_src.hgetall(key)
    puts "HASH   #{key} : #{value}"
    redis_dest.mapped_hmset(key, value)
  elsif type == "list"
    values = redis_src.lrange(key, 0, -1)
    puts "LIST.  #{key} : #{values}"
    redis_dest.lpush(key, values)
  elsif type == "set"
    values = redis_src.smembers(key)
    puts "SET    #{key} : #{values}"
    redis_dest.sadd(key, values)
  else
    data = redis_src.get(key)
    puts "STRING #{key} : #{data}"
    redis_dest.set(key, data)
  end
end

puts "Complete"

Make sure you have ruby redis library installed (gem install redis)

Run the script

ruby redis_copy.rb source_IP destination_IP

Setup redis replication

I think this is most proper option but also require more preparation

on master we need to update config

protected-mode no
requirepass 1234abcd

on replica server we need to put in config:

replicaof <master ip> 6379
masterauth 1234abcd

restart master, then replica, check logs

check status in redis-cli

INFO replication

more docs:

• https://redis.io/docs/manual/replication/
• https://www.vultr.com/docs/how-to-set-up-a-redis-replication-cluster
• https://medium.com/dlt-labs-publication/how-to-setup-redis-replication-c9cc89ba6c03

Pros and Cons

Simple script - It copy values once, can be used for not critical data such as cache, rate limit, and for rarely updated data such as configuration

I usually run it once before updating apps to connect to new server, and once again after to make sure all data is in sync

redis-migrate-tool - Constantly replicates all data from source to destination. For apps that constantly write updates to redis, e.g. user sessions, temporary important values.

But it doesn't replicate writes from destination server back to source. To avoid situation when one process write/update to new server and other process can't find it in old server, we need to restart all our apps at one time with updated config. (I usually scale app down to 1 replica during migration)

redis replication - replica is not writable, migration may require short downtime or we need to modify code to handle it smoothly. In our code we can try to save data to new server, and if got "not writable" error then save to old server

Have a smooth redis migration to everyone who's doing it now :)