Redis® Sentinel provides high availability for Redis. Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.
This article shows how to set up a high-availability architecture for Redis using Sentinel in Debian GNU/Linux:
Prerequisites
Debian preinstalled servers, e.g.:
Host name | IP | Role |
---|---|---|
redis-1 | 10.0.0.21 | Master node |
redis-2 | 10.0.0.22 | Slave node |
redis-3 | 10.0.0.23 | Slave node |
sentinel-1 | 10.0.0.24 | Sentinel node |
sentinel-2 | 10.0.0.25 | Sentinel node |
sentinel-3 | 10.0.0.26 | Sentinel node |
Setup Redis replication
Install Redis
- Connect to hosts with roles:
Master node
,Slave node
-
Run the following commands:
$ sudo apt-get update $ sudo apt-get install -y redis-server
-
Activate the redis-server service to start when Redis server boots:
$ sudo systemctl enable redis-server
-
Check redis-server status:
$ systemctl status redis-server ● redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor > Active: active (running) since Mon 2024-09-16 21:06:13; 19min ago Docs: http://redis.io/documentation, man:redis-server(1) Main PID: 3057 (redis-server) Status: "Ready to accept connections" Tasks: 5 (limit: 2307) Memory: 9.2M CPU: 2.146s CGroup: /system.slice/redis-server.service └─3057 /usr/bin/redis-server 127.0.0.1:6379
Service runs and ready to accept connections.
Set up the Master node
- Open the configuration file at
/etc/redis/redis.conf
-
Set following mandatory parameters:
protected-mode yes bind <masterip: 10.0.0.21> 127.0.0.1 requirepass <master-password> masterauth <master-password>
-
Restart the redis-server service:
$ sudo systemctl restart redis-server
-
Make sure that Redis accepts connections with authentication:
$ redis-cli --askpass info
Set up the Slave nodes
- Open the configuration file at
/etc/redis/redis.conf
-
Set following mandatory parameters:
bind <slaveip: 10.0.0.21/22> 127.0.0.1 requirepass <master-password> masterauth <master-password> replicaof <masterip: 10.0.0.21> 6379
-
Restart the redis-server service:
$ sudo systemctl restart redis-server
-
Make sure that Redis accepts connections with authentication:
$ redis-cli --askpass info
Check Redis replication setup
-
Check the redis-server log on Master node:
$ tail /var/log/redis/redis-server.log -n 20 3498:M 16 Sep 2024 22:25:53.086 * RDB memory usage when created 0.77 Mb 3498:M 16 Sep 2024 22:25:53.086 * DB loaded from disk: 0.000 seconds 3498:M 16 Sep 2024 22:25:53.086 * Ready to accept connections 3498:M 16 Sep 2024 23:29:52.119 * Replica 10.0.0.22:6379 asks for synchronization 3498:M 16 Sep 2024 23:29:52.119 * Full resync requested by replica 10.0.0.22:6379 3498:M 16 Sep 2024 23:29:52.119 * Replication backlog created, my new replication IDs are '368571d1e52c29535615e0043a3a6d6ddc1db70b' and '0000000000000000000000000000000000000000' 3498:M 16 Sep 2024 23:29:52.119 * Starting BGSAVE for SYNC with target: disk 3498:M 16 Sep 2024 23:29:52.119 * Background saving started by pid 3684 3684:C 16 Sep 2024 23:29:52.132 * DB saved on disk 3684:C 16 Sep 2024 23:29:52.133 * RDB: 0 MB of memory used by copy-on-write 3498:M 16 Sep 2024 23:29:52.185 * Background saving terminated with success 3498:M 16 Sep 2024 23:29:52.185 * Synchronization with replica 10.0.0.22:6379 succeeded 3498:M 16 Sep 2024 23:30:09.952 * Replica 10.0.0.23:6379 asks for synchronization 3498:M 16 Sep 2024 23:30:09.952 * Full resync requested by replica 10.0.0.23:6379 3498:M 16 Sep 2024 23:30:09.952 * Starting BGSAVE for SYNC with target: disk 3498:M 16 Sep 2024 23:30:09.952 * Background saving started by pid 3688 3688:C 16 Sep 2024 23:30:09.965 * DB saved on disk 3688:C 16 Sep 2024 23:30:09.965 * RDB: 0 MB of memory used by copy-on-write 3498:M 16 Sep 2024 23:30:09.985 * Background saving terminated with success 3498:M 16 Sep 2024 23:30:09.985 * Synchronization with replica 10.0.0.23:6379 succeeded
-
Check replication status on Master node:
$ redis-cli --askpass info replication Please input password: **************** # Replication role:master connected_slaves:2 slave0:ip=10.0.0.22,port=6379,state=online,offset=1652,lag=1 slave1:ip=10.0.0.23,port=6379,state=online,offset=1652,lag=1 master_replid:368571d1e52c29535615e0043a3a6d6ddc1db70b master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1652 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:1652
-
Run the following command on Master node:
$ $ redis-cli --askpass set abc 123 Please input password: **************** OK
-
Run the following command on Slave nodes:
$ redis-cli --askpass get abc Please input password: **************** "123"
Congrats! Master-to-Slave data replication succeed.
Setup Redis Sentinel
Install Redis Sentinel
- Connect to hosts with the role
Sentinel
-
Run the following commands:
$ sudo apt-get update $ sudo apt-get install -y redis-sentinel
-
Activate the redis-sentinel service to start when Sentinel server boots:
$ sudo systemctl enable redis-sentinel
-
Check redis-sentinel status:
$ systemctl status redis-sentinel ● redis-sentinel.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendo> Active: active (running) since Tue 2024-09-17 00:56:22 MSK; 1min 29s ago Docs: http://redis.io/documentation, man:redis-sentinel(1) Main PID: 3259 (redis-sentinel) Status: "Ready to accept connections" Tasks: 5 (limit: 2307) Memory: 7.2M CPU: 190ms CGroup: /system.slice/redis-sentinel.service └─3259 /usr/bin/redis-sentinel 127.0.0.1:26379 [sentinel]
Set up Sentinel nodes
- Open the configuration file at
/etc/redis/sentinel.conf
-
Set following mandatory parameters:
protected-mode yes bind <sentinelip: 10.0.0.24/25/26> 127.0.0.1 sentinel monitor mymaster <masterip: 10.0.0.21> 6379 2 sentinel down-after-milliseconds mymaster 30000 sentinel failover-timeout mymaster 180000 sentinel parallel-syncs mymaster 1 sentinel auth-pass mymaster <master-password> requirepass <sentinel-password>
-
Restart the redis-sentinel service:
$ sudo systemctl restart redis-sentinel
-
Make sure that Redis Sentinel accepts connections with authentication:
$ redis-cli -p 26379 --askpass info sentinel Please input password: **************** # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.21:6379,slaves=2,sentinels=3
Testing the failover
This section shows how to test the failover of the high availability Redis using Sentinel.
Master node failure
-
Run the following command on Master node:
$ redis-cli --askpass client pause 40000 Please input password: **************** OK
Note:
client pause 40000
due tosentinel down-after-milliseconds mymaster 30000
-
Run the following command on one of the Sentinel nodes:
$ tail -10 /var/log/redis/redis-sentinel.log 6175:X 18 Sep 2024 21:39:49.309 # +sdown master mymaster 10.0.0.21 6379 6175:X 18 Sep 2024 21:39:49.555 # +new-epoch 1 6175:X 18 Sep 2024 21:39:49.558 # +vote-for-leader 77fc4fbc9c6c7d063d62e38f528910ab67e8d907 1 6175:X 18 Sep 2024 21:39:50.450 # +odown master mymaster 10.0.0.21 6379 #quorum 3/2 6175:X 18 Sep 2024 21:39:50.451 # Next failover delay: I will not start a failover before Wed Sep 18 21:45:50 2024 6175:X 18 Sep 2024 21:39:51.853 # +config-update-from sentinel 77fc4fbc9c6c7d063d62e38f528910ab67e8d907 10.0.0.26 26379 @ mymaster 10.0.0.21 6379 6175:X 18 Sep 2024 21:39:51.853 # +switch-master mymaster 10.0.0.21 6379 10.0.0.22 6379 6175:X 18 Sep 2024 21:39:51.853 * +slave slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379 6175:X 18 Sep 2024 21:39:51.853 * +slave slave 10.0.0.21:6379 10.0.0.21 6379 @ mymaster 10.0.0.22 6379 6175:X 18 Sep 2024 21:40:09.323 * +convert-to-slave slave 10.0.0.21:6379 10.0.0.21 6379 @ mymaster 10.0.0.22 6379
Note:
Next failover delay: I will not start a failover before Wed Sep 18 21:45:50 2024
due tosentinel failover-timeout mymaster 180000
-
Run the following command on one of the Sentinel nodes:
$ redis-cli -p 26379 --askpass sentinel get-master-addr-by-name mymaster Please input password: **************** 1) "10.0.0.22" 2) "6379"
Redis node with IP
10.0.0.22
is the new Master node. -
Run the following command on the new Master node:
$ redis-cli --askpass info replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.23,port=6379,state=online,offset=519426,lag=1 slave1:ip=10.0.0.21,port=6379,state=online,offset=519559,lag=0 master_replid:ade635098e62dc68f3ee65acf37970f98e1a2f8a master_replid2:8777248b814d71f4c609f437c37a529e01b7d0c5 master_repl_offset:519692 second_repl_offset:195253 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:519692
Sentinel node failure
-
Run the following command on the first Sentinel node:
$ sudo systemctl stop redis-sentinel
-
Check
redis-sentinel.log
on the second Sentinel node:
tail -1 /var/log/redis/redis-sentinel.log 5587:X 18 Sep 2024 22:14:36.112 # +sdown sentinel d01aa131dc0ba6cc4069f3ec67bf6b4fa6b501f8 10.0.0.24 26379 @ mymaster 10.0.0.22 6379
-
Run the following command on Master node:
$ redis-cli --askpass client pause 40000 Please input password: **************** OK
-
Check
redis-sentinel.log
on the second Sentinel node:
$ tail -10 /var/log/redis/redis-sentinel.log 5587:X 18 Sep 2024 22:20:49.232 # +failover-state-reconf-slaves master mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:49.255 * +slave-reconf-sent slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:49.691 # -odown master mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:54.812 # -sdown master mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:56.253 * +slave-reconf-inprog slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:56.254 * +slave-reconf-done slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:56.278 # +failover-end master mymaster 10.0.0.22 6379 5587:X 18 Sep 2024 22:20:56.278 # +switch-master mymaster 10.0.0.22 6379 10.0.0.21 6379 5587:X 18 Sep 2024 22:20:56.279 * +slave slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.21 6379 5587:X 18 Sep 2024 22:20:56.279 * +slave slave 10.0.0.22:6379 10.0.0.22 6379 @ mymaster 10.0.0.21 6379
According to the log the node with IP
10.0.0.21
is the new Master node. -
Run the following command on the second Sentinel node:
$ sudo systemctl stop redis-sentinel
-
Check
redis-sentinel.log
on the third Sentinel node:
tail -1 /var/log/redis/redis-sentinel.log 5406:X 18 Sep 2024 22:26:57.116 # +sdown sentinel 4a1264d7e6946072945569a8a6c91ad5bff182aa 10.0.0.25 26379 @ mymaster 10.0.0.21 6379
At this point, the only single Sentinel node
10.0.0.26
running, and the Redis node with IP10.0.0.21
is the new Master node. -
Run the following command on Master node:
$ redis-cli --askpass client pause 40000 Please input password: **************** OK
-
Check
redis-sentinel.log
on the third Sentinel node:
$ tail -2 /var/log/redis/redis-sentinel.log 5406:X 18 Sep 2024 22:26:57.116 # +sdown sentinel 4a1264d7e6946072945569a8a6c91ad5bff182aa 10.0.0.25 26379 @ mymaster 10.0.0.21 6379 5406:X 18 Sep 2024 22:42:37.815 # +sdown master mymaster 10.0.0.21 6379
According to the log the Master node is down, but there was no voting for the new leader. This is due to the missing of quorum among the Sentinel nodes.
-
Don't forget to start
redis-sentinel
service on the first and second Sentinel nodes:
$ sudo systemctl start redis-sentinel
Congrats! You have configured Redis Sentinel.
Summary
In this post, we’ve seen:
- how to set up Redis Master-Slave replication
- how to set up Redis Sentinel
- hot to test the failover of the high availability Redis using Sentinel.
Top comments (0)