DEV Community

Ježek
Ježek

Posted on • Updated on

Set up a Redis Sentinel

Redis® Sentinel provides high availability for Redis. Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.

This article shows how to set up a high-availability architecture for Redis using Sentinel in Debian GNU/Linux:

Redis Sentinel


Prerequisites

Debian preinstalled servers, e.g.:

Host name IP Role
redis-1 10.0.0.21 Master node
redis-2 10.0.0.22 Slave node
redis-3 10.0.0.23 Slave node
sentinel-1 10.0.0.24 Sentinel node
sentinel-2 10.0.0.25 Sentinel node
sentinel-3 10.0.0.26 Sentinel node

Setup Redis replication

Install Redis

  1. Connect to hosts with roles: Master node, Slave node
  2. Run the following commands:

    $ sudo apt-get update
    $ sudo apt-get install -y redis-server
    
  3. Activate the redis-server service to start when Redis server boots:

    $ sudo systemctl enable redis-server
    
  4. Check redis-server status:

    $ systemctl status redis-server
    ● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor >
     Active: active (running) since Mon 2024-09-16 21:06:13; 19min ago
       Docs: http://redis.io/documentation,
             man:redis-server(1)
    Main PID: 3057 (redis-server)
     Status: "Ready to accept connections"
      Tasks: 5 (limit: 2307)
     Memory: 9.2M
        CPU: 2.146s
     CGroup: /system.slice/redis-server.service
             └─3057 /usr/bin/redis-server 127.0.0.1:6379
    
    

    Service runs and ready to accept connections.

Set up the Master node

  1. Open the configuration file at /etc/redis/redis.conf
  2. Set following mandatory parameters:

    protected-mode yes
    bind <masterip: 10.0.0.21> 127.0.0.1
    requirepass <master-password>
    masterauth <master-password>
    
  3. Restart the redis-server service:

    $ sudo systemctl restart redis-server
    
  4. Make sure that Redis accepts connections with authentication:

    $ redis-cli --askpass info
    

Set up the Slave nodes

  1. Open the configuration file at /etc/redis/redis.conf
  2. Set following mandatory parameters:

    bind <slaveip: 10.0.0.21/22> 127.0.0.1
    requirepass <master-password>
    masterauth <master-password>
    replicaof <masterip: 10.0.0.21> 6379
    
  3. Restart the redis-server service:

    $ sudo systemctl restart redis-server
    
  4. Make sure that Redis accepts connections with authentication:

    $ redis-cli --askpass info
    

Check Redis replication setup

  1. Check the redis-server log on Master node:

    $ tail /var/log/redis/redis-server.log -n 20
    3498:M 16 Sep 2024 22:25:53.086 * RDB memory usage when created 0.77 Mb
    3498:M 16 Sep 2024 22:25:53.086 * DB loaded from disk: 0.000 seconds
    3498:M 16 Sep 2024 22:25:53.086 * Ready to accept connections
    3498:M 16 Sep 2024 23:29:52.119 * Replica 10.0.0.22:6379 asks for synchronization
    3498:M 16 Sep 2024 23:29:52.119 * Full resync requested by replica 10.0.0.22:6379
    3498:M 16 Sep 2024 23:29:52.119 * Replication backlog created, my new replication IDs are '368571d1e52c29535615e0043a3a6d6ddc1db70b' and '0000000000000000000000000000000000000000'
    3498:M 16 Sep 2024 23:29:52.119 * Starting BGSAVE for SYNC with target: disk
    3498:M 16 Sep 2024 23:29:52.119 * Background saving started by pid 3684
    3684:C 16 Sep 2024 23:29:52.132 * DB saved on disk
    3684:C 16 Sep 2024 23:29:52.133 * RDB: 0 MB of memory used by copy-on-write
    3498:M 16 Sep 2024 23:29:52.185 * Background saving terminated with success
    3498:M 16 Sep 2024 23:29:52.185 * Synchronization with replica 10.0.0.22:6379 succeeded
    3498:M 16 Sep 2024 23:30:09.952 * Replica 10.0.0.23:6379 asks for synchronization
    3498:M 16 Sep 2024 23:30:09.952 * Full resync requested by replica 10.0.0.23:6379
    3498:M 16 Sep 2024 23:30:09.952 * Starting BGSAVE for SYNC with target: disk
    3498:M 16 Sep 2024 23:30:09.952 * Background saving started by pid 3688
    3688:C 16 Sep 2024 23:30:09.965 * DB saved on disk
    3688:C 16 Sep 2024 23:30:09.965 * RDB: 0 MB of memory used by copy-on-write
    3498:M 16 Sep 2024 23:30:09.985 * Background saving terminated with success
    3498:M 16 Sep 2024 23:30:09.985 * Synchronization with replica 10.0.0.23:6379 succeeded
    
  2. Check replication status on Master node:

    $ redis-cli --askpass info replication
    Please input password: ****************
    # Replication
    role:master
    connected_slaves:2
    slave0:ip=10.0.0.22,port=6379,state=online,offset=1652,lag=1
    slave1:ip=10.0.0.23,port=6379,state=online,offset=1652,lag=1
    master_replid:368571d1e52c29535615e0043a3a6d6ddc1db70b
    master_replid2:0000000000000000000000000000000000000000
    master_repl_offset:1652
    second_repl_offset:-1
    repl_backlog_active:1
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:1
    repl_backlog_histlen:1652
    
  3. Run the following command on Master node:

    $ $ redis-cli --askpass set abc 123
    Please input password: ****************
    OK
    
  4. Run the following command on Slave nodes:

    $ redis-cli --askpass get abc
    Please input password: ****************
    "123"
    

Congrats! Master-to-Slave data replication succeed.


Setup Redis Sentinel

Install Redis Sentinel

  1. Connect to hosts with the role Sentinel
  2. Run the following commands:

    $ sudo apt-get update
    $ sudo apt-get install -y redis-sentinel
    
  3. Activate the redis-sentinel service to start when Sentinel server boots:

    $ sudo systemctl enable redis-sentinel
    
  4. Check redis-sentinel status:

    $ systemctl status redis-sentinel
    ● redis-sentinel.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-sentinel.service; enabled; vendo>
     Active: active (running) since Tue 2024-09-17 00:56:22 MSK; 1min 29s ago
       Docs: http://redis.io/documentation,
             man:redis-sentinel(1)
    Main PID: 3259 (redis-sentinel)
     Status: "Ready to accept connections"
      Tasks: 5 (limit: 2307)
     Memory: 7.2M
        CPU: 190ms
     CGroup: /system.slice/redis-sentinel.service
             └─3259 /usr/bin/redis-sentinel 127.0.0.1:26379 [sentinel]
    

Set up Sentinel nodes

  1. Open the configuration file at /etc/redis/sentinel.conf
  2. Set following mandatory parameters:

    protected-mode yes
    bind <sentinelip: 10.0.0.24/25/26> 127.0.0.1
    sentinel monitor mymaster <masterip: 10.0.0.21> 6379 2
    sentinel down-after-milliseconds mymaster 30000
    sentinel failover-timeout mymaster 180000
    sentinel parallel-syncs mymaster 1
    sentinel auth-pass mymaster <master-password>
    requirepass <sentinel-password>
    
  3. Restart the redis-sentinel service:

    $ sudo systemctl restart redis-sentinel
    
  4. Make sure that Redis Sentinel accepts connections with authentication:

    $ redis-cli -p 26379 --askpass info sentinel
    Please input password: ****************
    # Sentinel
    sentinel_masters:1
    sentinel_tilt:0
    sentinel_running_scripts:0
    sentinel_scripts_queue_length:0
    sentinel_simulate_failure_flags:0
    master0:name=mymaster,status=ok,address=10.0.0.21:6379,slaves=2,sentinels=3
    
    

Testing the failover

This section shows how to test the failover of the high availability Redis using Sentinel.

Master node failure

  1. Run the following command on Master node:

    $ redis-cli --askpass client pause 40000
    Please input password: ****************
    OK
    

    Note: client pause 40000 due to sentinel down-after-milliseconds mymaster 30000

  2. Run the following command on one of the Sentinel nodes:

    $ tail -10 /var/log/redis/redis-sentinel.log
    6175:X 18 Sep 2024 21:39:49.309 # +sdown master mymaster 10.0.0.21 6379
    6175:X 18 Sep 2024 21:39:49.555 # +new-epoch 1
    6175:X 18 Sep 2024 21:39:49.558 # +vote-for-leader 77fc4fbc9c6c7d063d62e38f528910ab67e8d907 1
    6175:X 18 Sep 2024 21:39:50.450 # +odown master mymaster 10.0.0.21 6379 #quorum 3/2
    6175:X 18 Sep 2024 21:39:50.451 # Next failover delay: I will not start a failover before Wed Sep 18 21:45:50 2024
    6175:X 18 Sep 2024 21:39:51.853 # +config-update-from sentinel 77fc4fbc9c6c7d063d62e38f528910ab67e8d907 10.0.0.26 26379 @ mymaster 10.0.0.21 6379
    6175:X 18 Sep 2024 21:39:51.853 # +switch-master mymaster 10.0.0.21 6379 10.0.0.22 6379
    6175:X 18 Sep 2024 21:39:51.853 * +slave slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
    6175:X 18 Sep 2024 21:39:51.853 * +slave slave 10.0.0.21:6379 10.0.0.21 6379 @ mymaster 10.0.0.22 6379
    6175:X 18 Sep 2024 21:40:09.323 * +convert-to-slave slave 10.0.0.21:6379 10.0.0.21 6379 @ mymaster 10.0.0.22 6379
    

    Note: Next failover delay: I will not start a failover before Wed Sep 18 21:45:50 2024 due to sentinel failover-timeout mymaster 180000

  3. Run the following command on one of the Sentinel nodes:

    $ redis-cli -p 26379 --askpass sentinel get-master-addr-by-name mymaster
    Please input password: ****************
    1) "10.0.0.22"
    2) "6379"
    

    Redis node with IP 10.0.0.22 is the new Master node.

  4. Run the following command on the new Master node:

    $ redis-cli --askpass info replication
    # Replication
    role:master
    connected_slaves:2
    slave0:ip=10.0.0.23,port=6379,state=online,offset=519426,lag=1
    slave1:ip=10.0.0.21,port=6379,state=online,offset=519559,lag=0
    master_replid:ade635098e62dc68f3ee65acf37970f98e1a2f8a
    master_replid2:8777248b814d71f4c609f437c37a529e01b7d0c5
    master_repl_offset:519692
    second_repl_offset:195253
    repl_backlog_active:1
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:1
    repl_backlog_histlen:519692
    

Sentinel node failure

  1. Run the following command on the first Sentinel node:

    $ sudo systemctl stop redis-sentinel
    
  2. Check redis-sentinel.log on the second Sentinel node:

    tail -1 /var/log/redis/redis-sentinel.log
    5587:X 18 Sep 2024 22:14:36.112 # +sdown sentinel d01aa131dc0ba6cc4069f3ec67bf6b4fa6b501f8 10.0.0.24 26379 @ mymaster 10.0.0.22 6379
    
  3. Run the following command on Master node:

    $ redis-cli --askpass client pause 40000
    Please input password: ****************
    OK
    
  4. Check redis-sentinel.log on the second Sentinel node:

    $ tail -10 /var/log/redis/redis-sentinel.log
    5587:X 18 Sep 2024 22:20:49.232 # +failover-state-reconf-slaves master mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:49.255 * +slave-reconf-sent slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:49.691 # -odown master mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:54.812 # -sdown master mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:56.253 * +slave-reconf-inprog slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:56.254 * +slave-reconf-done slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:56.278 # +failover-end master mymaster 10.0.0.22 6379
    5587:X 18 Sep 2024 22:20:56.278 # +switch-master mymaster 10.0.0.22 6379 10.0.0.21 6379
    5587:X 18 Sep 2024 22:20:56.279 * +slave slave 10.0.0.23:6379 10.0.0.23 6379 @ mymaster 10.0.0.21 6379
    5587:X 18 Sep 2024 22:20:56.279 * +slave slave 10.0.0.22:6379 10.0.0.22 6379 @ mymaster 10.0.0.21 6379
    

    According to the log the node with IP 10.0.0.21 is the new Master node.

  5. Run the following command on the second Sentinel node:

    $ sudo systemctl stop redis-sentinel
    
  6. Check redis-sentinel.log on the third Sentinel node:

    tail -1 /var/log/redis/redis-sentinel.log
    5406:X 18 Sep 2024 22:26:57.116 # +sdown sentinel 4a1264d7e6946072945569a8a6c91ad5bff182aa 10.0.0.25 26379 @ mymaster 10.0.0.21 6379
    

    At this point, the only single Sentinel node 10.0.0.26 running, and the Redis node with IP 10.0.0.21 is the new Master node.

  7. Run the following command on Master node:

    $ redis-cli --askpass client pause 40000
    Please input password: ****************
    OK
    
  8. Check redis-sentinel.log on the third Sentinel node:

    $ tail -2 /var/log/redis/redis-sentinel.log
    5406:X 18 Sep 2024 22:26:57.116 # +sdown sentinel 4a1264d7e6946072945569a8a6c91ad5bff182aa 10.0.0.25 26379 @ mymaster 10.0.0.21 6379
    5406:X 18 Sep 2024 22:42:37.815 # +sdown master mymaster 10.0.0.21 6379
    

    According to the log the Master node is down, but there was no voting for the new leader. This is due to the missing of quorum among the Sentinel nodes.

  9. Don't forget to start redis-sentinel service on the first and second Sentinel nodes:

    $ sudo systemctl start redis-sentinel
    

Congrats! You have configured Redis Sentinel.


Summary

In this post, we’ve seen:

  • how to set up Redis Master-Slave replication
  • how to set up Redis Sentinel
  • hot to test the failover of the high availability Redis using Sentinel.

Top comments (0)