Redis 哨兵 Sentinel 配置

节点规划

redis-01 192.168.174.108   16379
redis-02 192.168.174.112   16379
redis-03 192.168.174.117   16379

chown -R redis:redis /usr/local/redis/etc/

redis 主从配置

在所有节点执行

sed -i -e 's@port 6379@port 16379@' -e 's@bind 127.0.0.1@bind 0.0.0.0@g' -e '/# requirepass /a requirepass 123456' -e 's@pidfile /var/run/redis_6379.pid@pidfile /data/apps/redis/redis.pid@' -e 's@dir ./@dir /data/apps/redis/data@' -e 's@logfile ""@logfile /data/apps/redis/logs/redis.log@' -e  's/^# masterauth .*/masterauth 123456/' /usr/local/redis/etc/redis.conf

在 slave 节点执行

echo "replicaof 192.168.174.108 16379" >> /usr/local/redis/etc/redis.conf

redis 服务重启

systemctl restart redis-server 

查看 redis 主从状态

连接 redis 

/usr/local/redis/bin/redis-cli -p 16379 -a 123456 

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:16379> 

查看 master 状态

127.0.0.1:16379> info replication

# Replication
role:master
connected_slaves:2
slave0:ip=192.168.174.112,port=16379,state=online,offset=56,lag=0
slave1:ip=192.168.174.117,port=16379,state=online,offset=56,lag=1
master_failover_state:no-failover
master_replid:97c2e4f7cd3b2c0809d9ee2213e348929fc33ccb
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:56
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:56

查看 slave 状态

127.0.0.1:16379> info replication

# Replication
role:slave
master_host:192.168.174.108
master_port:16379
master_link_status:up
master_last_io_seconds_ago:7
master_sync_in_progress:0
slave_read_repl_offset:154
slave_repl_offset:154
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:97c2e4f7cd3b2c0809d9ee2213e348929fc33ccb
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:154
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:57
repl_backlog_histlen:98

Sentinel 配置

sentinel.conf

port 26379 
daemonize yes
pidfile /var/run/redis-sentinel.pid
logfile ""
dir /tmp

# mymaster 是集群的名称，此行指定当前mymaster集群中master服务器的地址和端口 2为法定人数限制(quorum)，即有几个sentinel认为master down了就进行故障转移，一般此值是所有sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值，比如，总数是3，即3/2=1.5,取整为2,是master的ODOWN客观下线的依据
sentinel monitor mymaster 127.0.0.1 6379 2

# master 节点的密码 
sentinel auth-pass mymaster 123456

# 判断 mymaster 所有节点主观下线时间。 单位: 毫秒
sentinel down-after-milliseconds mymaster 30000

# 发生故障转移后，可以同时向新的master节点同步数据的slave的数量，数字越小总同步时间越长，但可以减轻master节点压力
sentinel parallel-syncs mymaster 1

# 所有slave节点指向新master节点的超时时间。默认：3分钟
sentinel failover-timeout mymaster 180000

# 禁止修改脚本
sentinel deny-scripts-reconfig yes

命令行修改

sed -i -e 's@^pidfile .*@pidfile /data/apps/redis/redis-sentinel.pid@' -e 's@^logfile .*@logfile /data/apps/redis/logs/redis-sentinel.log@' -e 's@dir /tmp@dir /data/apps/redis/data@' -e 's@sentinel monitor mymaster 127.0.0.1 6379 2@sentinel monitor mymaster 192.168.174.108 16379 2@' -e '/# sentinel auth-pass mymaster/a sentinel auth-pass mymaster 123456' -e 's/sentinel down-after-milliseconds mymaster 30000/sentinel down-after-milliseconds mymaster 3000/' /usr/local/redis/etc/sentinel.conf

redis-sentinel.service

cat > /lib/systemd/system/redis-sentinel.service <<EOF
[Unit]
Description=Redis persistent key-vaue database
After=network.target

[Service]
Restart=always
ExecStart=/usr/local/redis/bin/redis-sentinel /usr/local/redis/etc/sentinel.conf --supervised systemd
ExecStop=/bin/ki11 -S QUIT 
Type=notify 
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
LimitNOFILE=1000000  
TimeoutStartSec=30
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

EOF

systemctl enable redis-sentinel --now

验证 sentinel 服务

查看端口信息

netstat -tnlp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:26379           0.0.0.0:*               LISTEN      1910/redis-sentinel 
tcp        0      0 0.0.0.0:16379           0.0.0.0:*               LISTEN      1479/redis-server 0 

查看 sentinel id 信息

master 节点

grep myid /usr/local/redis/etc/sentinel.conf

sentinel myid 1be7ad30ff9a6b761b91362502124a777ea1908c

slave-01 节点

grep myid /usr/local/redis/etc/sentinel.conf

sentinel myid 4e075ba63eb20145b9dd0e5ed19eef0c1ecd585c

slave-02 节点

grep myid /usr/local/redis/etc/sentinel.conf

sentinel myid 88fad7df3bad3f474960131587fcddcbba39b333

查看 sentinel 日志信息

master 节点日志

cat redis-sentinel.log

1910:X 16 Dec 2024 08:59:05.756 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
1910:X 16 Dec 2024 08:59:05.756 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1910:X 16 Dec 2024 08:59:05.756 * Redis version=7.4.1, bits=64, commit=00000000, modified=0, pid=1910, just started
1910:X 16 Dec 2024 08:59:05.756 * Configuration loaded
1910:X 16 Dec 2024 08:59:05.757 * monotonic clock: POSIX clock_gettime
1910:X 16 Dec 2024 08:59:05.759 * Running mode=sentinel, port=26379.
1910:X 16 Dec 2024 08:59:05.762 * Sentinel new configuration saved on disk
1910:X 16 Dec 2024 08:59:05.763 * Sentinel ID is 1be7ad30ff9a6b761b91362502124a777ea1908c
1910:X 16 Dec 2024 08:59:05.763 # +monitor master mymaster 192.168.174.108 16379 quorum 2
1910:X 16 Dec 2024 08:59:05.765 * +slave slave 192.168.174.112:16379 192.168.174.112 16379 @ mymaster 192.168.174.108 16379
1910:X 16 Dec 2024 08:59:05.769 * Sentinel new configuration saved on disk
1910:X 16 Dec 2024 08:59:05.769 * +slave slave 192.168.174.117:16379 192.168.174.117 16379 @ mymaster 192.168.174.108 16379
1910:X 16 Dec 2024 08:59:05.771 * Sentinel new configuration saved on disk
1910:X 16 Dec 2024 08:59:07.783 * +sentinel sentinel 88fad7df3bad3f474960131587fcddcbba39b333 192.168.174.117 26379 @ mymaster 192.168.174.108 16379
1910:X 16 Dec 2024 08:59:07.784 * Sentinel new configuration saved on disk
1910:X 16 Dec 2024 08:59:07.785 * +sentinel sentinel 4e075ba63eb20145b9dd0e5ed19eef0c1ecd585c 192.168.174.112 26379 @ mymaster 192.168.174.108 16379
1910:X 16 Dec 2024 08:59:07.787 * Sentinel new configuration saved on disk

slave 节点日志

cat redis-sentinel.log

6973:X 16 Dec 2024 08:59:05.764 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
6973:X 16 Dec 2024 08:59:05.764 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
6973:X 16 Dec 2024 08:59:05.764 * Redis version=7.4.1, bits=64, commit=00000000, modified=0, pid=6973, just started
6973:X 16 Dec 2024 08:59:05.764 * Configuration loaded
6973:X 16 Dec 2024 08:59:05.765 * monotonic clock: POSIX clock_gettime
6973:X 16 Dec 2024 08:59:05.767 * Running mode=sentinel, port=26379.
6973:X 16 Dec 2024 08:59:05.770 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 08:59:05.770 * Sentinel ID is 88fad7df3bad3f474960131587fcddcbba39b333
6973:X 16 Dec 2024 08:59:05.770 # +monitor master mymaster 192.168.174.108 16379 quorum 2
6973:X 16 Dec 2024 08:59:05.774 * +slave slave 192.168.174.112:16379 192.168.174.112 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 08:59:05.777 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 08:59:05.777 * +slave slave 192.168.174.117:16379 192.168.174.117 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 08:59:05.779 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 08:59:07.784 * +sentinel sentinel 4e075ba63eb20145b9dd0e5ed19eef0c1ecd585c 192.168.174.112 26379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 08:59:07.786 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 08:59:07.797 * +sentinel sentinel 1be7ad30ff9a6b761b91362502124a777ea1908c 192.168.174.108 26379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 08:59:07.800 * Sentinel new configuration saved on disk

查看 sentinel 状态

连接 sentinel

 /usr/local/redis/bin/redis-cli -p 26379 

127.0.0.1:26379> 

sentinel 状态

127.0.0.1:26379> info sentinel

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.174.108:16379,slaves=2,sentinels=3

sentinel 故障转移

查看 redis master 节点 

127.0.0.1:26379> info sentinel

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.174.108:16379,slaves=2,sentinels=3

停止 redis master 节点

systemctl stop redis-server

查看 sentinel 日志

tail -f /data/apps/redis/logs/redis-sentinel.log

6973:X 16 Dec 2024 09:16:21.032 # +sdown master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.095 # +odown master mymaster 192.168.174.108 16379 #quorum 2/2
6973:X 16 Dec 2024 09:16:21.096 # +new-epoch 1
6973:X 16 Dec 2024 09:16:21.096 # +try-failover master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.098 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 09:16:21.098 # +vote-for-leader 88fad7df3bad3f474960131587fcddcbba39b333 1
6973:X 16 Dec 2024 09:16:21.106 * 1be7ad30ff9a6b761b91362502124a777ea1908c voted for 88fad7df3bad3f474960131587fcddcbba39b333 1
6973:X 16 Dec 2024 09:16:21.107 * 4e075ba63eb20145b9dd0e5ed19eef0c1ecd585c voted for 88fad7df3bad3f474960131587fcddcbba39b333 1
6973:X 16 Dec 2024 09:16:21.162 # +elected-leader master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.162 # +failover-state-select-slave master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.253 # +selected-slave slave 192.168.174.112:16379 192.168.174.112 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.253 * +failover-state-send-slaveof-noone slave 192.168.174.112:16379 192.168.174.112 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.331 * +failover-state-wait-promotion slave 192.168.174.112:16379 192.168.174.112 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.718 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 09:16:21.718 # +promoted-slave slave 192.168.174.112:16379 192.168.174.112 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.718 # +failover-state-reconf-slaves master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:21.771 * +slave-reconf-sent slave 192.168.174.117:16379 192.168.174.117 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:22.201 # -odown master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:22.726 * +slave-reconf-inprog slave 192.168.174.117:16379 192.168.174.117 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:22.726 * +slave-reconf-done slave 192.168.174.117:16379 192.168.174.117 16379 @ mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:22.818 # +failover-end master mymaster 192.168.174.108 16379
6973:X 16 Dec 2024 09:16:22.818 # +switch-master mymaster 192.168.174.108 16379 192.168.174.112 16379
6973:X 16 Dec 2024 09:16:22.818 * +slave slave 192.168.174.117:16379 192.168.174.117 16379 @ mymaster 192.168.174.112 16379
6973:X 16 Dec 2024 09:16:22.818 * +slave slave 192.168.174.108:16379 192.168.174.108 16379 @ mymaster 192.168.174.112 16379
6973:X 16 Dec 2024 09:16:22.820 * Sentinel new configuration saved on disk
6973:X 16 Dec 2024 09:16:25.843 # +sdown slave 192.168.174.108:16379 192.168.174.108 16379 @ mymaster 192.168.174.112 16379

验证故障转移

查看 sentinel 状态

/usr/local/redis/bin/redis-cli -p 26379 

127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.174.112:16379,slaves=2,sentinels=3

查看 redis 状态

/usr/local/redis/bin/redis-cli -p 16379 -a 123456

127.0.0.1:16379> info replication
# Replication
role:slave
master_host:192.168.174.112
master_port:16379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:332529
slave_repl_offset:332529
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:b36ee5353ca96257081fe29516dcb5948a15f153
master_replid2:97c2e4f7cd3b2c0809d9ee2213e348929fc33ccb
master_repl_offset:332529
second_repl_offset:227445
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:57
repl_backlog_histlen:332473

查看 redis 配置文件

grep ^replicaof  /usr/local/redis/etc/redis.conf

replicaof 192.168.174.112 16379

查看 sentinel 配置文件

grep "^sentinel monitor"  /usr/local/redis/etc/sentinel.conf

sentinel monitor mymaster 192.168.174.112 16379 2

故障节点从新上线

重启 redis 服务

systemctl restart redis-server

在故障节点查看 redis 信息

127.0.0.1:16379> info replication
# Replication
role:slave
master_host:192.168.174.112
master_port:16379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:392060
slave_repl_offset:392060
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:b36ee5353ca96257081fe29516dcb5948a15f153
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:392060
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:361182
repl_backlog_histlen:30879

在新 master 节点查看 redis 信息

127.0.0.1:16379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.174.117,port=16379,state=online,offset=408072,lag=0
slave1:ip=192.168.174.108,port=16379,state=online,offset=408072,lag=0
master_failover_state:no-failover
master_replid:b36ee5353ca96257081fe29516dcb5948a15f153
master_replid2:97c2e4f7cd3b2c0809d9ee2213e348929fc33ccb
master_repl_offset:408072
second_repl_offset:227445
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:15
repl_backlog_histlen:408058

查看 sentinel 日志

6973:X 16 Dec 2024 09:26:24.579 # -sdown slave 192.168.174.108:16379 192.168.174.108 16379 @ mymaster 192.168.174.112 16379

参考文档

https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/#obtaining-sentinel