redis实现sentinel哨兵架构
1、redis哨兵（Sentinel）
1.1、redis集群介绍
前面文章讲的主从复制集群是无法实现master和slave角色的自动切换的，如果master节点出现现redis服务异常、主机断电、磁盘损坏等问题导致master无法使用，而redis主从复制无法实现自动的故障转移(将slave 自动提升为新master)，需要手动修改环境配置,才能切换到slave redis服务器，另外当单台Redis服务器性能无法满足业务写入需求的时候,也无法横向扩展Redis服务的并行写入性能。
需要解决以上的两个核心问题：
- master和slave角色的无缝切换，让业务无感知从而不影响业务使用；
- 可横向动态扩展Redis服务器，从而实现多台服务器并行写入以实现更高并发的目的。
Redis集群实现的方式：
- 客户端分片: 由应用决定将不同的KEY发送到不同的Redis服务器
- 代理分片: 由代理决定将不同的KEY发送到不同的Redis服务器,代理程序如:codis,twemproxy等
- Redis Cluster
1.2、redis哨兵（Sentinel）的工作原理
Sentinel可以管理多个redis主从集群
Sentinel 进程是用于监控redis集群中Master主服务器工作的状态，在Master主服务器发生故障的时候，可以实现Master和Slave服务器的切换，保证系统的高可用，此功能在redis2.6+的版本已引用，Redis的哨兵模式到了2.8版本之后就稳定了下来。一般在生产环境也建议使用Redis的2.8版本的以后版本
哨兵(Sentinel) 是一个分布式系统，可以在一个架构中运行多个哨兵(sentinel) 进程，这些进程使用流言协议(gossip protocols)来接收关于Master主服务器是否下线的信息，并使用投票协议(Agreement Protocols)来决定是否执行自动故障迁移,以及选择哪个Slave作为新的Master
每个哨兵(Sentinel)进程会向其它哨兵(Sentinel)、Master、Slave定时发送消息，以确认对方是否”活”着，如果发现对方在指定配置时间(此项可配置)内未得到回应，则暂时认为对方已离线，也就是所谓的”主观认为宕机” (主观:是每个成员都具有的独自的而且可能相同也可能不同的意识)，英文名称：Subjective Down，简称SDOWN
有主观宕机，对应的有客观宕机。当“哨兵群”中的多数Sentinel进程在对Master主服务器做出SDOWN 的判断，并且通过 SENTINEL is-master-down-by-addr 命令互相交流之后，得出的Master Server下线判断，这种方式就是“客观宕机”(客观:是不依赖于某种意识而已经实际存在的一切事物)，英文名称是：Objectively Down， 简称 ODOWN
通过一定的vote算法，从剩下的slave从服务器节点中，选一台提升为Master服务器节点，然后自动修改相关配置，并开启故障转移（failover）
Sentinel 机制可以解决master和slave角色的自动切换问题，但单个 Master 的性能瓶颈问题无法解决,类似于MySQL中的MHA功能
Redis Sentinel中的Sentinel节点个数应该为大于等于3且最好为奇数
客户端初始化时连接的是Sentinel节点集合，不再是具体的Redis节点，但Sentinel只是配置中心不是代理。
Redis Sentinel节点与普通redis没有区别,要实现读写分离依赖于客户端程序
redis 3.0之前版本中,生产环境一般使用哨兵模式,3.0后推出redis cluster功能,可以支持更大规模的生产环境sentinel中的三个定时任务：
1.2.1 sentinel中的三个定时任务：
- 每10秒每个sentinel对master和slave执行info，发现slave节点，确认主从关系。
- 每2秒每个sentinel通过master节点的channel交换信息(pub/sub)通过sentinel__:hello频道交互，交互对节点的“看法”和自身信息
- 每1秒每个sentinel对其他sentinel和redis执行ping
1.3、实现哨兵
环境准备：
准备三台主机搭建主从集群，再在每个机器上搭建sentinel
| IP | redis版本 | 主机名 |
| 10.0.0.101 | redis-5.0.14 | node1 |
| 10.0.0.102 | redis-5.0.14 | node2 |
| 10.0.0.103 | redis-5.0.14 | node3 |
1.3.1、实现哨兵需要先实现一下主从复制的架构
哨兵的前提是已经实现了一个redis的主从复制的运行环境，从而实现一个一主两从基于哨兵的高可用redis架构
注意: master的配置文件中masterauth和slave都必须相同
主从复制上篇文章已经实现了，这里继续用上一篇的环境
#主节点 [root@master etc]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.102,port=6379,state=online,offset=14,lag=1 slave1:ip=10.0.0.103,port=6379,state=online,offset=14,lag=0 master_replid:e9310fdb8dd9f91d265d4c9a8621a6879e0262ff master_replid2:0000000000000000000000000000000000000000 master_repl_offset:14 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:14 127.0.0.1:6379> #slave1节点 [root@slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:slave master_host:10.0.0.101 master_port:6379 master_link_status:up master_last_io_seconds_ago:8 master_sync_in_progress:0 slave_repl_offset:224 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:e9310fdb8dd9f91d265d4c9a8621a6879e0262ff master_replid2:0000000000000000000000000000000000000000 master_repl_offset:224 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:224 #slave2节点 [root@slave2 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:slave master_host:10.0.0.101 master_port:6379 master_link_status:up master_last_io_seconds_ago:7 master_sync_in_progress:0 slave_repl_offset:2016 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:ee9d24063f9d79d698e875634517923d6a9c2a10 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:2016 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:2016
1.3.2、编辑哨兵的配置文件
Sentinel实际上是一个特殊的redis服务器,有些redis指令支持,但很多指令并不支持.默认监听在26379/tcp端口.所有redis节点使用相同的以下的配置文件.
#这里因为我是源码编译安装的，所以要人源码的路径下找到sentinel.conf将其复制到redis安装的目录下 [root@master]# cd /usr/local/src/redis-5.0.14[root@master /usr/local/src/redis-5.0.14# ls 00-RELEASENOTES CONTRIBUTING INSTALL README.md runtest-cluster sentinel.conf TLS.md BUGS COPYING Makefile redis.conf runtest-moduleapi src utils CONDUCT deps MANIFESTO runtest runtest-sentinel tests
[root@master etc]#sed -i "/^#/d" redis-sentinel.conf
[root@master etc]#sed -i "/^$/d" redis-sentinel.conf
[root@master etc]#vim redis-sentinel.con
bind 0.0.0.0 port 26379 daemonize no pidfile /apps/redis/run/redis-sentinel.pid logfile /apps/redis/log/sentinel.log dir /tmp sentinel monitor mymaster 10.0.0.100 6379 2 #mymaster是集群的名称，此行指定当前mymaster集群中master服务器的地址和端口 #2为法定人数限制(quorum)，即有几个sentinel认为master down了就进行故障转移，一般此值是所有sentinel节点(一般总数是>=3的 奇数,如:3,5,7等)的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2,是master的ODOWN客观下线的依据 sentinel auth-pass mymaster wm521314 #mymaster集群中master的密码，注意此行要在上面行的下面 sentinel down-after-milliseconds mymaster 3000 #(SDOWN)判断mymaster集群中所有节点的主观下线的时间，单位：毫秒，建议3000 sentinel parallel-syncs mymaster 1 #发生故障转移后，可以同时向新master同步数据的slave的数量，数字越小总同步时间越长，但可以减轻新master的负载压力 sentinel failover-timeout mymaster 180000 #所有slaves指向新的master所需的超时时间，单位：毫秒 sentinel deny-scripts-reconfig yes #禁止修改脚本 [root@master etc]# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.102:/apps/redis/etc/[root@master etc]# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.103:/apps/redis/etc/
三个哨兵服务器的配置都如下：
[root@slave2 etc]#cat redis-sentinel.conf port 26379 daemonize no pidfile /apps/redis/run/redis-sentinel.pid logfile /apps/redis/log/sentinel.log dir /tmp sentinel monitor mymaster 10.0.0.101 6379 2 sentinel down-after-milliseconds mymaster 30000 sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 180000 sentinel deny-scripts-reconfig yes
1.3.2、启动哨兵
#我这里是编译安装，所以在所有节点生成新的service文件 [root@master ~]#vim /lib/systemd/system/redis-sentinel.service [Unit] Description=Redis Sentinel After=network.target [Service] ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf --supervised systemd ExecStop=/bin/kill -s QUIT $MAINPID User=redis Group=redis RuntimeDirectory=redis RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target [root@master ~]#scp /lib/systemd/system/redis-sentinel.service 10.0.0.102:/lib/systemd/system/ [root@master ~]#scp /lib/systemd/system/redis-sentinel.service 10.0.0.103:/lib/systemd/system/
#在所有节点修改一下文件的所有者所属组，不然无法启动
chown -R redis.redis /apps/redis/
#在所有节点启动哨兵
[root@master ~]#systemctl daemon-reload
[root@master ~]#systemctl enable --now redis-sentinel.service
1.3.3、验证哨兵端口
[root@master ~]#ss -nlt State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 511 0.0.0.0:26379 0.0.0.0:* #已经启动，另外两个节点同样查看 LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* LISTEN 0 1024 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* LISTEN 0 511 [::]:26379 [::]:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 128 [::1]:6010 [::]:*
1.3.4、查看哨兵日志
[root@master ~]#tail -f /apps/redis/log/sentinel.log #查看主节点日志
37184:X 04 Nov 2022 23:32:07.023 # Configuration loaded
37184:X 04 Nov 2022 23:32:07.023 # systemd supervision requested, but NOTIFY_SOCKET not found
37184:X 04 Nov 2022 23:32:07.024 * Increased maximum number of open files to 10032 (it was originally set to 1024).
37184:X 04 Nov 2022 23:32:07.024 * Running mode=sentinel, port=26379.
37184:X 04 Nov 2022 23:32:07.032 # Sentinel ID is 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120
37184:X 04 Nov 2022 23:32:07.032 # +monitor master mymaster 10.0.0.101 6379 quorum 2
37184:X 04 Nov 2022 23:32:07.033 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
37184:X 04 Nov 2022 23:32:07.036 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
37184:X 04 Nov 2022 23:32:07.595 * +sentinel sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379
37184:X 04 Nov 2022 23:32:12.779 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379
#查看从节点日志
[root@slave1 ~]#tail -f /apps/redis/log/sentinel.log
6376:X 04 Nov 2022 23:31:59.446 # Configuration loaded
6376:X 04 Nov 2022 23:31:59.446 # systemd supervision requested, but NOTIFY_SOCKET not found
6376:X 04 Nov 2022 23:31:59.446 * Increased maximum number of open files to 10032 (it was originally set to 1024).
6376:X 04 Nov 2022 23:31:59.447 * Running mode=sentinel, port=26379.
6376:X 04 Nov 2022 23:31:59.455 # Sentinel ID is f505346a3640b4942963b26829e5d99a1fa6691e
6376:X 04 Nov 2022 23:31:59.455 # +monitor master mymaster 10.0.0.101 6379 quorum 2
6376:X 04 Nov 2022 23:31:59.457 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
6376:X 04 Nov 2022 23:31:59.458 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
6376:X 04 Nov 2022 23:32:08.989 * +sentinel sentinel 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120 10.0.0.101 26379 @ mymaster 10.0.0.101 6379
6376:X 04 Nov 2022 23:32:12.716 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379
1.3.5、当前sentinel状态
[root@slave1 ~]#redis-cli -p 26379 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.101:6379,slaves=2,sentinels=3
1.3.6、验证数据是否同步
[root@master ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> set k1 v1 OK 127.0.0.1:6379> set k2 v2 OK [root@slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> get k1 "v1" [root@slave2 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> get k1 "v1"
1.3.7、制造Redis Master节点故障故障转移并查看其过程
#停掉master节点的redis服务来模拟节点故障 [root@master ~]#killall redis-server #这时候查看日志发现主节点已经变成了10.0.0.102机器 [root@master ~]#tail -f /apps/redis/log/sentinel.log 800:X 06 Nov 2022 14:36:38.502 # -sdown sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379 800:X 06 Nov 2022 14:36:38.604 # -sdown slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 800:X 06 Nov 2022 15:21:26.435 # +sdown master mymaster 10.0.0.101 6379 800:X 06 Nov 2022 15:21:26.500 # +new-epoch 1 800:X 06 Nov 2022 15:21:26.501 # +vote-for-leader f505346a3640b4942963b26829e5d99a1fa6691e 1 800:X 06 Nov 2022 15:21:27.019 # +config-update-from sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379 800:X 06 Nov 2022 15:21:27.019 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379 800:X 06 Nov 2022 15:21:27.019 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379 800:X 06 Nov 2022 15:21:27.019 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 800:X 06 Nov 2022 15:21:30.032 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
#查看各节点sentinel信息
[root@master ~]#redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3
#故障转移后的sentinel配置文件会被自动修改
[root@slave1 ~]#grep "^[a-Z]" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel myid f505346a3640b4942963b26829e5d99a1fa6691e
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.0.0.102 6379 2 #这里发现主节点已经变成10.0.0.102
sentinel down-after-milliseconds mymaster 3000
sentinel auth-pass mymaster 123456
sentinel config-epoch mymaster 1
protected-mode no
supervised systemd
sentinel leader-epoch mymaster 1
sentinel known-replica mymaster 10.0.0.101 6379
sentinel known-replica mymaster 10.0.0.103 6379
sentinel known-sentinel mymaster 10.0.0.103 26379 b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a
sentinel known-sentinel mymaster 10.0.0.101 26379 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120
sentinel current-epoch 1
#进入redis查看现在info replication信息
[root@slave1 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.103,port=6379,state=online,offset=905513,lag=0
master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb
master_replid2:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:905648
second_repl_offset:537903
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:905648
[root@slave2 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe .
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:10.0.0.102
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:901300
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb
master_replid2:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:901300
second_repl_offset:537903
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:564
repl_backlog_histlen:900737
1.3.8、查看选举新主的过程日志监测
[root@slave2 ~]#tail -f /apps/redis/log/sentinel.log 798:X 06 Nov 2022 15:21:26.509 # +sdown master mymaster 10.0.0.101 6379 798:X 06 Nov 2022 15:21:26.534 # +new-epoch 1 798:X 06 Nov 2022 15:21:26.535 # +vote-for-leader f505346a3640b4942963b26829e5d99a1fa6691e 1 798:X 06 Nov 2022 15:21:26.609 # +odown master mymaster 10.0.0.101 6379 #quorum 3/2 798:X 06 Nov 2022 15:21:26.609 # Next failover delay: I will not start a failover before Sun Nov 6 15:27:27 2022 798:X 06 Nov 2022 15:21:27.053 # +config-update-from sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379 798:X 06 Nov 2022 15:21:27.053 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379 798:X 06 Nov 2022 15:21:27.053 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379 798:X 06 Nov 2022 15:21:27.053 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 798:X 06 Nov 2022 15:21:30.094 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
1.3.9、恢复故障的原master重新加入redis集群
#重新其他原master的redis服务，会发现会自动指向新的master [root@master ~]#systemctl restart redis #在原master上观察状态 [root@master ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:slave master_host:10.0.0.102 master_port:6379 master_link_status:up master_last_io_seconds_ago:1 master_sync_in_progress:0 slave_repl_offset:942522 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb master_replid2:0000000000000000000000000000000000000000 master_repl_offset:942522 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:936383 repl_backlog_histlen:6140 [root@master ~]#redis-cli -p 26379 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3 #再回到新master节点观察状态 [root@slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.103,port=6379,state=online,offset=976915,lag=0 slave1:ip=10.0.0.101,port=6379,state=online,offset=977185,lag=0 master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb master_replid2:488fa34f2c093be4886741a38323dd1a0ecf9e03 master_repl_offset:977185 second_repl_offset:537903 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:977185 #在新主查看一下日志 [root@slave1 ~]#tail -f /apps/redis/log/sentinel.log 786:X 06 Nov 2022 15:21:27.030 * +slave-reconf-sent slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:27.580 * +slave-reconf-inprog slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:27.639 # -odown master mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:28.621 * +slave-reconf-done slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:28.672 # +failover-end master mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:28.672 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379 786:X 06 Nov 2022 15:21:28.672 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379 786:X 06 Nov 2022 15:21:28.672 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 786:X 06 Nov 2022 15:21:31.738 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 786:X 06 Nov 2022 15:54:28.253 # -sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
