返回顶部

博客-redis主从、哨兵sentinel和cluster集群

1.RDB和AOF的优缺点

 

RDB优点:

  1. RDB快照保存了某个时间点的数据,可以通过脚本执行redis指令bgsave(非阻塞,后台执行)或者save(会阻塞写操作,不推荐)命令自定义时间点备份,
  2. 可以保留多个备份,当出现问题可以恢复到不同时间点的版本,很适合备份。
  3. 并且此文件格式也支持不少第三方工具可以进行后续的数据分析。

4.RDB可以最大化redis的性能,父进程在保存RDB文件时唯一要做的就是fork出一个子进程,然后这个子进程就会处理接下来的所有保存工作,父进程无需执行热河磁盘IO操作。5.RDB在大量数据比如几个G数据,恢复速度比AOF快。

RDB缺点:

不能实时保存数据,可能丢失自上一次RDB备份到当前的内存数据。当数据量非常大的时候,从父进程fork子进程进行保存至RDB文件时需要一点时间,可能是毫秒或是秒,取决于磁盘IO性能在数据集比较庞大时,fork()可能会非常耗时,造成服务器在一定时间内停止处理客户端

 

AOF优点:

  1. 数据安全性相对较高,根据所使用的fsync策略(fsync是同步于redis所有已经修改的文件到存储设备),默认是appendfsync  everysec  ,即每秒执行一次fsync,在这种配置下,redis仍然可以保持良好性能,并且就算发生故障停机,也最多只会丢失一秒钟 数据(fsync 会在后台线程执行,所以主线程可以继续努力处理命令请求)
  2. 由于该机制对日志文件的写入操作采用的是append模式,因此在写入过程中不需要seek,即使出现宕机现象,也不会破坏日志中已经存在的内容。
  3. Redis可以在AOF文件体积变得过大时自动的在后台对AOF进行重写:重写后的新AOF文件包含了恢复当前数据所需要的最小命令集合
  4. AOF包含了一个格式清晰、易于理解的日志文件用于记录所有的修改操作

缺点:

1.即使有些操作是重复的也会全部记录,AOF的文件大小要大于RDB格式的文件;

2.AOF在恢复大数据的速度比RDB的恢复速度慢;

3.根据fsync策略不同,AOF速度可能会慢于RDB

4.Bug 出现的可能性更多

RDB 和AOF的选择

如果主要充当缓存功能,或者可以承受数分钟的数据丢失,通常生产环境一般只需启用RDB即可,此也是默认值

如果数据需要持久保存,一点也不能丢失,可以选择同时开启RDB和AOF,一般不建议只开启AOF

 

2. redis的Master和slave同步过程

即redis主从复制

 

 

 

环境准备:

master:10.0.0.18

slave1 :10.0.0.28

slave2: 10.0.0.38

#mater上设置key1
[root@centos8  ~]#vim /etc/redis.conf
appendonly yes    #开启持久化
bind 0.0.0.0      #开启主从同步
requirepass 123456


#systemctl start redis

[root@centos8 ~]#redis-cli    #redis-cli  -a 123456
127.0.0.1:6379> AUTH 123456
OK
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:0
master_replid:eccadbe6c2e8b7f1ee9dd3a06bb3b59e698434b5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0


127.0.0.1:6379> SET key1 v1-master
OK
127.0.0.1:6379> KEYS *   #生产慎用
1) "key1"
127.0.0.1:6379> GET key1
"v1-master"
127.0.0.1:6379>

#以下都在slave1上执行,登录
[root@centos8 ~]#redis-cli  
127.0.0.1:6379> info
NOAUTH Authentication required.
127.0.0.1:6379> AUTH 123456
OK
127.0.0.1:6379> INFO replication  #查看当前角色默认为master

# Replication
role:master    #默认master
connected_slaves:0
master_replid:cdfdaa8fa64e76bfeacbb2ad8b12644330663212
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

127.0.0.1:6379> set key1 v1-slave-28
OK
127.0.0.1:6379> KEYS *
1) "key1"
127.0.0.1:6379> get key1
"v1-slave-28"
127.0.0.1:6379>

#在第二个slave2,也设置相同的key1,但值不同
127.0.0.1:6379> KEYS *
1) "key1"
127.0.0.1:6379> get key1
"v1-slave-38"
127.0.0.1:6379>

127.0.0.1:6379> INFO replication
# Replication
role:master    #默认master
connected_slaves:0
master_replid:9d8630e8344e938f6e81b281d7b9c3f44d74ebe4
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0


#slave上设置masterIP和端口,4.0版之前的指令为slaveof
127.0.0.1:6379> REPLICAOF 10.0.0.18 6379 #仍可使用SLAVEOF MasterIP Port
OK
#slave上设置master的密码,才可以同步
127.0.0.1:6379> CONFIG SET masterauth 123456
OK
127.0.0.1:6379> INFO replication
# Replication   #角色变为slave
role:slave
master_host:10.0.0.18   #指向masterIP
master_port:6379
master_link_status:up
master_last_io_seconds_ago:4
master_sync_in_progress:0
slave_repl_offset:70
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:70
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:70


#查看已经同步成功
127.0.0.1:6379> GET key1
"v1-master"


#master上可以看到所有slave信息
127.0.0.1:6379> INFO replication
# Replication
role:master
connected_slaves:2   #连接从节点数量有2
slave0:ip=10.0.0.38,port=6379,state=online,offset=420,lag=1   #slave信息
slave1:ip=10.0.0.28,port=6379,state=online,offset=420,lag=1
master_replid:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:420
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:420

删除主从同步

REPLIATOF NO ONE 指令可以取消主从复制

#取消复制,slave上执行,会断开和master的连接不再主从复制, 但不会清除slave上已有的数据
127.0.0.1:6379> REPLICAOF no one

master观察日志

[root@centos8  ~]#tail -f /var/log/redis/redis.log 36766:M 22 Oct 2020 16:15:16.432 * Background saving terminated with success
36766:M 22 Oct 2020 16:15:16.433 * Synchronization with replica 10.0.0.38:6379 succeeded
36766:M 22 Oct 2020 16:19:46.148 * Replica 10.0.0.28:6379 asks for synchronization
36766:M 22 Oct 2020 16:19:46.148 * Full resync requested by replica 10.0.0.28:6379
36766:M 22 Oct 2020 16:19:46.148 * Starting BGSAVE for SYNC with target: disk
36766:M 22 Oct 2020 16:19:46.150 * Background saving started by pid 36929
36929:C 22 Oct 2020 16:19:46.156 * DB saved on disk
36929:C 22 Oct 2020 16:19:46.157 * RDB: 2 MB of memory used by copy-on-write
36766:M 22 Oct 2020 16:19:46.157 * Background saving terminated with success
36766:M 22 Oct 2020 16:19:46.158 * Synchronization with replica 10.0.0.28:6379 succeeded

slave观察日志

[root@centos28  ~]#tail -f /var/log/redis/redis.log
2682:S 22 Oct 2020 16:19:45.378 * Full resync from master: 34ead6a506afd682dc7e7564c6fd429e8fc7042f:378
2682:S 22 Oct 2020 16:19:45.383 * MASTER <-> REPLICA sync: receiving 226 bytes from master
2682:S 22 Oct 2020 16:19:45.383 * MASTER <-> REPLICA sync: Flushing old data
2682:S 22 Oct 2020 16:19:45.383 * MASTER <-> REPLICA sync: Loading DB in memory
2682:S 22 Oct 2020 16:19:45.384 * MASTER <-> REPLICA sync: Finished with success
2682:S 22 Oct 2020 16:22:54.021 * 1 changes in 900 seconds. Saving...
2682:S 22 Oct 2020 16:22:54.142 * Background saving started by pid 2692
2692:C 22 Oct 2020 16:22:54.157 * DB saved on disk
2692:C 22 Oct 2020 16:22:54.158 * RDB: 4 MB of memory used by copy-on-write
2682:S 22 Oct 2020 16:22:54.244 * Background saving terminated with success

修改slave节点配置文件

[root@centos8  ~]#vim /etc/redis.conf
.............
 replicaof 10.0.0.0.18 6379  #指定masterIP和端口号
 masterauth  123456    #如果密码需要设置
。。。。。。略......

[root@centos8  ~]#systemctl restart redis

master和slave查看状态

#master上查看状态
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.38,port=6379,state=online,offset=2520,lag=0
slave1:ip=10.0.0.28,port=6379,state=online,offset=2520,lag=0
master_replid:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2520
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:2520

#slave上查看状态
127.0.0.1:6379> get key1    
"v1-master"     #同步成功后,slave上的key信息丢失,从master复制过来新的值
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.18
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:2660
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2660
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:2660

#停止masterredis服务:systemctl stop redis ,在slave上可以看到以下现象
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.18
master_port:6379
master_link_status:down   #显示无法连接masterdown
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:2940
master_link_down_since_seconds:20
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2940
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:2940

slave 状态只读无法写入数据

127.0.0.1:6379> set key1 v1-slave
(error) READONLY You can't write against a read only replica.

主从故障恢复

当前主节点10.0.0.18故障,提升10.0.0.28为新的master

#查看当前10.0.0.28节点状态为slavemaster指向10.0.0.0.18
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.18
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:2940
master_link_down_since_seconds:380
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2940
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2493
repl_backlog_histlen:448

停止slave同步并提升为新的master

#提升为master角色
127.0.0.1:6379> REPLICAOF no one   #旧版使用slaveof  no  one
OK
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:0
master_replid:19f022e5662d3a7d6fa5902cbd3bab54ffa9b5d4
master_replid2:34ead6a506afd682dc7e7564c6fd429e8fc7042f
master_repl_offset:2940
second_repl_offset:2941
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2493
repl_backlog_histlen:448


[root@centos8  ~]#vim /etc/redis.conf
appendonly yes
bind 0.0.0.0
requirepass 123456
replicaof 10.0.0.0.18 6379    #注释掉

[root@centos8  ~]#systemctl restart redis

测试能否写入数据

127.0.0.1:6379> set key test1
OK

修改所有slave指向新的master节点

修改10.0.0.38节点指向新的master节点10.0.0.28
127.0.0.1:6379> SLAVEOF 10.0.0.28 6379
OK
127.0.0.1:6379> set key v10
(error) READONLY You can't write against a read only replica


#查看日志
[root@centos8  ~]#tail -f /var/log/redis/redis.log
2909:M 22 Oct 2020 17:17:30.555 * DB loaded from append only file: 0.000 seconds
2909:M 22 Oct 2020 17:17:30.555 * Ready to accept connections
2909:M 22 Oct 2020 17:17:30.779 * Replica 10.0.0.38:6379 asks for synchronization
2909:M 22 Oct 2020 17:17:30.779 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '34ead6a506afd682dc7e7564c6fd429e8fc7042f', my replication IDs are 'ffb82ddba16ff7b99f6db27622d16c2f54ddd221' and '0000000000000000000000000000000000000000')
2909:M 22 Oct 2020 17:17:30.779 * Starting BGSAVE for SYNC with target: disk
2909:M 22 Oct 2020 17:17:30.783 * Background saving started by pid 2914
2914:C 22 Oct 2020 17:17:30.787 * DB saved on disk
2914:C 22 Oct 2020 17:17:30.788 * RDB: 4 MB of memory used by copy-on-write
2909:M 22 Oct 2020 17:17:30.858 * Background saving terminated with success
2909:M 22 Oct 2020 17:17:30.859 * Synchronization with replica 10.0.0.38:6379 succeeded

在新master上可看到slave

#在新master节点slave上查看状态
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1   #连接主机slave一台
slave0:ip=10.0.0.38,port=6379,state=online,offset=182,lag=0
master_replid:d3c0429e43689f57e7581a003a4a088884644ed4
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:182
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:182

 

 2.哨兵的使用和实现机制

 

redis哨兵(sentinel)

  

 

 

 

 

 

 

sentinel中的三个定时任务

10秒每个sentinelmasterslave执行info

发现slave节点

确认主从关系

2秒每个sentinel通过master节点的channel交换信息(pub/sub

通过sentinel_:hello频道交互

交互对节点的看法和自身信息

1秒每个sentinel对其他的sentinelredis执行ping

实现哨兵sentinel

 

 

 

1.哨兵的准备实现主从复制架构

哨兵的前提是已经实现了一个redis的主从复制的运行环境,从而实现基于哨兵的高可用redis架构

注:master的配置文件中masterslave都必须相同

所有主从节点的redis.conf中关键配置

[root@centos8  ~]#dnf -y install redis
[root@centos8  ~]#vim /etc/redis.conf
bind 0.0.0.0
masterauth  123456
requirepass 123456  
 
 #或者非交互执行
 [root@centos8  ~]#sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e 's/^# masterauth.*/masterauth 123456/' -e 's/^# requirepass .*/requirepass 123456/' /etc/redis.conf
 
 #在所有从节点执行
 [root@centos8  ~]#echo "replicaof 10.0.0.8 6379" >> /etc/redis.conf
  [root@centos8  ~]#sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e 's/^# masterauth.*/masterauth 123456/' -e 's/^# requirepass .*/requirepass 123456/' /etc/redis.conf
 
 #在所有主从节点执行
 [root@centos8  ~]#systemctl   enable --now redis

2. master 服务器状态 10.0.0.8

[root@centos8  ~]#redis-cli -a 123456
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:0
master_replid:b460e6fc3bde3231c60496dededa775373920e62
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

3.配置slave1 10.0.0.18

[root@centos8  ~]#redis-cli
127.0.0.1:6379> replicaof 10.0.0.8 6379
OK Already connected to specified master
127.0.0.1:6379> config set masterauth 123456
OK
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.8
master_port:6379
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:0
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:a43dfe3e4932bd2005da1c8f17a005731b744acc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:0

[root@centos8  ~]#systemctl restart redis

4.配置slave2 10.0.0.28

[root@centos8  ~]#redis-cli
127.0.0.1:6379> replicaof 10.0.0.8 6379
OK Already connected to specified master
127.0.0.1:6379> config set masterauth 123456
OK
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.8
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:280
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:a43dfe3e4932bd2005da1c8f17a005731b744acc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:280
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:267
repl_backlog_histlen:14

5.编辑哨兵的配置文件

sentinel配置

sentinel实际上是一个特殊的redis服务器有些redis指令支持,但很多指令并不=支持,默认监听在26379/tcp端口

哨兵可以不和redis服务器部署在一起,但一般部署在一起,所有redis节点使用相同的以下示例文件

#如果是编译安装,在源码目录有sentinel.conf,复制到安装目录即可,如:/apps/redis/etc/sentinel.conf
[root@centos8 ~]#vim /etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize yes
pidfile "redis-sentinel.pid"
logfile "sentinel_26379.log"
dir "/tmp"  #工作目录
sentinel myid 50547f34ed71fd48c197924969937e738a39975b

sentinel monitor mymaster 10.0.0.8 6379 2 #指定当前mymaster集群中master服务器的地址和端口
#2为法定人数限制(quorum),即有几个sentinel认为master down了就进行故障转移,一般此值是所有sentinel节点(一般总数是>=3的 奇数,:3,5,7)的一半以上的整数值,比如,总数是3,即3/2=1.5,取整为2,masterODOWN客观下线的依据

sentinel auth-pass mymaster 123456 #mymaster集群中master的密码,注意此行要在上面行的下面

sentinel down-after-milliseconds mymaster 30000 #(SDOWN)判断mymaster集群中所有节点的主观下线的时间,单位:毫秒,建议3000
sentinel parallel-syncs mymaster 1 #发生故障转移后,同时向新master同步数据的slave数量,数字越小总同步时间越长,但可以减轻新master的负载压力
sentinel failover-timeout mymaster 180000 #所有slaves指向新的master所需的超时时间,单位:毫秒
sentinel deny-scripts-reconfig yes #禁止修改脚本
logfile /var/log/redis/sentinel.log

6. 三个哨兵服务器的配置都如下

[root@centos8  ~]#grep -vE '^#|^$' /etc/redis-sentinel.conf
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile "/var/log/redis/sentinel.log"
dir /tmp
sentinel monitor mymaster 10.0.0.8 6379 2   #修改此行
sentinel auth-pass mymaster 123456  #增加此行
sentinel down-after-milliseconds mymaster 3000    #修改此行
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
logfile /var/log/redis/sentinel.log

#以下内容自动生成,不需要修改
sentinel myid 50547f34ed71fd48c197924969937e738a39975b  #此行每个哨兵主机自动生成,且必须唯一
.....
# Generated by CONFIG REWRITE
protected-mode no
supervised systemd
sentinel leader-epoch mymaster 0
sentinel known-replica mymaster 10.0.0.28 6379
sentinel known-replica mymaster 10.0.0.18 6379
sentinel current-epoch 0

[root@centos8  ~]#scp /etc/redis-sentinel.conf  10.0.0.18:/etc/
[root@centos8  ~]#scp /etc/redis-sentinel.conf  10.0.0.8:/etc/

7.启动哨兵

三台哨兵服务都要启动

#确保每个哨兵主机myid不同
[root@centos18  ~]#vim /etc/redis-sentinel.conf
sentinel myid b460e6fc3bde3231c60496dededa775373920e63
[root@centos28  ~]#vim /etc/redis-sentinel.conf
sentinel myid b460e6fc3bde3231c60496dededa775373920e64


[root@centos8  ~]#systemctl enable --now redis-sentinel.service
[root@centos18  ~]#systemctl enable --now redis-sentinel.service
[root@centos28  ~]#systemctl enable --now redis-sentinel.service

如果是编译安装,在所有哨兵服务器执行下面操作启动哨兵

#vim /apps/redis/etc/sentinel.conf
bind 0.0.0.0
port 26379
daemonize yes
pidfile "redis-sentinel.pid"
Logfile "sentinel_26379.log"
dir "/apps/redis/data"
sentinel monitor mymaster 10.0.0.8 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 15000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes


#/apps/redis/bin/redis-sentinel /apps/redis/etc/sentinel.conf

8.验证哨兵端口

[root@centos8  ~]#ss -ntl
State  Recv-Q  Send-Q    Local Address:Port                Peer Address:Port              
LISTEN 0       128             0.0.0.0:111                      0.0.0.0:*                
LISTEN 0       32        192.168.122.1:53                       0.0.0.0:*                
LISTEN 0       128             0.0.0.0:22                       0.0.0.0:*                
LISTEN 0       5             127.0.0.1:631                      0.0.0.0:*                
LISTEN 0       128             0.0.0.0:42661                    0.0.0.0:*                
LISTEN 0       128             0.0.0.0:26379                    0.0.0.0:*                
LISTEN 0       128             0.0.0.0:6379                     0.0.0.0:*                
LISTEN 0       128                [::]:111                         [::]:*                
LISTEN 0       128                [::]:60501                       [::]:*                
LISTEN 0       128                [::]:22                          [::]:*                
LISTEN 0       5                 [::1]:631                         [::]:*                
LISTEN 0       128                [::]:26379                       [::]:*  

9.查看哨兵日志

master的哨兵日志

[root@centos8  ~]#tail -f /var/log/redis/sentinel.log
2740:X 23 Oct 2020 20:53:32.364 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2740:X 23 Oct 2020 20:53:32.364 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=2740, just started
2740:X 23 Oct 2020 20:53:32.364 # Configuration loaded
2740:X 23 Oct 2020 20:53:32.364 * supervised by systemd, will signal readiness
2740:X 23 Oct 2020 20:53:32.366 * Running mode=sentinel, port=26379.
2740:X 23 Oct 2020 20:53:32.366 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2740:X 23 Oct 2020 20:53:32.368 # Sentinel ID is d3ed95f03e9cb551499cdb7f421c43ca1338cd35
2740:X 23 Oct 2020 20:53:32.368 # +monitor master mymaster 10.0.0.8 6379 quorum 2
2740:X 23 Oct 2020 20:54:06.112 * +sentinel sentinel b460e6fc3bde3231c60496dededa775373920e63 10.0.0.18 26379 @ mymaster 10.0.0.8 6379
2740:X 23 Oct 2020 20:54:07.917 * +sentinel sentinel b460e6fc3bde3231c60496dededa775373920e64 10.0.0.28 26379 @ mymaster 10.0.0.8 6379

slave 的哨兵日志

[root@centos8  ~]#tail -f /var/log/redis/sentinel.log36538:X 23 Oct 2020 20:54:06.522 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
36538:X 23 Oct 2020 20:54:06.522 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=36538, just started
36538:X 23 Oct 2020 20:54:06.522 # Configuration loaded
36538:X 23 Oct 2020 20:54:06.522 * supervised by systemd, will signal readiness
36538:X 23 Oct 2020 20:54:06.550 * Running mode=sentinel, port=26379.
36538:X 23 Oct 2020 20:54:06.550 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
36538:X 23 Oct 2020 20:54:06.550 # Sentinel ID is b460e6fc3bde3231c60496dededa775373920e63
36538:X 23 Oct 2020 20:54:06.550 # +monitor master mymaster 10.0.0.8 6379 quorum 2

10.当前sentinel状态

sentinel状态中尤其是最后一行,涉及到masterIP是多少,有几个slave,有几个sentinel,必须是符合全部服务器数量

127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.8:6379,slaves=2,sentinels=3   #2slave3sentinel服务器,如果sentinels值不符合检查myid冲突问题

11.停止redis master测试故障转移

[root@centos8  ~]#killall redis-server

查看各节点上哨兵信息

[root@centos8  ~]#redis-cli  -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=odown,address=10.0.0.8:6379,slaves=2,sentinels=3

故障转移时sentinel的信息

[root@centos8  ~]#tail -f /var/log/redis/sentinel.log
5154:X 23 Oct 2020 22:50:36.336 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379
5154:X 23 Oct 2020 22:51:00.575 # +new-epoch 2
5154:X 23 Oct 2020 22:51:01.323 # +config-update-from sentinel 97afb4b25c0138df82e31f9396071a559c69803f 10.0.0.18 26379 @ mymaster 10.0.0.18 6379
5154:X 23 Oct 2020 22:51:01.323 # +switch-master mymaster 10.0.0.18 6379 10.0.0.28 6379
5154:X 23 Oct 2020 22:51:01.324 * +slave slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.28 6379
5154:X 23 Oct 2020 22:51:01.324 * +slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.28 6379
5154:X 23 Oct 2020 22:51:04.375 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.28 6379
5154:X 23 Oct 2020 22:51:11.440 * +convert-to-slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.28 6379

12.故障转移后redis配置文件会被自动修改

故障转移后redis.confreplicaof行的master IP会被修改

[root@centos8  ~]#grep '^replicaof' /etc/redis.conf
replicaof 10.0.0.28 6379

哨兵配置文件的sentinel monitor IP 同样会被修改

[root@centos18  ~]#grep '^[a-z]' /etc/redis-sentinel.conf
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis/sentinel.log"
dir "/tmp"
sentinel myid 97afb4b25c0138df82e31f9396071a559c69803f
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.0.0.28 6379 2   #自动修改此行
sentinel down-after-milliseconds mymaster 3000
sentinel auth-pass mymaster 123456
sentinel config-epoch mymaster 2
protected-mode no
supervised systemd
sentinel leader-epoch mymaster 2
sentinel known-replica mymaster 10.0.0.18 6379
sentinel known-replica mymaster 10.0.0.8 6379
sentinel known-sentinel mymaster 10.0.0.28 26379 9ff3139e05a7639265ed41ac097190349d917f0a
sentinel known-sentinel mymaster 10.0.0.8 26379 c58682825a17f3e8f85d419cf47d5cb7c373faec
sentinel current-epoch 2

[root@centos28  ~]#grep '^[a-z]' /etc/redis-sentinel.conf
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis/sentinel.log"
dir "/tmp"
sentinel myid 9ff3139e05a7639265ed41ac097190349d917f0a
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.0.0.28 6379 2
sentinel down-after-milliseconds mymaster 3000
sentinel auth-pass mymaster 123456
sentinel config-epoch mymaster 2
protected-mode no
supervised systemd
sentinel leader-epoch mymaster 2
sentinel known-replica mymaster 10.0.0.18 6379
sentinel known-replica mymaster 10.0.0.8 6379
sentinel known-sentinel mymaster 10.0.0.8 26379 c58682825a17f3e8f85d419cf47d5cb7c373faec
sentinel known-sentinel mymaster 10.0.0.18 26379 97afb4b25c0138df82e31f9396071a559c69803f
sentinel current-epoch 2

13.当前redis状态

新的master状态

[root@centos28  ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.18,port=6379,state=online,offset=337121,lag=1
master_replid:81e4aae2b4643238b217767db618daebbc8f4c2f
master_replid2:ab2c58ecd006c17d7aedf055b91daf68a84f53f7
master_repl_offset:337386
second_repl_offset:174767
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:99
repl_backlog_histlen:337288

另一个slave指向新的master

[root@centos8  ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.28
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:354626
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:81e4aae2b4643238b217767db618daebbc8f4c2f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:354626
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:177044
repl_backlog_histlen:177583

14.恢复故障的原master重新加入redis集群

#sentinel会自动修改下面行指定新的master
[root@centos8  ~]#grep '^replicaof'  /etc/redis.conf
replicaof 10.0.0.28 6379

在原master上观察状态

[root@centos8  ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.28
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:431611
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:81e4aae2b4643238b217767db618daebbc8f4c2f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:431611
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:423573
repl_backlog_histlen:8039

[root@centos8  ~]#redis-cli  -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.28:6379,slaves=2,sentinels=3

观察新master上状态和日志

[root@centos18  ~]# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.28
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:454451
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:81e4aae2b4643238b217767db618daebbc8f4c2f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:454451
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:177044
repl_backlog_histlen:277408


[root@centos8  ~]#tail -f /var/log/redis/sentinel.log
38773:X 23 Oct 2020 22:51:02.043 # +new-epoch 2
38773:X 23 Oct 2020 22:51:02.043 # +vote-for-leader 9ff3139e05a7639265ed41ac097190349d917f0a 2
38773:X 23 Oct 2020 22:51:02.065 # +odown master mymaster 10.0.0.8 6379 #quorum 2/2
38773:X 23 Oct 2020 22:51:02.065 # Next failover delay: I will not start a failover before Fri Oct 23 22:57:02 2020
38773:X 23 Oct 2020 22:51:03.115 # +config-update-from sentinel 9ff3139e05a7639265ed41ac097190349d917f0a 10.0.0.28 26379 @ mymaster 10.0.0.8 6379
38773:X 23 Oct 2020 22:51:03.115 # +switch-master mymaster 10.0.0.8 6379 10.0.0.28 6379
38773:X 23 Oct 2020 22:51:03.115 * +slave slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.28 6379
38773:X 23 Oct 2020 22:51:06.173 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.28 6379
38773:X 23 Oct 2020 22:51:14.136 * +slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.28 6379
38773:X 23 Oct 2020 23:11:57.644 # -sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.28 6379

15.sentinel 运维

手动让主节点下线

sentinel failover <masterName>

示例:手动故障转移

[root@centos8 ~]#vim /etc/redis.conf
replica-priority 10 #指定优先级,值越小sentinel会优先将之选为新的master,默为值为100
[root@centos8 ~]#redis-cli   -p 26379
127.0.0.1:26379> sentinel failover mymaster
OK

 

4. Redis cluster集群创建和使用

基于redis 5 的redis cluster部署

官方文档:

redis cluster相关命令:

查看 --cluster选项帮助

[root@centos8  ~]#redis-cli --cluster help
Cluster Manager Commands:
  create         host1:port1 ... hostN:portN
                 --cluster-replicas <arg>
  check          host:port
                 --cluster-search-multiple-owners
  info           host:port
  fix            host:port
                 --cluster-search-multiple-owners
  reshard        host:port
                 --cluster-from <arg>
                 --cluster-to <arg>
                 --cluster-slots <arg>
                 --cluster-yes
                 --cluster-timeout <arg>
                 --cluster-pipeline <arg>
                 --cluster-replace
  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace
  add-node       new_host:new_port existing_host:existing_port
                 --cluster-slave
                 --cluster-master-id <arg>
  del-node       host:port node_id
  call           host:port command arg arg .. arg
  set-timeout    host:port milliseconds
  import         host:port
                 --cluster-from <arg>
                 --cluster-copy
                 --cluster-replace
  help          

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

查看cluster 指令的帮助

[root@centos8  ~]# redis-cli -a 123456 cluster help
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
 1) CLUSTER <subcommand> arg arg ... arg. Subcommands are:
 2) ADDSLOTS <slot> [slot ...] -- Assign slots to current node.
 3) BUMPEPOCH -- Advance the cluster config epoch.
 4) COUNT-failure-reports <node-id> -- Return number of failure reports for <node-id>.
 5) COUNTKEYSINSLOT <slot> - Return the number of keys in <slot>.
 6) DELSLOTS <slot> [slot ...] -- Delete slots information from current node.
 7) FAILOVER [force|takeover] -- Promote current replica node to being a master.
 8) FORGET <node-id> -- Remove a node from the cluster.
 9) GETKEYSINSLOT <slot> <count> -- Return key names stored by current node in a slot.
10) FLUSHSLOTS -- Delete current node own slots information.
11) INFO - Return onformation about the cluster.
12) KEYSLOT <key> -- Return the hash slot for <key>.
13) MEET <ip> <port> [bus-port] -- Connect nodes into a working cluster.
14) MYID -- Return the node id.
15) NODES -- Return cluster configuration seen by node. Output format:
16)     <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot> ... <slot>
17) REPLICATE <node-id> -- Configure current node as replica to <node-id>.
18) RESET [hard|soft] -- Reset current node (default: soft).
19) SET-config-epoch <epoch> - Set config epoch of current node.
20) SETSLOT <slot> (importing|migrating|stable|node <node-id>) -- Set slot state.
21) REPLICAS <node-id> -- Return <node-id> replicas.
22) SLOTS -- Return information about slots range mappings. Each range is made of:
23)     start, end, master and replicas IP addresses, ports and ids

1.创建redis cluster集群的环境准备

1.每个redis节点采用相同的硬件配置、相同的密码、相同的redis版本

2.所有redis 服务器必须没有任何数据

3.先启动为单机redis且没有任何key value

4,主备6台主机,地址如下

10.0.0.18 ,10.0.0.28,10.0.0.38,10.0.0.48,10.0.0.58,10.0.0.68

2.启用redis cluster 配置

所有6台主机都执行以下配置


[root@centos8  ~]#yum  -y install redis

 

每个节点修改redis配置,必须开启cluster 功能的参数

 #手动修改配置文件

[root@redis-node1 ~]vim /etc/redis.conf
bind 0.0.0.0
masterauth 123456   #建议配置,否则后期的master和slave主从复制无法成功,还需再配置
requirepass 123456
cluster-enabled yes #取消此行注释,必须开启集群,开启后redis 进程会有cluster显示
cluster-config-file nodes-6379.conf #取消此行注释,此为集群状态文件,记录主从关系及slot范围信息,由redis cluster 集群自动创建和维护
cluster-require-full-coverage no   #默认值为yes,设为no可以防止一个节点不可用导致整个cluster不可能

#或者执行下面命令,批量修改
[root@centos8  ~]#sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth 123456' -e '/# requirepass/a requirepass 123456' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e '/cluster-require-full-coverage yes/c cluster-require-full-coverage no' /etc/redis.conf

[root@centos8 ~]#systemctl  enable  --now  redis

 

验证当前redis服务状态

 #开启了16379的cluster的端口,实际的端口=redis port + 10000


[root@centos8 ~]#ss -ntl
State Recv-Q  Send-Q    Local Address:Port    Peer Address:Port
LISTEN0       128             0.0.0.0:22           0.0.0.0:*    
LISTEN0       128             0.0.0.0:16379        0.0.0.0:*    
LISTEN0       128             0.0.0.0:6379         0.0.0.0:*    
LISTEN0       128                [::]:22              [::]:*

#注意进程有[cluster]状态
[root@centos8 ~]#ps -ef |grep redis
redis      1754      1  0 01:22 ?        00:00:00 /usr/bin/redi-server 0.0.0.0:6379 [cluster]
root       1765   1345  0 01:23 pts/0    00:00:00 grep --color=auto redis

3.创建集群

# redis-cli --cluster-replicas 1 表示每个master对应一个slave节点

[root@centos8  ~]#redis-cli -a 123456 --cluster  create  10.0.0.18:6379 10.0.0.28:6379 10.0.0.38:6379 10.0.0.48:6379  10.0.0.58:6379 10.0.0.68:6379  --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.48:6379 to 10.0.0.18:6379
Adding replica 10.0.0.58:6379 to 10.0.0.28:6379
Adding replica 10.0.0.68:6379 to 10.0.0.38:6379
M: 08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379   #带M的为master
   slots:[0-5460] (5461 slots) master   #当前master槽位起始位和结束位
M: b6a81babbde1118f490d40ddf7d15f910d0d8e55 10.0.0.28:6379
   slots:[5461-10922] (5462 slots) master
M: ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
S: b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379    #带S的slave
   replicates 08da9eb03abff895db97391593ef3a30df0c71dc
S: 1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379
   replicates b6a81babbde1118f490d40ddf7d15f910d0d8e55
S: f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379
   replicates ce0d7ac851e8ff829246bc185c9f2c9eef0907d1
Can I set the above configuration? (type 'yes' to accept): yes   #输入yes自动创建集群
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: 08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379
   slots:[0-5460] (5461 slots) master  #已经分配的槽位  
   1 additional replica(s)  #分配了一个slave
S: b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379
   slots: (0 slots) slave  #slave没有分配槽位
   replicates 08da9eb03abff895db97391593ef3a30df0c71dc  #对应的master的10.0.0.18的ID  
M: b6a81babbde1118f490d40ddf7d15f910d0d8e55 10.0.0.28:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379
   slots: (0 slots) slave
   replicates ce0d7ac851e8ff829246bc185c9f2c9eef0907d1  #对应的master的10.0.0.38的ID
S: 1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379
   slots: (0 slots) slave
   replicates b6a81babbde1118f490d40ddf7d15f910d0d8e55
   #对应的master的10.0.0.28的ID
[OK] All nodes agree about slots configuration.
>>> Check for open slots...  #检查打开的槽位
>>> Check slots coverage...  #检查插槽覆盖范围
[OK] All 16384 slots covered.  #所有槽位(16384个)分配完成

4.查看主从状态

[root@centos18  ~]#redis-cli -a 123456 -c info replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.48,port=6379,state=online,offset=910,lag=2
master_replid:cd0680ecee6226bf381946e7ab582706cb7a8406
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:910
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:910


[root@centos28  ~]#redis-cli -a 123456 -c info replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.58,port=6379,state=online,offset=868,lag=0
master_replid:ab2f4c2aa0adcd9a9296fe4131613427ded6f8f8
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:868
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:868


[root@centos38  ~]#redis-cli -a 123456 -c info replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.68,port=6379,state=online,offset=952,lag=1
master_replid:80de45b116763228c802923be08639e599c17410
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:966
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:966

[root@centos48 ~]# redis-cli -a 123456 -c info replication
# Replication
role:slave
master_host:10.0.0.18
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:1008
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:cd0680ecee6226bf381946e7ab582706cb7a8406
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1008
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1008

[root@centos58 ~]# redis-cli -a 123456 -c info replication
# Replication
role:slave
master_host:10.0.0.28
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:1078
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:ab2f4c2aa0adcd9a9296fe4131613427ded6f8f8
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1078
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1078

[root@centos68 ~]#redis-cli -a 123456 -c info replication
# Replication
role:slave
master_host:10.0.0.38
master_port:6379
master_link_status:up
master_last_io_seconds_ago:9
master_sync_in_progress:0
slave_repl_offset:1078
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:80de45b116763228c802923be08639e599c17410
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1078
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1078

查看指定master节点的slave节点信息

[root@centos28 ~]#redis-cli  -a 123456 cluster nodes
ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379@16379 master - 0 1603561347104 3 connected 10923-16383
b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379@16379 slave 08da9eb03abff895db97391593ef3a30df0c71dc 0 1603561350164 1 connected
f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379@16379 myself,slave ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 0 1603561348000 6 connected
b6a81babbde1118f490d40ddf7d15f910d0d8e55 10.0.0.28:6379@16379 master - 0 1603561352194 2 connected 5461-10922
1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379@16379 slave b6a81babbde1118f490d40ddf7d15f910d0d8e55 0 1603561349142 2 connected
08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379@16379 master - 0 1603561351180 1 connected 0-5460

#以下命令查看指定master节点的slave节点信息,其中
#08da9eb03abff895db97391593ef3a30df0c71dc 为master节点的ID
[root@centos28  ~]# redis-cli -a 123456 cluster slaves 08da9eb03abff895db97391593ef3a30df0c71dc

1) "b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379@16379 slave 08da9eb03abff895db97391593ef3a30df0c71dc 0 1603532723083 4 connected"

5.验证集群状态

[root@centos18  ~]#redis-cli -a 123456 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6    #节点数6个
cluster_size:3           #集群数3个
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_ping_sent:1175
cluster_stats_messages_pong_sent:1154
cluster_stats_messages_meet_sent:4
cluster_stats_messages_sent:2333
cluster_stats_messages_ping_received:1153
cluster_stats_messages_pong_received:1179
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:2333

#查看任意节点的集群状态
[root@centos18  ~]# redis-cli -a 123456 --cluster info 10.0.0.38:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.38:6379 (ce0d7ac8...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.28:6379 (b6a81bab...) -> 0 keys | 5462 slots | 1 slaves.
10.0.0.18:6379 (08da9eb0...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

6.查看集群node对应关系

[root@centos8  ~]#redis-cli -a 123456 cluster nodes
f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379@16379 slave ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 0 1603532995000 3 connected
08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379@16379 master - 0 1603532995000 1 connected 0-5460
ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379@16379 master - 0 1603532997000 3 connected 10923-16383
b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379@16379 slave 08da9eb03abff895db97391593ef3a30df0c71dc 0 1603532997419 4 connected
b6a81babbde1118f490d40ddf7d15f910d0d8e55 10.0.0.28:6379@16379 myself,master - 0 1603532993000 2 connected 5461-10922
1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379@16379 slave b6a81babbde1118f490d40ddf7d15f910d0d8e55 0 1603532998435 5 connected

[root@centos28  ~]# redis-cli -a 123456 --cluster check 10.0.0.48:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.28:6379 (b6a81bab...) -> 0 keys | 5462 slots | 1 slaves.
10.0.0.38:6379 (ce0d7ac8...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.18:6379 (08da9eb0...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.48:6379)
S: b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379
   slots: (0 slots) slave
   replicates 08da9eb03abff895db97391593ef3a30df0c71dc
M: b6a81babbde1118f490d40ddf7d15f910d0d8e55 10.0.0.28:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379
   slots: (0 slots) slave
   replicates b6a81babbde1118f490d40ddf7d15f910d0d8e55
M: ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379
   slots: (0 slots) slave
   replicates ce0d7ac851e8ff829246bc185c9f2c9eef0907d1
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

7.验证集群写入key

 

 

 

7.1 redis cluster写入key

#经过算法计算,当前key的槽位需要写入指定的node
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.18 set key1 values1
(error) MOVED 9189 10.0.0.28:6379    #槽位不在当前node所以无法写入
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.28 set key1 values1
OK
#指定node可写入
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.28 get key1
"values1"
#对应slave 节点可以KEYS * ,但get key1 失败,可以到master上执行get可以
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.58 keys "*"
1) "key1"
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.58 get key1
(error) MOVED 9189 10.0.0.28:6379

7.2 redis cluster 计算key所有slot

[root@centos28  ~]#redis-cli -h 10.0.0.28 -a 123456  --no-auth-warning  cluster nodes
f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379@16379 slave ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 0 1603536175008 3 connected
08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379@16379 master - 0 1603536173991 1 connected 0-5460
ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379@16379 master - 0 1603536173000 3 connected 10923-16383
b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379@16379 slave 08da9eb03abff895db97391593ef3a30df0c71dc 0 1603536176022 4 connected
b6a81babbde1118f490d40ddf7d15f910d0d8e55 10.0.0.28:6379@16379 myself,master - 0 1603536170000 2 connected 5461-10922
1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379@16379 slave b6a81babbde1118f490d40ddf7d15f910d0d8e55 0 1603536177039 5 connected

#计算得到hello对应的slot
[root@centos18  ~]#redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster  keyslot hello
(integer) 866
[root@centos18  ~]#redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning set hello magku
OK
[root@centos18  ~]#redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster keyslot name
(integer) 5798
[root@centos18  ~]#redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning set name wei
(error) MOVED 5798 10.0.0.28:6379
[root@centos18  ~]#redis-cli -h 10.0.0.28 -a 123456 --no-auth-warning set name wei
OK
[root@centos18  ~]#redis-cli -h 10.0.0.28 -a 123456 --no-auth-warning get name
"wei"

8.python 脚本实现redis cluster集群写入

[root@centos28  ~]#dnf -y install python3
[root@centos28  ~]#pip3 install redis-py-cluster
[root@centos28  ~]#cat redis_cluster_test.py
#!/usr/bin/env python3
from rediscluster  import RedisCluster
startup_nodes = [
    {"host":"10.0.0.18","port":6379},
    {"host":"10.0.0.28","port":6379},
    {"host":"10.0.0.38","port":6379},
    {"host":"10.0.0.48","port":6379},
    {"host":"10.0.0.58","port":6379},
    {"host":"10.0.0.68","port":6379},
    {"host":"10.0.0.78","port":6379},
]

redis_conn= RedisCluster(startup_nodes=startup_nodes,password='123456',decode_responses=True)
for i in range(0,10000):
    redis_conn.set('key'+str(i),'value'+str(i))
    print('key'+str(i)+':',redis_conn.get('key'+str(i)))


[root@centos28  ~]#chmod +x redis_cluster_test.py
[root@centos28  ~]#./redis_cluster_test.py
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.28
10.0.0.28:6379> dbsize
(integer) 3341
10.0.0.28:6379> get key1
"value1"
10.0.0.28:6379> get key2
(error) MOVED 4998 10.0.0.18:6379
10.0.0.28:6379> keys *
3329) "key7832"
3330) "key2325"
3331) "key2880"
10.0.0.8:6379>
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.28 dbsize
(integer) 3340
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.28 --no-auth-warning get key1
"value1"
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.18 --no-auth-warning dbsize
(integer) 3329
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.18 --no-auth-warning  get key5
(error) MOVED 9057 10.0.0.28:6379
[root@centos28  ~]#redis-cli -a 123456 -h 10.0.0.28 --no-auth-warning  get key5
"value5"

9.模拟master故障,对应节点的slave节点自动提升为新master

#模拟节点2,10.0.0.28出故障,需要相应的数秒故障转移时间
[root@centos28  ~]#tail -f /var/log/redis/redis.log
[root@centos28  ~]#redis-cli -a 123456
127.0.0.1:6379> shutdown
not connected> exit
[root@centos28  ~]#ss -ntl
State   Recv-Q   Send-Q        Local Address:Port                     Peer Address:Port                
LISTEN  0        128                 0.0.0.0:22                            0.0.0.0:*                    
LISTEN  0        128                    [::]:22                               [::]:*  
[root@centos18  ~]#redis-cli -a 123456 --cluster info 10.0.0.18:6379
Could not connect to Redis at 10.0.0.28:6379: Connection refused
10.0.0.18:6379 (08da9eb0...) -> 3332 keys | 5461 slots | 1 slaves.
10.0.0.38:6379 (ce0d7ac8...) -> 3329 keys | 5461 slots | 1 slaves.
10.0.0.58:6379 (1d60d150...) -> 3341 keys | 5462 slots | 0 slaves.  #10.0.0.58为新master
[OK] 10002 keys in 3 masters.
0.61 keys per slot on average.


[root@centos18  ~]#redis-cli -a 123456 --cluster check 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 10.0.0.28:6379: Connection refused
10.0.0.18:6379 (08da9eb0...) -> 3332 keys | 5461 slots | 1 slaves.
10.0.0.38:6379 (ce0d7ac8...) -> 3329 keys | 5461 slots | 1 slaves.
10.0.0.58:6379 (1d60d150...) -> 3341 keys | 5462 slots | 0 slaves.
[OK] 10002 keys in 3 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: 08da9eb03abff895db97391593ef3a30df0c71dc 10.0.0.18:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: b10a16e9aa5ff45d5aecfc101f275ad80779a252 10.0.0.48:6379
   slots: (0 slots) slave
   replicates 08da9eb03abff895db97391593ef3a30df0c71dc
M: ce0d7ac851e8ff829246bc185c9f2c9eef0907d1 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: f2cedb37c6a337f4f7edda5919e24c3ddab262ff 10.0.0.68:6379
   slots: (0 slots) slave
   replicates ce0d7ac851e8ff829246bc185c9f2c9eef0907d1
M: 1d60d150030f349ddfec267b5e33fe7b930e3501 10.0.0.58:6379
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


[root@centos18  ~]#redis-cli -a 123456 -h 10.0.0.58
10.0.0.58:6379> info replication
# Replication
role:master
connected_slaves:0
master_replid:b07a9d330cb31da18b5c47d44ec2125b9493e3b6
master_replid2:ab2f4c2aa0adcd9a9296fe4131613427ded6f8f8
master_repl_offset:146681
second_repl_offset:146682
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:146681

#恢复10.0.0.28故障节点2
[root@centos28  ~]#systemctl start redis

#查看自动生成的配置文件,可以查看node2自动生成slave节点
[root@centos28  ~]#cat   /var/log/redis/redis.log
2488:M 24 Oct 2020 19:40:56.538 * DB loaded from disk: 0.005 seconds
2488:M 24 Oct 2020 19:40:56.538 * Ready to accept connections
2488:M 24 Oct 2020 19:40:56.541 # Configuration change detected. Reconfiguring myself as a replica of 1d60d150030f349ddfec267b5e33fe7b930e3501
2488:S 24 Oct 2020 19:40:56.541 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
2488:S 24 Oct 2020 19:40:56.541 # Cluster state changed: ok
2488:S 24 Oct 2020 19:40:57.556 * Connecting to MASTER 10.0.0.58:6379
2488:S 24 Oct 2020 19:40:57.557 * MASTER <-> REPLICA sync started
2488:S 24 Oct 2020 19:40:57.558 * Non blocking connect for SYNC fired the event.
2488:S 24 Oct 2020 19:40:57.559 * Master replied to PING, replication can continue...
2488:S 24 Oct 2020 19:40:57.564 * Trying a partial resynchronization (request 3bfc827ac3a3d9070e26081fece2a52978505b4b:1).
2488:S 24 Oct 2020 19:40:57.567 * Full resync from master: b07a9d330cb31da18b5c47d44ec2125b9493e3b6:146681
2488:S 24 Oct 2020 19:40:57.567 * Discarding previously cached master state.
2488:S 24 Oct 2020 19:40:57.627 * MASTER <-> REPLICA sync: receiving 62908 bytes from master
2488:S 24 Oct 2020 19:40:57.628 * MASTER <-> REPLICA sync: Flushing old data
2488:S 24 Oct 2020 19:40:57.632 * MASTER <-> REPLICA sync: Loading DB in memory
2488:S 24 Oct 2020 19:40:57.649 * MASTER <-> REPLICA sync: Finished with success

[root@centos8 ~]# redis-cli -a 123456 -h 10.0.0.58
10.0.0.58:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.28,port=6379,state=online,offset=147199,lag=1
master_replid:b07a9d330cb31da18b5c47d44ec2125b9493e3b6
master_replid2:ab2f4c2aa0adcd9a9296fe4131613427ded6f8f8
master_repl_offset:147199
second_repl_offset:146682
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:147199

posted @ 2020-10-24 20:31  九尾cat  阅读(316)  评论(0编辑  收藏  举报