非关系数据库型--Redis集群和哨兵

redis集群和高可用

redis主从复制

主从复制特点

​ 一个master可以有多个slave

​ 一个slave只能有一个master

​ 数据流向是单向的，master到slave

主从复制实现

建议redis master和slave开启持久化，设置相同连接密码，slave提升为master时会清空之前所有数据。slave启动时会清空数据并将master数据导入，断开同步时不会删除已同步数据。因redis数据存储在内存中，如不开启持久化在服务重启时会丢失数据。

命令行实现主从同步

replicaof ip port password[建议设置，提高安全性]
在slave服务器上将master指向主服务器(早期版本命令slaveof)
127.0.0.1:6379> replicaof 10.0.0.18 6379
从节点info
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.18
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:14
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:e06b4eafe10a3c480c1852041460213fffd2f5c2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:14
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:14
主节点info
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.28,port=6379,state=online,offset=84,lag=0
slave1:ip=10.0.0.38,port=6379,state=online,offset=84,lag=0
master_failover_state:no-failover
master_replid:e06b4eafe10a3c480c1852041460213fffd2f5c2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:84
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:84

删除主从
127.0.0.1:6379> replicaof no one
OK
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:0
master_replid:913e849889521bfcb33952ddf39b858ac099d050
master_replid2:7a454969dc929554766d288372a506bbf7f631cb
master_repl_offset:994
second_repl_offset:995
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:994

主从服务器日志

主服务器日志
[root@localhost ~]# tail -f /apps/redis/log/redis.log 
2056:M 03 Oct 2022 00:30:13.445 * Background saving terminated with success
2056:M 03 Oct 2022 00:30:13.445 * Synchronization with replica 10.0.0.28:6379 succeeded
2056:M 03 Oct 2022 00:30:13.782 * Replica 10.0.0.38:6379 asks for synchronization
2056:M 03 Oct 2022 00:30:13.783 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'e06b4eafe10a3c480c1852041460213fffd2f5c2', my replication IDs are '7a454969dc929554766d288372a506bbf7f631cb' and '0000000000000000000000000000000000000000')
2056:M 03 Oct 2022 00:30:13.783 * Starting BGSAVE for SYNC with target: disk
2056:M 03 Oct 2022 00:30:13.783 * Background saving started by pid 2065
2065:C 03 Oct 2022 00:30:13.785 * DB saved on disk
2065:C 03 Oct 2022 00:30:13.786 * RDB: 0 MB of memory used by copy-on-write
2056:M 03 Oct 2022 00:30:13.847 * Background saving terminated with success
2056:M 03 Oct 2022 00:30:13.847 * Synchronization with replica 10.0.0.38:6379 succeeded

从服务器日志
[root@localhost ~]# tail -f /var/log/redis/redis.log 
33475:S 03 Oct 2022 00:26:24.635 * MASTER <-> REPLICA sync started
33475:S 03 Oct 2022 00:26:24.636 * Non blocking connect for SYNC fired the event.
33475:S 03 Oct 2022 00:26:24.636 * Master replied to PING, replication can continue...
33475:S 03 Oct 2022 00:26:24.637 * Trying a partial resynchronization (request ef7919589db4eb59db33ee578e5a3ece33008ebc:1).
33475:S 03 Oct 2022 00:26:24.638 * Full resync from master: e06b4eafe10a3c480c1852041460213fffd2f5c2:0
33475:S 03 Oct 2022 00:26:24.638 * Discarding previously cached master state.
33475:S 03 Oct 2022 00:26:24.718 * MASTER <-> REPLICA sync: receiving 528 bytes from master
33475:S 03 Oct 2022 00:26:24.718 * MASTER <-> REPLICA sync: Flushing old data
33475:S 03 Oct 2022 00:26:24.718 * MASTER <-> REPLICA sync: Loading DB in memory
33475:S 03 Oct 2022 00:26:24.718 * MASTER <-> REPLICA sync: Finished with success

在从服务器上修改配置，也可

replicaof 10.0.0.8 6379 #指定master的IP和端口号
masterauth 123456     #如果密码需要设置

主从同步实现过程和优化

实现过程

1）从服务器连接主服务器，发送PSYNC命令
2）主服务器接收到PSYNC命令后，开始执行BGSAVE命令生成RDB快照文件并使用缓冲区记录此后执行的所有写命令
3）主服务器BGSAVE执行完后，向所有从服务器发送RDB快照文件，并在发送期间继续记录被执行的写命令
4）从服务器收到快照文件后丢弃所有旧数据，载入收到的快照至内存
5）主服务器快照发送完毕后,开始向从服务器发送缓冲区中的写命令
6）从服务器完成对快照的载入，开始接收命令请求，并执行来自主服务器缓冲区的写命令
7）后期同步会先发送自己slave_repl_offset位置，只同步新增加的数据，不再全量同步

优化

避免全量复制

​ 第一次全量复制无可避免，建议尽量避免主节点重启，或者利用哨兵和集群技术实现故障转移

​ 设置合适的复制缓冲区

避免复制风暴

​ 建议级联拓扑，避免主节点重启时多从节点复制

​ 建议将主节点不要放在同一台服务器，机器宕机后会触发大量全量复制

配置优化

repl-diskless-sync no # 是否使用无盘同步RDB文件，默认为no，no为不使用无盘，需要将RDB文件保存到磁盘后再发送给slave，yes为支持无盘，支持无盘就是RDB文件不需要保存至本地磁盘，而且直接通过socket文件发送给slave
repl-diskless-sync-delay 5 #diskless时复制的服务器等待的延迟时间
repl-ping-slave-period 10 #slave端向server端发送ping的时间间隔，默认为10秒
repl-timeout 60 #设置主从ping连接超时时间,超过此值无法连接,master_link_status显示为down,并记录错误日志
repl-disable-tcp-nodelay no #是否启用TCP_NODELAY，如设置成yes，则redis会合并小的TCP包从而节省带宽， 但会增加同步延迟（40ms），造成master与slave数据不一致，假如设置成no，则redis master会立即发送同步数据，没有延迟，yes关注性能，no关注redis服务中的数据一致性
repl-backlog-size 1mb #master的写入数据缓冲区，用于记录自上一次同步后到下一次同步过程中间的写入命令，计算公式：repl-backlog-size = 允许从节点最大中断时长 * 主实例offset每秒写入量，比如master每秒最大写入64mb，最大允许60秒，那么就要设置为64mb*60秒=3840MB(3.8G),建议此值是设置的足够大
repl-backlog-ttl 3600 #如果一段时间后没有slave连接到master，则backlog size的内存将会被释放。如果值为0则 表示永远不释放这部份内存。
slave-priority 100 #slave端的优先级设置，值是一个整数，数字越小表示优先级越高。当master故障时将会按照优先级来选择slave端进行恢复，如果值设置为0，则表示该slave永远不会被选择。
min-replicas-to-write 1 #设置一个master的可用slave不能少于多少个，否则master无法执行写
min-slaves-max-lag 20 #设置至少有上面数量的slave延迟时间都大于多少秒时，master不接收写操作(拒绝写入)

redis 哨兵(Sentinel)

哨兵(Sentinel) 是一个分布式系统，可以在一个架构中运行多个哨兵(sentinel) 进程，这些进程使用流言协议(gossip protocols)来接收关于Master主服务器是否下线的信息，并使用投票协议(Agreement Protocols)来决定是否执行自动故障迁移,以及选择哪个Slave作为新的Master

Sentinel 进程是用于监控redis集群中Master主服务器工作的状态，在Master主服务器发生故障的时候，可以实现Master和Slave服务器的切换，保证系统的高可用

主观宕机：哨兵进程向其他哨兵，master，slave定时发送消息，来确认对方是否存活，在指定配置时间内为得到响应，则暂时任务对方离线。

主观宕机：哨兵群中多数对master多出SDOWN的判断，并通过SENTINEL is-master-down-by-addr 命令互相交流之后，得出的Master Server下线判断

Redis Sentinel中的Sentinel节点个数应该为大于等于3且最好为奇数

客户端初始化时连接的是Sentinel节点集合，不再是具体的Redis节点，但Sentinel只是配置中心不是代理。

Redis Sentinel 节点与普通redis 没有区别,要实现读写分离依赖于客户端程序

sentinel中的三个定时任务

​ 每10秒每个sentinel对master和slave执行info

​ 1.发现slave节点

​ 确认主从关系

​ 2.每2秒每个sentinel通过master节点的channel交换信息(pub/sub)

​ 通过sentinel__:hello频道交互

3. 交互对节点的“看法”和自身信息

​ 每1秒每个sentinel对其他sentinel和redis执行ping

实现哨兵

1.准备主从
master节点配置
修改配置文件，修改连接密码 认证密码，开启RDB AOF持久化
bind 0.0.0.0
logfile "/apps/redis/log/redis.log"
save 30 1
save 300 100
save 6000 10000
dir /data/rdb
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
masterauth "123456"
requirepass "123456"
appendonly yes
appendfilename "appendonly.aof"

slave节点修改配置
bind 0.0.0.0
replicaof 10.0.0.18 6379
masterauth "123456"
requirepass "123456"

2.修改sentinel配置文件
[root@localhost ~]# grep -Ev "^#|^$" /apps/redis/etc/sentinel.conf 
port 26379
daemonize no
pidfile "/var/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log" #以redis启动服务无法创建log文件
dir "/tmp"
sentinel monitor mymaster 10.0.0.18 6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel deny-scripts-reconfig yes
sentinel resolve-hostnames no
sentinel announce-hostnames no
protected-mode no
supervised systemd
user default on nopass ~* &* +@all
sentinel myid e8f0ceb57fd27f24f4ebe54119769ffc414342b1
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel current-epoch 0
sentinel known-replica mymaster 10.0.0.28 6379
sentinel known-replica mymaster 10.0.0.38 6379
sentinel known-sentinel mymaster 10.0.0.28 26379 2a54d2aad937e9727f836e4ad767a2a4bf334f26
sentinel known-sentinel mymaster 10.0.0.38 26379 b0ef251d57c0b875530740f364ccf5a901f6b427

3.启动redis sentinel
[root@localhost ~]# systemctl start redis redis-sentinel.service 
[root@localhost ~]# redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.18:6379,slaves=2,sentinels=3

编译安装准备service文件
[root@localhost ~]# cat /lib/systemd/system/redis-sentinel.service
[Unit]
Description=Redis Sentinel
After=network.target
[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
[Install]
WantedBy=multi-user.target

模拟故障

停止master服务
[root@localhost ~]# systemctl stop  redis
[root@localhost ~]# tail -f /apps/redis/log/redis.log
1519:X 03 Oct 2022 02:56:48.484 # +try-failover master mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:48.486 # +vote-for-leader e8f0ceb57fd27f24f4ebe54119769ffc414342b1 1
1519:X 03 Oct 2022 02:56:48.488 # b0ef251d57c0b875530740f364ccf5a901f6b427 voted for e8f0ceb57fd27f24f4ebe54119769ffc414342b1 1
1519:X 03 Oct 2022 02:56:48.488 # 2a54d2aad937e9727f836e4ad767a2a4bf334f26 voted for e8f0ceb57fd27f24f4ebe54119769ffc414342b1 1
1519:X 03 Oct 2022 02:56:48.553 # +elected-leader master mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:48.553 # +failover-state-select-slave master mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:48.615 # +selected-slave slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:48.615 * +failover-state-send-slaveof-noone slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:48.674 * +failover-state-wait-promotion slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:49.685 # +promoted-slave slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:49.685 # +failover-state-reconf-slaves master mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:49.731 * +slave-reconf-sent slave 10.0.0.38:6379 10.0.0.38 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:50.615 # -odown master mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:50.687 * +slave-reconf-inprog slave 10.0.0.38:6379 10.0.0.38 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:50.687 * +slave-reconf-done slave 10.0.0.38:6379 10.0.0.38 6379 @ mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:50.763 # +failover-end master mymaster 10.0.0.18 6379
1519:X 03 Oct 2022 02:56:50.763 # +switch-master mymaster 10.0.0.18 6379 10.0.0.28 6379
1519:X 03 Oct 2022 02:56:50.765 * +slave slave 10.0.0.38:6379 10.0.0.38 6379 @ mymaster 10.0.0.28 6379
1519:X 03 Oct 2022 02:56:50.766 * +slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.28 6379

查看10.0.0.28
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.38,port=6379,state=online,offset=275559,lag=1
master_replid:ea4f351a95e5d9c595b942ff5886b5d7869ef4d7
master_replid2:21698ad2088e9db30abfcddb24c6c97c1a627eb3
master_repl_offset:275692
second_repl_offset:259997
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:275692

10.0.0.38
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.28
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:302474
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:ea4f351a95e5d9c595b942ff5886b5d7869ef4d7
master_replid2:21698ad2088e9db30abfcddb24c6c97c1a627eb3
master_repl_offset:302474
second_repl_offset:259997
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:302474

查看10.0.0.38配置文件
[root@localhost ~]# grep -Ev "^#|^$" /etc/redis.conf
replicaof 10.0.0.28 6379
[root@localhost ~]# grep -Ev "^#|^$" /etc/redis-sentinel.conf
sentinel monitor mymaster 10.0.0.28 6379 2

启动10.0.0.18 redis服务
[root@localhost ~]# systemctl start  redis
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.28
master_port:6379

10.0.0.28
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.38,port=6379,state=online,offset=361437,lag=0
slave1:ip=10.0.0.18,port=6379,state=online,offset=361437,lag=1


sentinel failover <masterName> #手动下线节点
或者修改配置文件
replica-priority 10 #指定优先级,值越小sentinel会优先将之选为新的master,默为值为100

Redis Cluster

Redis Cluster 采用哈希槽 (hash slot)的方式来分配16384个slot，每个节点负责其中一部分槽位。槽位的信息存储于每个节点中。

当 Redis Cluster 的客户端来连接集群时，它也会得到一份集群的槽位配置信息并将其缓存在客户端本地。这 样当客户端要查找某个 key 时，可以直接定位到目标节点。同时因为槽位的信息可能会存在客户端与服务器不 一致的情况，还需要纠正机制来实现槽位信息的校验调整。

Redis Cluster特点如下：

1. 所有Redis节点使用(PING机制)互联

2. 集群中某个节点的是否失效，是由整个集群中超过半数的节点监测都失效，才能算真正的失效

3. 客户端不需要proxy即可直接连接redis，应用程序中需要配置有全部的redis服务器IP

4. redis cluster把所有的redis node 平均映射到 0-16383个槽位(slot)上，读写需要到指定的redis node上进行操作，因此有多少个redis node相当于redis 并发扩展了多少倍，每个redis node 承担16384/N个槽位

5. Redis cluster预先分配16384个(slot)槽位，当需要在redis集群中写入一个key -value的时候，会使用CRC16(key) mod 16384之后的值，决定将key写入值哪一个槽位从而决定写入哪一个Redis节点上，从而有效解决单机瓶颈。

原生命令手动部署

配置相关

bind 0.0.0.0
masterauth 123456   #建议配置，否则后期的master和slave主从复制无法成功，还需再配置
requirepass 123456
cluster-enabled yes #取消此行注释,必须开启集群，开启后redis 进程会有cluster标识
cluster-config-file nodes-6379.conf #取消此行注释,此为集群状态文件,记录主从关系及slot范围信息,由redis cluster 集群自动创建和维护
cluster-require-full-coverage no   #默认值为yes,设为no可以防止一个节点不可用导致整个cluster不可能

修改配置文件,在所有服务器
[root@localhost ~]# sed -ri.bak -e "s/^bind.*/bind 0.0.0.0/" -e  "s/# masterauth.*/masterauth "123456"/" -e "s/# requirepass.*/requirepass "123456"/" -e "s/# cluster-enabled.*/cluster-enabled yes/"  -e "s/# cluster-config-file.*/cluster-config-file  nodes-6379.conf/" -e "s/# cluster-require-full-coverage.*/cluster-require-full-coverage no/" /etc/redis.conf

执行meet 操作实现相互通信
[root@localhost ~]# for i in {2..6}; do echo redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster meet "10.0.0."$i"8" "6379";done;
redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster meet 10.0.0.28 6379
redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster meet 10.0.0.38 6379
redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster meet 10.0.0.48 6379
redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster meet 10.0.0.58 6379
redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster meet 10.0.0.68 6379

[root@localhost ~]# redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning cluster nodes
40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379@16379 master - 0 1664743394027 2 connected
200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379@16379 master - 0 1664743393000 1 connected
9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379@16379 master - 0 1664743392022 2 connected
635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379@16379 master - 0 1664743391000 1 connected
91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379@16379 master - 0 1664743393025 3 connected
c898bfbe8333ff3d68b5c90bae93678c8a75884a 10.0.0.18:6379@16379 myself,master - 0 1664743391000 0 connected

[root@localhost ~]# redis-cli -a 123456 cluster info
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:0
cluster_stats_messages_ping_sent:172
cluster_stats_messages_pong_sent:198
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:375
cluster_stats_messages_ping_received:198
cluster_stats_messages_pong_received:177
cluster_stats_messages_received:375

为各节点指派槽位
[root@localhost ~]# cat addslot.sh 
#!/bin/bash
host=$1
start=$2
end=$3
port=6379
pass=123456

for slot in `seq ${start} ${end}`
do
	echo $slot
	redis-cli -h ${host} -p ${port} -a ${pass} --no-auth-warning cluster addslots ${slot}
done
指定主从
[root@localhost ~]# redis-cli -h slaveIP -a 123456 --no-auth-warning cluster replicate masterid

[root@localhost ~]# redis-cli -h 10.0.0.18 -a 123456 cluster slots
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "10.0.0.38"
      2) (integer) 6379
      3) "40eb503b7d7f871f8dd4b2637717fe0575ef791b"
   4) 1) "10.0.0.68"
      2) (integer) 6379
      3) "9f8412d694b28035428ba326ac5b9b496702b2a1"
2) 1) (integer) 5463
   2) (integer) 10922
   3) 1) "10.0.0.28"
      2) (integer) 6379
      3) "200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3"
   4) 1) "10.0.0.58"
      2) (integer) 6379
      3) "635e6e4885ac93baf4710b46f82571c8ac4e1ff7"
3) 1) (integer) 0
   2) (integer) 5462
   3) 1) "10.0.0.18"
      2) (integer) 6379
      3) "c898bfbe8333ff3d68b5c90bae93678c8a75884a"
   4) 1) "10.0.0.48"
      2) (integer) 6379
      3) "91c665428ab5186563044d796bbb0ba2e6497df4"
      
[root@localhost ~]# redis-cli -h 10.0.0.18 -a 123456 --no-auth-warning info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.48,port=6379,state=online,offset=252,lag=1
master_replid:7d4d91a9aa775268d1365d4ceb2af805d6e391f5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:252
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:252
[root@localhost ~]# redis-cli -h 10.0.0.28 -a 123456 --no-auth-warning info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.58,port=6379,state=online,offset=210,lag=1
master_replid:944b9c6edcdc9ec0b4d35f2970fc5ad1b87c0e64
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:210
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:210
[root@localhost ~]# redis-cli -h 10.0.0.38 -a 123456 --no-auth-warning info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.68,port=6379,state=online,offset=182,lag=1
master_replid:4f41b7eb74c9cd235818d38687429e1680243fe4
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:182
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:182

[root@localhost ~]# redis-cli -h 10.0.0.68 -a 123456 --no-auth-warning cluster nodes
40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379@16379 master - 0 1664745262000 5 connected 10923-16383
91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379@16379 slave c898bfbe8333ff3d68b5c90bae93678c8a75884a 0 1664745263000 6 connected
9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379@16379 myself,slave 40eb503b7d7f871f8dd4b2637717fe0575ef791b 0 1664745260000 2 connected
c898bfbe8333ff3d68b5c90bae93678c8a75884a 10.0.0.18:6379@16379 master - 0 1664745262685 6 connected 0-5462
200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379@16379 master - 0 1664745263688 4 connected 5463-10922
635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379@16379 slave 200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 0 1664745264691 4 connected

自动创建集群

#命令redis-cli的选项 --cluster-replicas 1 表示每个master对应一个slave节点
[root@localhost ~]# redis-cli -a 123456 --cluster create 10.0.0.18:6379   10.0.0.28:6379   10.0.0.38:6379  10.0.0.48:6379 10.0.0.58:6379   10.0.0.68:6379 --cluster-replicas 1

故障模拟

停止10.0.0.28 redis服务
查看10.0.0.58状态
[root@localhost ~]# redis-cli -a 123456 --cluster info 10.0.0.58:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 10.0.0.28:6379: Connection refused
10.0.0.58:6379 (635e6e48...) -> 1 keys | 5460 slots | 0 slaves.
10.0.0.18:6379 (c898bfbe...) -> 0 keys | 5463 slots | 1 slaves.
10.0.0.38:6379 (40eb503b...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average

启动10.0.0.28 redis服务后
[root@localhost ~]# redis-cli -h 10.0.0.68 -a 123456 --no-auth-warning cluster nodes
40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379@16379 master - 0 1664746342000 5 connected 10923-16383
91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379@16379 slave c898bfbe8333ff3d68b5c90bae93678c8a75884a 0 1664746343000 6 connected
9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379@16379 myself,slave 40eb503b7d7f871f8dd4b2637717fe0575ef791b 0 1664746341000 2 connected
c898bfbe8333ff3d68b5c90bae93678c8a75884a 10.0.0.18:6379@16379 master - 0 1664746342801 6 connected 0-5462
200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379@16379 slave 635e6e4885ac93baf4710b46f82571c8ac4e1ff7 0 1664746343804 8 connected
635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379@16379 master - 0 1664746342000 8 connected 5463-10922

集群扩容

10.0.0.78 10.0.0.88
修改新增服务器配置文件
[root@localhost ~]# sed -ri.bak -e "s/^bind.*/bind 0.0.0.0/" -e  "s/# masterauth.*/masterauth "123456"/" -e "s/# requirepass.*/requirepass "123456"/" -e "s/# cluster-enabled.*/cluster-enabled yes/"  -e "s/# cluster-config-file.*/cluster-config-file  nodes-6379.conf/" -e "s/# cluster-require-full-coverage.*/cluster-require-full-coverage no/" /etc/redis.conf
[root@localhost ~]# systemctl enable --now redis
将新主机10.0.0.78添加到集群
[root@localhost ~]# redis-cli -a 123456 --cluster add-node 10.0.0.78:6379 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 10.0.0.78:6379 to cluster 10.0.0.18:6379
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: c898bfbe8333ff3d68b5c90bae93678c8a75884a 10.0.0.18:6379
   slots:[0-5462] (5463 slots) master
   1 additional replica(s)
M: 40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379
   slots: (0 slots) slave
   replicates 635e6e4885ac93baf4710b46f82571c8ac4e1ff7
S: 9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379
   slots: (0 slots) slave
   replicates 40eb503b7d7f871f8dd4b2637717fe0575ef791b
M: 635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379
   slots:[5463-10922] (5460 slots) master
   1 additional replica(s)
S: 91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379
   slots: (0 slots) slave
   replicates c898bfbe8333ff3d68b5c90bae93678c8a75884a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.0.0.78:6379 to make it join the cluster.
[OK] New node added correctly.

[root@localhost ~]# redis-cli -a 123456 --cluster info 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.18:6379 (c898bfbe...) -> 0 keys | 5463 slots | 1 slaves.
10.0.0.38:6379 (40eb503b...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.78:6379 (7dc0d6bb...) -> 0 keys | 0 slots | 0 slaves.
10.0.0.58:6379 (635e6e48...) -> 1 keys | 5460 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
重新分配槽位
注意: 重新分配槽位需要清空数据,所以需要先备份数据,扩展后再恢复数据
[root@localhost ~]# redis-cli -a 123456 --cluster reshard 10.0.0.18:6379
[root@localhost ~]# redis-cli -a 123456 --cluster info 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.18:6379 (c898bfbe...) -> 0 keys | 4097 slots | 1 slaves.
10.0.0.38:6379 (40eb503b...) -> 0 keys | 4096 slots | 1 slaves.
10.0.0.78:6379 (7dc0d6bb...) -> 0 keys | 4096 slots | 0 slaves.
10.0.0.58:6379 (635e6e48...) -> 1 keys | 4095 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
添加slave
[root@localhost ~]# redis-cli -a 123456 --cluster add-node 10.0.0.88:6379 10.0.0.18:6379 --cluster-slave --cluster-master-id 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8
[root@localhost ~]# redis-cli -a 123456 --cluster check 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.18:6379 (c898bfbe...) -> 0 keys | 4097 slots | 1 slaves.
10.0.0.38:6379 (40eb503b...) -> 0 keys | 4096 slots | 1 slaves.
10.0.0.78:6379 (7dc0d6bb...) -> 0 keys | 4096 slots | 1 slaves.
10.0.0.58:6379 (635e6e48...) -> 1 keys | 4095 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: c898bfbe8333ff3d68b5c90bae93678c8a75884a 10.0.0.18:6379
   slots:[1366-5462] (4097 slots) master
   1 additional replica(s)
M: 40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379
   slots: (0 slots) slave
   replicates 635e6e4885ac93baf4710b46f82571c8ac4e1ff7
M: 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8 10.0.0.78:6379
   slots:[0-1365],[5463-6827],[10923-12287] (4096 slots) master
   1 additional replica(s)
S: 9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379
   slots: (0 slots) slave
   replicates 40eb503b7d7f871f8dd4b2637717fe0575ef791b
M: 635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379
   slots:[6828-10922] (4095 slots) master
   1 additional replica(s)
S: 0dd5e7fe2c1385d7812ff2098fa7a2b40c6cfad0 10.0.0.88:6379
   slots: (0 slots) slave
   replicates 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8
S: 91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379
   slots: (0 slots) slave
   replicates c898bfbe8333ff3d68b5c90bae93678c8a75884a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

先添加到集群在指定master
redis-cli -a 123456 --cluster add-node 10.0.0.88:6379 10.0.0.18:6379
10.0.0.78:6380> CLUSTER REPLICATE 886338acd50c3015be68a760502b239f4509881c #将其设置slave，命令格式为cluster replicate MASTERID

动态缩容

注意: 被迁移Redis master源服务器必须保证没有数据，否则迁移报错并会被强制中断。
迁移槽位slot，迁移到其他master节点
redis-cli -a 123456 --cluster reshard 10.0.0.18:6379 --cluster-slots 1365 --cluster-from c898bfbe8333ff3d68b5c90bae93678c8a75884a --cluster-to 200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 --cluster-yes
[root@localhost ~]# redis-cli -a 123456 --cluster check 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.18:6379 (c898bfbe...) -> 0 keys | 0 slots | 0 slaves.
10.0.0.38:6379 (40eb503b...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.78:6379 (7dc0d6bb...) -> 0 keys | 5462 slots | 1 slaves.
10.0.0.58:6379 (635e6e48...) -> 1 keys | 5461 slots | 2 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: c898bfbe8333ff3d68b5c90bae93678c8a75884a 10.0.0.18:6379
   slots: (0 slots) master
M: 40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379
   slots:[1366-2730],[12288-16383] (5461 slots) master
   1 additional replica(s)
S: 200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379
   slots: (0 slots) slave
   replicates 635e6e4885ac93baf4710b46f82571c8ac4e1ff7
M: 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8 10.0.0.78:6379
   slots:[0-1365],[2731-4096],[5463-6827],[10923-12287] (5462 slots) master
   1 additional replica(s)
S: 9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379
   slots: (0 slots) slave
   replicates 40eb503b7d7f871f8dd4b2637717fe0575ef791b
M: 635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379
   slots:[4097-5462],[6828-10922] (5461 slots) master
   2 additional replica(s)
S: 0dd5e7fe2c1385d7812ff2098fa7a2b40c6cfad0 10.0.0.88:6379
   slots: (0 slots) slave
   replicates 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8
S: 91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379
   slots: (0 slots) slave
   replicates 635e6e4885ac93baf4710b46f82571c8ac4e1ff7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

删除节点
[root@localhost ~]# redis-cli -a 123456 --cluster del-node 10.0.0.28:6379 c898bfbe8333ff3d68b5c90bae93678c8a75884a

[root@localhost ~]# redis-cli -a 123456 --cluster check 10.0.0.28:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.58:6379 (635e6e48...) -> 1 keys | 5461 slots | 2 slaves.
10.0.0.38:6379 (40eb503b...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.78:6379 (7dc0d6bb...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.28:6379)
S: 200e0b0e3f0a0dfbac6fce0e55b52ec3a5ce7bf3 10.0.0.28:6379
   slots: (0 slots) slave
   replicates 635e6e4885ac93baf4710b46f82571c8ac4e1ff7
S: 91c665428ab5186563044d796bbb0ba2e6497df4 10.0.0.48:6379
   slots: (0 slots) slave
   replicates 635e6e4885ac93baf4710b46f82571c8ac4e1ff7
M: 635e6e4885ac93baf4710b46f82571c8ac4e1ff7 10.0.0.58:6379
   slots:[4097-5462],[6828-10922] (5461 slots) master
   2 additional replica(s)
S: 0dd5e7fe2c1385d7812ff2098fa7a2b40c6cfad0 10.0.0.88:6379
   slots: (0 slots) slave
   replicates 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8
M: 40eb503b7d7f871f8dd4b2637717fe0575ef791b 10.0.0.38:6379
   slots:[1366-2730],[12288-16383] (5461 slots) master
   1 additional replica(s)
S: 9f8412d694b28035428ba326ac5b9b496702b2a1 10.0.0.68:6379
   slots: (0 slots) slave
   replicates 40eb503b7d7f871f8dd4b2637717fe0575ef791b
M: 7dc0d6bb03f1ce094c7e3076ed5bdec96fff10a8 10.0.0.78:6379
   slots:[0-1365],[2731-4096],[5463-6827],[10923-12287] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
删除多余从节点
redis-cli -a 123456 --cluster del-node 10.0.0.28:6379 91c665428ab5186563044d796bbb0ba2e6497df4
删除集群文件
rm -f /var/lib/redis/nodes-6379.conf

数据导入

取消认证密码
[root@localhost ~]#  redis-cli -h 10.0.0.18 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
[root@localhost ~]#  redis-cli --cluster import <集群服务器IP:PORT> --cluster-from <外部Redis node-IP:PORT> --cluster-copy --cluster-replace
#只使用cluster-copy，则要导入集群中的key不能存在
#如果集群中已有同样的key，如果需要替换，可以cluster-copy和cluster-replace联用，这样集群中的key就会被替换为外部数据

简述redis集群的实现原理

Redis Cluster 采用哈希槽 (hash slot)的方式来分配16384个slot，每个节点负责其中一部分槽位。槽位的信息存储于每个节点中。
当 Redis Cluster 的客户端来连接集群时，它也会得到一份集群的槽位配置信息并将其缓存在客户端本地。这样当客户端要查找某个key时，可以直接定位到目标节点。同时因为槽位的信息可能会存在客户端与服务器不一致的情况，还需要纠正机制来实现槽位信息的校验调整。

基于redis5的redis cluster部署

修改配置文件
[root@localhost ~]# sed -ri.bak -e "s/^bind.*/bind 0.0.0.0/" -e  "s/# masterauth.*/masterauth "123456"/" -e "s/# requirepass.*/requirepass "123456"/" -e "s/# cluster-enabled.*/cluster-enabled yes/"  -e "s/# cluster-config-file.*/cluster-config-file  nodes-6379.conf/" -e "s/# cluster-require-full-coverage.*/cluster-require-full-coverage no/" /etc/redis.conf

[root@localhost ~]# redis-cli -a 123456 --cluster create 10.0.0.18:6379 10.0.0.28:6379 10.0.0.38:6379 10.0.0.48:6379 10.0.0.58:6379 10.0.0.68:6379 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.48:6379 to 10.0.0.18:6379
Adding replica 10.0.0.58:6379 to 10.0.0.28:6379
Adding replica 10.0.0.68:6379 to 10.0.0.38:6379
M: 89b882b367ce43c26757db78dfa5359a6f3bfdc4 10.0.0.18:6379
   slots:[0-5460] (5461 slots) master
M: 6c2c48b3f80fc3032a22b6d3dff77aa9100d6829 10.0.0.28:6379
   slots:[5461-10922] (5462 slots) master
M: d6be4e4d3ea644889d28e82a82c2064ec045de42 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
S: 0b1b8a6473068af939b47d1564f03d9277959d0d 10.0.0.48:6379
   replicates 89b882b367ce43c26757db78dfa5359a6f3bfdc4
S: b9fafc2253d483e5141ee70a63518a88c28387da 10.0.0.58:6379
   replicates 6c2c48b3f80fc3032a22b6d3dff77aa9100d6829
S: ca4f37704a241645e9854618232fc9571b6e7e07 10.0.0.68:6379
   replicates d6be4e4d3ea644889d28e82a82c2064ec045de42
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: 89b882b367ce43c26757db78dfa5359a6f3bfdc4 10.0.0.18:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: b9fafc2253d483e5141ee70a63518a88c28387da 10.0.0.58:6379
   slots: (0 slots) slave
   replicates 6c2c48b3f80fc3032a22b6d3dff77aa9100d6829
S: ca4f37704a241645e9854618232fc9571b6e7e07 10.0.0.68:6379
   slots: (0 slots) slave
   replicates d6be4e4d3ea644889d28e82a82c2064ec045de42
M: d6be4e4d3ea644889d28e82a82c2064ec045de42 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 0b1b8a6473068af939b47d1564f03d9277959d0d 10.0.0.48:6379
   slots: (0 slots) slave
   replicates 89b882b367ce43c26757db78dfa5359a6f3bfdc4
M: 6c2c48b3f80fc3032a22b6d3dff77aa9100d6829 10.0.0.28:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@localhost ~]# redis-cli -a 123456 --cluster check 10.0.0.18:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.18:6379 (89b882b3...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.38:6379 (d6be4e4d...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.28:6379 (6c2c48b3...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.18:6379)
M: 89b882b367ce43c26757db78dfa5359a6f3bfdc4 10.0.0.18:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: b9fafc2253d483e5141ee70a63518a88c28387da 10.0.0.58:6379
   slots: (0 slots) slave
   replicates 6c2c48b3f80fc3032a22b6d3dff77aa9100d6829
S: ca4f37704a241645e9854618232fc9571b6e7e07 10.0.0.68:6379
   slots: (0 slots) slave
   replicates d6be4e4d3ea644889d28e82a82c2064ec045de42
M: d6be4e4d3ea644889d28e82a82c2064ec045de42 10.0.0.38:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 0b1b8a6473068af939b47d1564f03d9277959d0d 10.0.0.48:6379
   slots: (0 slots) slave
   replicates 89b882b367ce43c26757db78dfa5359a6f3bfdc4
M: 6c2c48b3f80fc3032a22b6d3dff77aa9100d6829 10.0.0.28:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.