Redis Cluster 集群部署
一、什么是redis Cluster集群
Redis集群是一种基于数据分片（sharding）的分布式缓存和存储系统。是一个由多个主从节点群组成的分布式服务集群，它具有复制、高可用和分片特性。Redis集群不需要sentinel哨兵也能完成节点移除和故障转移的功能。需要将每个节点设置成集群模式，这种集群模式没有中心节点，可水平扩展，据官方文档称可以线性扩展到上万个节点(官方推荐不超过1000个节点)。redis集群的性能和高可用性均优于之前版本的哨兵模式，且集群配置非常简单。它是Redis为了应对大规模数据存储、高并发访问和高可用性需求而提供的原生集群解决方案。
二、集群架构图
redis 3.0版本之后推出了无中心架构的redis cluster机制，在无中心的redis集群当中，其每个节点保存当前节点数据和整个集群状态,每个节点都和其他所有节点连接
三、redis Cluster部署架构
1、测试环境
3台服务器，每台服务器启动6379和6380两个redis 服务实例
2、生产环境
6台服务器，分别是三组master/slave，适用于生产环境
四、原生命令手动部署Redis Cluster
- 在所有节点安装redis,并配置开启cluster功能
- 各个节点执行meet,实现所有节点的相互通信
- 为各个master 节点指派槽位范围
- 指定各个节点的主从关系
1、环境准备
| 角色 | 主机名 | IP地址 |
|---|---|---|
| 主节点 | m1 | 192.168.112.10 |
| 主节点 | m2 | 192.168.112.20 |
| 主节点 | m3 | 192.168.112.30 |
| 从节点 | s1 | 192.168.112.40 |
| 从节点 | s2 | 192.168.112.50 |
| 从节点 | s3 | 192.168.112.60 |
所有主机都安装了redis
2、为所有节点启用redis集群支持
echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf
echo "masterauth centos" >> /apps/redis/etc/redis.conf
[root@m1 ~]# egrep -v "^#|^$" /apps/redis/etc/redis.conf
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /apps/redis/run/redis_6379.pid
loglevel notice
logfile /apps/redis/log/redis_6379.log
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /apps/redis/data/
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
requirepass centos
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
cluster-enable yes
masterauth centos
#重启redis
systemctl restart redis
3、执行meet操作实现互相通信
在任意一节点上和其它所有节点进行meet通信，以m1为例
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.20 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.30 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.40 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.50 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.60 6379
OK
#可以看到所有节点之间可以相互连接通信，以m1为例
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster nodes
a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 myself,master - 0 1713451004000 1 connected
69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 master - 0 1713451005000 5 connected
4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 master - 0 1713451005696 4 connected
efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713451006719 0 connected
c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 master - 0 1713451003656 2 connected
b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713451004000 3 connected
4、为每个master节点指派槽位范围
由于没有分配槽位，无法创建key
[root@m1 ~]# redis-cli -a centos --no-auth-warning set name misaki
(error) CLUSTERDOWN Hash slot not served
查看当前状态，以m1为例
[root@m1 ~]# redis-cli -a centos --no-auth-warning cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:495
cluster_stats_messages_pong_sent:469
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:969
cluster_stats_messages_ping_received:469
cluster_stats_messages_pong_received:500
cluster_stats_messages_received:969
为每个master 节点指派槽位范围
[root@m1 ~]# vim addslots.sh
#!/bin/bash
HOST=$1
PORT=$2
START=$3
END=$4
PASS=centos
for slot in `seq ${START} ${END}`;do
echo "slot: ${slot}"
redis-cli -h ${HOST} -p ${PORT} -a ${PASS} --no-auth-warning cluster addslots ${slot}
done
* 为三个master分配槽位，共16364/3=5461.33333，平均每个master分配5461个槽位
[root@m1 ~]# bash addslots.sh 192.168.175.10 6379 0 5461
当master1分配完槽位后，可以看到下面信息
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster info
cluster_state:fail
cluster_slots_assigned:5462
cluster_slots_ok:5462
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:1
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:745
cluster_stats_messages_pong_sent:691
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:1441
cluster_stats_messages_ping_received:691
cluster_stats_messages_pong_received:750
cluster_stats_messages_received:1441
为剩下的master节点分配槽位
[root@m1 ~]# bash addslots.sh 192.168.112.20 6379 5462 10922
[root@m1 ~]# bash addslots.sh 192.168.112.30 6379 10923 16383
为所有的三个master分配完槽位后，验证信息
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster nodes
a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 myself,master - 0 1713452132000 1 connected 0-5461
69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 master - 0 1713452137000 5 connected
4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 master - 0 1713452135000 4 connected
efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713452138634 0 connected 10923-16383
c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 master - 0 1713452137000 2 connected 5462-10922
b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713452137612 3 connected
5、指定各个节点的主从关系
通过上面的cluster nodes 查看master的ID信息，执行下面操作，将对应的slave 指定相应的master节点，实现三对主从节点
[root@m1 ~]# redis-cli -h 192.168.112.40 -a centos --no-auth-warning cluster replicate a8f0a6fa7de004d25996eefd652e4bf8fb995386
OK
[root@m1 ~]# redis-cli -h 192.168.112.50 -a centos --no-auth-warning cluster replicate c3ddc4db15222b0cdb2a64974a87b6bd731d8f84
OK
[root@m1 ~]# redis-cli -h 192.168.112.60 -a centos --no-auth-warning cluster replicate efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a
OK
6、查看主节点关系及槽位信息
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster slots
1) 1) (integer) 0
2) (integer) 5461
3) 1) "192.168.112.10"
2) (integer) 6379
3) "a8f0a6fa7de004d25996eefd652e4bf8fb995386"
4) 1) "192.168.112.40"
2) (integer) 6379
3) "b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68"
2) 1) (integer) 10923
2) (integer) 16383
3) 1) "192.168.112.30"
2) (integer) 6379
3) "efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a"
4) 1) "192.168.112.60"
2) (integer) 6379
3) "69f2aee99fc15ddde526dad3cb55b6532c5c6b06"
3) 1) (integer) 5462
2) (integer) 10922
3) 1) "192.168.112.20"
2) (integer) 6379
3) "c3ddc4db15222b0cdb2a64974a87b6bd731d8f84"
4) 1) "192.168.112.50"
2) (integer) 6379
3) "4f5fa3759a45025478aaa062c4290b4b478ab3ce"
7、验证redis Cluster访问
-c 表示以集群方式连接
[root@m1 ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning set name kk
OK
[root@m1 ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning get name
"kk"
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning get name
(error) MOVED 5798 192.168.112.20:6379
[root@m1 ~]# redis-cli -h 192.168.112.20 -a centos --no-auth-warning get name
"kk"
[root@m1 ~]# redis-cli -h 192.168.112.30 -a centos --no-auth-warning get name
(error) MOVED 5798 192.168.112.20:6379
8、测试故障切换
模拟m1主机点宕机
systemctl stop redis
[root@m2 ~]# redis-cli -c -h 192.168.112.20 -a centos --no-auth-warning cluster nodes
c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 myself,master - 0 1713489205000 2 connected 5462-10922
69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 slave efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 0 1713489207000 5 connected
b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713489207436 6 connected 0-5461
4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 slave c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 0 1713489206000 4 connected
a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 master,fail - 1713489156541 1713489154401 1 disconnected
efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713489208451 0 connected 10923-16383
可以看到s1已经由slave转变为master，过程大概在5-10s
五、基于官方脚本搭建redis cluster
创建 redis cluster集群的环境准备：
- 每个redis 节点采用相同的硬件配置、相同的密码、相同的redis版本
- 所有redis服务器必须没有任何数据
- 准备6台主机，开启cluster配置
1、哨兵配置
echo "masterauth centos" >> /apps/redis/etc/redis.conf
echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf
2、开启集群
systemctl restart redis
# redis-cli --cluster-replicas 1 表示每个master对应一个slave节点
任意一台机器上执行就行
[root@localhost ~]# redis-cli -a centos --no-auth-warning --cluster create \
> 192.168.112.10:6379 192.168.112.20:6379 192.168.112.30:6379 \
> 192.168.112.40:6379 192.168.112.50:6379 192.168.112.60:6379 \
> --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.112.50:6379 to 192.168.112.10:6379
Adding replica 192.168.112.60:6379 to 192.168.112.20:6379
Adding replica 192.168.112.40:6379 to 192.168.112.30:6379
M: c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379
slots:[0-5460] (5461 slots) master
M: 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379
slots:[5461-10922] (5462 slots) master
M: e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379
slots:[10923-16383] (5461 slots) master
S: 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379
replicates e621db96b845eb7818eb01a37fccdc7796f8d7de
S: bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379
replicates c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7
S: 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379
replicates 43906d49ffe4be7085b4d37f52c50ac7bb4df751
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 192.168.112.10:6379)
M: c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379
slots: (0 slots) slave
replicates c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7
M: e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379
slots: (0 slots) slave
replicates e621db96b845eb7818eb01a37fccdc7796f8d7de
S: 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379
slots: (0 slots) slave
replicates 43906d49ffe4be7085b4d37f52c50ac7bb4df751
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
3、查看主从状态
[root@localhost ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.112.50,port=6379,state=online,offset=630,lag=0
master_replid:54b3983a20bf4824a13d2699a9775fe3cb85ef0b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:630
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:630
[root@localhost ~]# redis-cli -c -h 192.168.112.50 -a centos --no-auth-warning info replication
# Replication
role:slave
master_host:192.168.112.10
master_port:6379
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:756
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:54b3983a20bf4824a13d2699a9775fe3cb85ef0b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:756
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:756
4、查看集群node对应关系
[root@localhost ~]# redis-cli -a centos --no-auth-warning cluster nodes
c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379@16379 myself,master - 0 1713536198000 1 connected 0-5460
bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379@16379 slave c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 0 1713536200000 5 connected
e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379@16379 master - 0 1713536202879 3 connected 10923-16383
43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379@16379 master - 0 1713536201862 2 connected 5461-10922
5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379@16379 slave e621db96b845eb7818eb01a37fccdc7796f8d7de 0 1713536201000 3 connected
510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379@16379 slave 43906d49ffe4be7085b4d37f52c50ac7bb4df751 0 1713536200845 6 connected
5、验证集群状态
[root@localhost ~]# redis-cli -a centos --no-auth-warning cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6 #节点数
cluster_size:3 #3个集群
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:895
cluster_stats_messages_pong_sent:791
cluster_stats_messages_sent:1686
cluster_stats_messages_ping_received:786
cluster_stats_messages_pong_received:895
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1686
6、模拟master故障
- 对应的slave节点自动提升为新master
- 模拟master1节点出故障,需要相应的数秒故障转移时间
master1关闭redis服务
systemctl stop redis
master2上查看集群信息
[root@localhost ~]# redis-cli -h 192.168.112.20 -a centos --no-auth-warning cluster nodes
e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379@16379 master - 0 1713536909635 3 connected 10923-16383
510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379@16379 slave 43906d49ffe4be7085b4d37f52c50ac7bb4df751 0 1713536907592 2 connected
43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379@16379 myself,master - 0 1713536908000 2 connected 5461-10922
c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379@16379 slave,fail bb81813b153345f0406cfe88c1899e11e2e7a84f 1713536880567 1713536878000 7 disconnected
bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379@16379 master - 0 1713536908615 7 connected 0-5460
5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379@16379 slave e621db96b845eb7818eb01a37fccdc7796f8d7de 0 1713536908000 4 connected
可以看到master1对应的slave3已经提升为master了
