Redis搭建集群架构
使用docker搭建
6.x 版本以后的镜像 docker 支持部署集群模式，由于 Redis 要求集群至少要有三个主节点，因此本次测试搭建了三主三从的 Redis 集群。
不基于 Host 网络模式
配置docker-compose.yml文件
version: "3"
networks:
redis-cluster:
driver: bridge
ipam:
config:
- subnet: 172.26.0.0/24
services:
node1:
image: redis
container_name: redis-cluster-node-1
ports:
- "6371:6379"
volumes:
- "/root/test_redis/cluster_with_network/redis1.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_network/data1:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
networks:
redis-cluster:
ipv4_address: 172.26.0.101
node2:
image: redis
container_name: redis-cluster-node-2
ports:
- "6372:6379"
volumes:
- "/root/test_redis/cluster_with_network/redis2.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_network/data2:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
networks:
redis-cluster:
ipv4_address: 172.26.0.102
node3:
image: redis
container_name: redis-cluster-node-3
ports:
- "6373:6379"
volumes:
- "/root/test_redis/cluster_with_network/redis3.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_network/data3:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
networks:
redis-cluster:
ipv4_address: 172.26.0.103
node4:
image: redis
container_name: redis-cluster-node-4
ports:
- "6374:6379"
volumes:
- "/root/test_redis/cluster_with_network/redis4.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_network/data4:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
networks:
redis-cluster:
ipv4_address: 172.26.0.104
node5:
image: redis
container_name: redis-cluster-node-5
ports:
- "6375:6379"
volumes:
- "/root/test_redis/cluster_with_network/redis5.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_network/data5:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
networks:
redis-cluster:
ipv4_address: 172.26.0.105
node6:
image: redis
container_name: redis-cluster-node-6
ports:
- "6376:6379"
volumes:
- "/root/test_redis/cluster_with_network/redis6.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_network/data6:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
networks:
redis-cluster:
ipv4_address: 172.26.0.106
配置一个子网
配置redis.conf文件
port 6379
protected-mode no
daemonize no
requirepass szz123
masterauth szz123
################################ REDIS CLUSTER ###############################
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
# cluster-announce-ip 127.0.0.1
# cluster-announce-port 6380
cluster-announce-bus-port 16379
所有节点配置都一样
启动服务
docker-compose up -d
进入容器内部启动集群
docker exec -it redis-cluster-node-1 /bin/bash
redis-cli -a szz123 --cluster create 172.26.0.101:6379 172.26.0.102:6379 172.26.0.103:6379 172.26.0.104:6379 172.26.0.105:6379 172.26.0.106:6379 --cluster-replicas 1
查询节点信息
docker exec -it redis-cluster-node-1 /bin/bash
redis-cli -c
cluster nodes
-c 表示进入集群节点(以集群模式运行)，不然报错
(error) MOVED 5798 172.26.0.102:6379
8797811270768e018e2ed116301cad593b1ca9a3 172.26.0.101:6379@16379 myself,master - 0 1672360102000 1 connected 0-5460
72cadf7dd3fe948b0ed3dd404fd1c102d0337096 172.26.0.103:6379@16379 master - 0 1672360104683 3 connected 10923-16383
e1958246b594d74d1fc446e8d7ce25737e143574 172.26.0.102:6379@16379 master - 0 1672360105687 2 connected 5461-10922
f20d78fb8ef57cfa85abff828771d5fb233ffa49 172.26.0.105:6379@16379 slave 8797811270768e018e2ed116301cad593b1ca9a3 0 1672360103680 1 connected
6bd4525a9d3442b9ecd436134142d89e76f1e8a4 172.26.0.106:6379@16379 slave e1958246b594d74d1fc446e8d7ce25737e143574 0 1672360104000 2 connected
07b15a2ff44b9e50c7e8371cbbb9b5937f537565 172.26.0.104:6379@16379 slave 72cadf7dd3fe948b0ed3dd404fd1c102d0337096 0 1672360104000 3 connected
1,2,3为master,4,5,6为slave,对应关系如下
| Master | Slave | Slots |
|---|---|---|
| 1 | 5 | 0-5460 |
| 2 | 6 | 5461-10922 |
| 3 | 4 | 10923-16383 |
Redis Cluster 采用数据分片机制，定义了 16384 个 Slot 槽位，集群中的每个Redis 实例负责维护一部分槽以及槽所映射的键值数据。
验证集群
关闭master节点
docker stop redis-cluster-node-1
再次查看cluster nodes，可以看到，node1的从节点node5已经成为了master
出现网络不转发的问题
我们切到 6371 机器上查询的时候，6371 会自动重定向到存储该 key 的节点上。但是因为我们定义的是内网的集群，所以重定向用的也是内网的 IP 地址，外网无法连接，所以客户端就无法使用了(一直等待中)。
Redis 提供了 cluster-announce-ip 和 cluster-announce-port 让集群支持在转发的网络中工作，但是尝试之后发现无效，以后再来研究。
cluster-announce-ip 127.0.0.1 #告诉redis实例外部映射地址
cluster-announce-port 6371 #告诉redis实例外部映射端口 6371-6376
添加了这两个配置之后，创建集群不成功，一直等待中。
重启容器的注意事项
data/dump.rdb文件中可能也保存着集群的信息，
[ERR] Node 172.26.0.101:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
也需要删除。data/nodes.conf保存着当前集群的信息，我们重启容器服务之前需要将此文件也删除，可以写一个脚本rm_nodes_conf.sh
rm -rf /root/test_redis/cluster_with_network/data1/*
rm -rf /root/test_redis/cluster_with_network/data2/*
rm -rf /root/test_redis/cluster_with_network/data3/*
rm -rf /root/test_redis/cluster_with_network/data4/*
rm -rf /root/test_redis/cluster_with_network/data5/*
rm -rf /root/test_redis/cluster_with_network/data6/*
先停止集群服务，再执行脚本，因为停止服务还会创建dump.rdb文件
docker-compose down
sh rm_nodes_conf.sh
docker-compose up -d
重启容器之后还要进入容器内部重新创建集群
docker exec -it redis-cluster-node-1 /bin/bash
redis-cli -a szz123 --cluster create 172.26.0.101:6379 172.26.0.102:6379 172.26.0.103:6379 172.26.0.104:6379 172.26.0.105:6379 172.26.0.106:6379 --cluster-replicas 1
基于 Host 网络模式
直接使用宿主机网络
配置docker-compose.yml
version: "3"
services:
node1:
image: redis
container_name: redis-cluster-node-1
volumes:
- "/root/test_redis/cluster_with_host/redis1.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_host/data1:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
network_mode: "host"
node2:
image: redis
container_name: redis-cluster-node-2
volumes:
- "/root/test_redis/cluster_with_host/redis2.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_host/data2:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
network_mode: "host"
node3:
image: redis
container_name: redis-cluster-node-3
volumes:
- "/root/test_redis/cluster_with_host/redis3.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_host/data3:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
network_mode: "host"
node4:
image: redis
container_name: redis-cluster-node-4
volumes:
- "/root/test_redis/cluster_with_host/redis4.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_host/data4:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
network_mode: "host"
node5:
image: redis
container_name: redis-cluster-node-5
volumes:
- "/root/test_redis/cluster_with_host/redis5.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_host/data5:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
network_mode: "host"
node6:
image: redis
container_name: redis-cluster-node-6
volumes:
- "/root/test_redis/cluster_with_host/redis6.conf:/etc/redis.conf"
- "/root/test_redis/cluster_with_host/data6:/data"
command: ["redis-server", "/etc/redis.conf"]
restart: always
network_mode: "host"
不需要配置子网了
配置 redis.conf
port 6371
protected-mode no
daemonize no
requirepass szz123
masterauth szz123
################################ REDIS CLUSTER ###############################
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
# cluster-announce-ip 127.0.0.1
# cluster-announce-port 6380
cluster-announce-bus-port 16371
对于六个节点，这里所配置的 Port 是 6371 ~ 6376，bus-port 是 16371 ~ 16376。
对于 cluster-config-file 这个节点配置文件，是自动生成的，不需要我们进行配置，只需要定义文件名即可。
启动服务
docker-compose up -d
启动集群
因为使用的是 Host 网络，因此直接 ip + port 指定就行，--cluster-replicas 设置每个 Master 复制的数量。
docker exec -it redis-cluster-node-1 /bin/bash
redis-cli -a szz123 --cluster create ${your_ip}:6371 ${your_ip}:6372 ${your_ip}:6373 ${your_ip}:6374 ${your_ip}:6375 ${your_ip}:6376 --cluster-replicas 1
注意，下面这种 localhost+port 的写法也能创建集群成功，但客户端转发网络还是会有问题，不认识127.0.0.1
redis-cli -a szz123 --cluster create 127.0.0.1:6371 127.0.0.1:6372 127.0.0.1:6373 127.0.0.1:6374 127.0.0.1:6375 127.0.0.1:6376 --cluster-replicas 1
至此，集群模式搭建成功。
