k8s部署redis cluster集群
1.redis cluster部署说明
- Redis Cluster 的三主三从配置通过分片存储数据、主从复制和自动故障转移，提供了高可用性、数据冗余、负载均衡和弹性扩展，确保在分布式环境下高效处理高并发读写请求并保持系统的稳定性与可靠性。
2.部署环境
| IP | 节点 | 操作系统 | k8s版本 | redis版本 | docker版本 | redis节点 |
| 172.16.4.85 | master1 | centos7.8 | 1.23.17 | 6.0 | 20.10.9 | |
| 172.16.4.86 | node1 | centos7.8 | 1.23.17 | 6.0 | 20.10.9 | 主从 |
| 172.16.4.87 | node2 | centos7.8 | 1.23.17 | 6.0 | 20.10.9 | 主从 |
| 172.16.4.89 | node3 | centos7.8 | 1.23.17 | 6.0 | 20.10.9 | 主从 |
3.部署k8s环境
- 已经安装好k8s环境和colica网络插件，或者参考 https://www.cnblogs.com/Leonardo-li/p/18648449 进行部署
4.部署redis cluster集群
4.1 nfs部署
- centos7安装nfs
yum install -y nfs-utils- 创建nfs共享目录
mkdir /nfs_share/k8s/redis_cluster/pv{1..6}- nfs配置文件编辑
[root@localhost redis_cluster]# cat /etc/exports
/nfs_share/k8s *(rw,sync,no_root_squash,no_subtree_check)
/nfs_share/k8s/redis_cluster/pv1 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/redis_cluster/pv2 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/redis_cluster/pv3 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/redis_cluster/pv4 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/redis_cluster/pv5 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/redis_cluster/pv6 *(rw,sync,no_subtree_check,no_root_squash)- 启动nfs服务
# 启动 NFS 服务
systemctl start nfs-server
# 设置 NFS 服务在系统启动时自动启动
systemctl enable nfs-server- 加载配置文件，并输出
[root@localhost redis_cluster]# exportfs -r
[root@localhost redis_cluster]# exportfs -v
/nfs_share/k8s <world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/redis_cluster/pv1
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/redis_cluster/pv2
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/redis_cluster/pv3
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/redis_cluster/pv4
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/redis_cluster/pv5
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/redis_cluster/pv6
<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)4.2 创建namespace
kubectl create ns redis-clu-94.3 redis pv部署
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redis-cluster-pv1
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # 添加此行，确保 PV 的存储类与 PVC 匹配
nfs:
server: 172.16.4.60
path: "/nfs_share/k8s/redis_cluster/pv1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redis-cluster-pv2
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # 添加此行
nfs:
server: 172.16.4.60
path: "/nfs_share/k8s/redis_cluster/pv2"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redis-cluster-pv3
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # 添加此行
nfs:
server: 172.16.4.60
path: "/nfs_share/k8s/redis_cluster/pv3"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redis-cluster-pv4
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # 添加此行
nfs:
server: 172.16.4.60
path: "/nfs_share/k8s/redis_cluster/pv4"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redis-cluster-pv5
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # 添加此行
nfs:
server: 172.16.4.60
path: "/nfs_share/k8s/redis_cluster/pv5"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-redis-cluster-pv6
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
storageClassName: nfs # 添加此行
nfs:
server: 172.16.4.60
path: "/nfs_share/k8s/redis_cluster/pv6"kubectl apply -f redis-pv.yaml4.4 redis configmap部署
- fix-pod-ip.sh 脚本的作用用于当 redis 集群某 pod 重建后 Pod IP 发生变化，在 /data/nodes.conf 中将新的 Pod IP 替换原 Pod IP，不然集群会出问题。
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-conf
namespace: redis-clu-9
data:
fix-pod-ip.sh: |
#!/bin/sh
CLUSTER_CONFIG="/var/lib/redis/nodes.conf"
if [ -f ${CLUSTER_CONFIG} ]; then
if [ -z "${POD_IP}" ]; then
echo "Unable to determine Pod IP address!"
exit 1
fi
echo "Updating pod IP to ${POD_IP} in ${CLUSTER_CONFIG}"
sed -i -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
count=`grep -c ${POD_IP} ${CLUSTER_CONFIG}`
if [[ $count > 0 ]];then
echo "Successful updated pod Ip to ${POD_IP} in ${CLUSTER_CONFIG}"
fi
fi
exec "$@"
redis.conf: |
# 在这里粘贴你的 redis.conf 文件的内容
# 或者你可以使用 kubectl create configmap --dry-run -o yaml ... 来生成
#开启集群模式
cluster-enabled yes
# 监听ip
bind 0.0.0.0
port 26379
#保护模式
protected-mode no
#redis后台运行
#daemonize yes
#设置客户端连接的超时时间，避免长时间占用连接资源
timeout 300
#设置集群节点之间通信的超时时间
cluster-node-timeout 5000
#指定集群配置文件名称
cluster-config-file /var/lib/redis/nodes.conf
#数据存储目录
dir /var/lib/redis
#设置同时连接客户端的最大数量
maxclients 10000
#指定服务器冗余级别
loglevel notice
# 设置Redis能够使用的最大内存量。这有助于防止Redis因内存耗尽而崩溃
maxmemory 1000mb
#内存淘汰策略
maxmemory-policy volatile-lru
# 数据持久化
appendonly yes
#设置AOF的同步策略,如everysec(每秒同步一次)以平衡性能和数据安全性。
appendfsync everysec
#AOF文件名
appendfilename "appendonly.aof"
#自动重写附加文件条件配置
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# 允许在AOF重写期间进行增量fsync操作它可以帮助减少延迟并减轻fsync对应用程序性能的影响
aof-rewrite-incremental-fsync yes
# 优化内存使用
hash-max-ziplist-entries 512
hash-max-ziplist-value 64kb
# 频率
hz 10
masterauth 123456
requirepass 123456kubectl apply -f redis-configmap.yaml4.5 redis statefulset部署
- 如果目标集群的节点名称不同，需要修改
nodeAffinity部分，将其中的node1、node2、node3修改为目标集群节点的实际名称，获取节点：kubectl get node
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node1
- node2
- node3- redis statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
namespace: redis-clu-9
spec:
serviceName: "redis-service"
replicas: 6
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
appCluster: redis-cluster
spec:
terminationGracePeriodSeconds: 20
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis
topologyKey: kubernetes.io/hostname
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node1
- node2
- node3
containers:
- name: redis
image: 172.16.4.17:8090/public/redis:6.0
command: ["/usr/local/etc/redis/fix-pod-ip.sh", "redis-server", "/usr/local/etc/redis/redis.conf"]
resources:
requests:
cpu: "100m"
memory: "100Mi"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- name: redis
containerPort: 26379
protocol: "TCP"
- name: cluster
containerPort: 16379
protocol: "TCP"
volumeMounts:
- name: "redis-conf"
mountPath: "/usr/local/etc/redis"
readOnly: false
- name: "redis-data"
mountPath: "/var/lib/redis"
readOnly: false
volumes:
- name: "redis-conf"
configMap:
name: "redis-conf"
defaultMode: 0755
volumeClaimTemplates:
- metadata:
name: redis-data
namespace: redis-clu-9
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: nfs
volumeMode: Filesystemkubectl apply -f redis.yaml4.6 redis svc部署
- 不分配 clusterIP，直接暴露每个 Pod 的 IP 地址。用于 StatefulSet 或集群内部节点发现，通常用于 Redis 集群中的节点间通信
apiVersion: v1
kind: Service
metadata:
name: redis-service
namespace: redis-clu-9
labels:
app: redis
spec:
ports:
- name: redis-port
port: 26379
- name: redis-cluster-port
port: 16379
clusterIP: None
selector:
app: redis
appCluster: redis-clusterkubectl apply -f headless-service.yaml- 将服务暴露在每个节点的指定端口（如 32702）。适用于外部访问 Kubernetes 内部的服务，允许外部流量通过集群节点的端口访问 Redis 集群。
apiVersion: v1
kind: Service
metadata:
name: redis-cluster-access-service
namespace: redis-clu-9
labels:
app: redis
spec:
ports:
- name: redis-cluster-port
protocol: TCP
port: 26379
targetPort: 26379
nodePort: 32702
selector:
app: redis
appCluster: redis-cluster
type: NodePortkubectl apply -f redis-cluster-access-service.yaml4.7 部署完查看状态
[root@master1 redis-n2]# kubectl get pv | grep redis
nfs-redis-cluster-pv1 10Gi RWX Retain Bound redis-clu-9/redis-data-redis-cluster-1 nfs 103m
nfs-redis-cluster-pv2 10Gi RWX Retain Bound redis-clu-9/redis-data-redis-cluster-3 nfs 103m
nfs-redis-cluster-pv3 10Gi RWX Retain Bound redis-clu-9/redis-data-redis-cluster-0 nfs 103m
nfs-redis-cluster-pv4 10Gi RWX Retain Bound redis-clu-9/redis-data-redis-cluster-2 nfs 103m
nfs-redis-cluster-pv5 10Gi RWX Retain Bound redis-clu-9/redis-data-redis-cluster-4 nfs 103m
nfs-redis-cluster-pv6 10Gi RWX Retain Bound redis-clu-9/redis-data-redis-cluster-5 nfs 103m
[root@master1 redis-n2]# kubectl get pvc -n redis-clu-9
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
redis-data-redis-cluster-0 Bound nfs-redis-cluster-pv3 10Gi RWX nfs 103m
redis-data-redis-cluster-1 Bound nfs-redis-cluster-pv1 10Gi RWX nfs 103m
redis-data-redis-cluster-2 Bound nfs-redis-cluster-pv4 10Gi RWX nfs 102m
redis-data-redis-cluster-3 Bound nfs-redis-cluster-pv2 10Gi RWX nfs 102m
redis-data-redis-cluster-4 Bound nfs-redis-cluster-pv5 10Gi RWX nfs 102m
redis-data-redis-cluster-5 Bound nfs-redis-cluster-pv6 10Gi RWX nfs 102m
[root@master1 redis-n2]# kubectl get statefulset -n redis-clu-9
NAME READY AGE
redis-cluster 6/6 103m
[root@master1 redis-n2]# kubectl get pods -n redis-clu-9 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-cluster-0 1/1 Running 0 103m 10.244.135.11 node3 <none> <none>
redis-cluster-1 1/1 Running 0 103m 10.244.104.20 node2 <none> <none>
redis-cluster-2 1/1 Running 0 103m 10.244.166.148 node1 <none> <none>
redis-cluster-3 1/1 Running 0 103m 10.244.135.12 node3 <none> <none>
redis-cluster-4 1/1 Running 0 103m 10.244.104.21 node2 <none> <none>
redis-cluster-5 1/1 Running 0 103m 10.244.166.149 node1 <none> <none>
[root@master1 redis-n2]# kubectl get cm -n redis-clu-9
NAME DATA AGE
kube-root-ca.crt 1 23h
redis-conf 2 18h
[root@master1 redis-n2]# kubectl get svc -n redis-clu-9
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-cluster-access-service NodePort 10.106.98.102 <none> 26379:32702/TCP 83m
redis-service ClusterIP None <none> 26379/TCP,16379/TCP 83m5.redis cluster集群初始化
5.1 容器内初始化（master节点）
- pod-name 容器名称，kubectl get pods -n redis-clu-9，选择一个即可
- node1-6_IP node节点ip，kubectl get node -o wide ，可以看到
# 手动获取集群node ip
kubectl get pod -n redis-clu-9 -o wide
# 命令获取集群node ip
kubectl get pods -n redis-clu-9 -l app=redis -o jsonpath='{range.items[*]}{.status.podIP}:26379 ' | sed 's/ :26379//g'
# 输出下面类似格式的IP:PORT列表, 复制这个输出下面命令会用到
10.244.135.11:26379 10.244.104.20:26379 10.244.166.148:26379 10.244.135.12:26379 10.244.104.21:26379 10.244.166.149:26379
# 进入容器
kuectl exec -it -n redis-clu-9 pod-name bash
# 以下为进入容器操作，node_IP 列表为上面命令输出
redis-cli --cluster create node1_IP:26379 node2_IP:26379 node3_IP:26379 node4_IP:26379 node5_IP:26379 node6_IP:26379 --cluster-replicas 1 -a 1234565.2 容器外初始化（master节点）
- pod-name 容器名称，kubectl get pods -n redis-clu-9，选择一个即可
# 也可以在容器外一键创建
kubectl -n redis-clu-9 exec -it pod-name -- redis-cli --cluster create $(kubectl get pods -n redis-clu-9 -l app=redis -o jsonpath='{range.items[*]}{.status.podIP}:26379 ' | sed 's/ :26379//g') --cluster-replicas -a 123456 1- 实际命令
kubectl -n redis-clu-9 exec -it redis-cluster-0 -- redis-cli --cluster create $(kubectl get pods -n redis-clu-9 -l app=redis -o jsonpath='{range.items[*]}{.status.podIP}:26379 ' | sed 's/ :26379//g') --cluster-replicas 1 -a 1234565.3 初始化结果
- 可以看到主从信息、槽的分布情况
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.244.104.21:26379 to 10.244.135.11:26379
Adding replica 10.244.166.149:26379 to 10.244.104.20:26379
Adding replica 10.244.135.12:26379 to 10.244.166.148:26379
M: 83871015d630c2d4cfcd5e4a548c84825dcb9a67 10.244.135.11:26379
slots:[0-5460] (5461 slots) master
M: f5ae65b46ddc9f37e2fdc4e0e06bcde4085964ec 10.244.104.20:26379
slots:[5461-10922] (5462 slots) master
M: c3a58c5fb0691f1a9ff270051e080f339e18a01e 10.244.166.148:26379
slots:[10923-16383] (5461 slots) master
S: 6dac3649bd15986f5704420434e63d452ee89e60 10.244.135.12:26379
replicates c3a58c5fb0691f1a9ff270051e080f339e18a01e
S: bda9732d1be2c328b05cb7001392435a40ba55ff 10.244.104.21:26379
replicates 83871015d630c2d4cfcd5e4a548c84825dcb9a67
S: 9f262f137f3639f24744ac12ac591cf6697c4bc1 10.244.166.149:26379
replicates f5ae65b46ddc9f37e2fdc4e0e06bcde4085964ec
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 10.244.135.11:26379)
M: 83871015d630c2d4cfcd5e4a548c84825dcb9a67 10.244.135.11:26379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: bda9732d1be2c328b05cb7001392435a40ba55ff 10.244.104.21:26379
slots: (0 slots) slave
replicates 83871015d630c2d4cfcd5e4a548c84825dcb9a67
M: f5ae65b46ddc9f37e2fdc4e0e06bcde4085964ec 10.244.104.20:26379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 9f262f137f3639f24744ac12ac591cf6697c4bc1 10.244.166.149:26379
slots: (0 slots) slave
replicates f5ae65b46ddc9f37e2fdc4e0e06bcde4085964ec
S: 6dac3649bd15986f5704420434e63d452ee89e60 10.244.135.12:26379
slots: (0 slots) slave
replicates c3a58c5fb0691f1a9ff270051e080f339e18a01e
M: c3a58c5fb0691f1a9ff270051e080f339e18a01e 10.244.166.148:26379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.5.4 集群状态验证
#查看pods
[root@master1 redis-n2]# kubectl get pods -n redis-clu-9
NAME READY STATUS RESTARTS AGE
redis-cluster-0 1/1 Running 0 4h17m
redis-cluster-1 1/1 Running 0 4h17m
redis-cluster-2 1/1 Running 0 4h17m
redis-cluster-3 1/1 Running 0 4h17m
redis-cluster-4 1/1 Running 0 4h17m
redis-cluster-5 1/1 Running 0 4h17m
#进入redis pod中
[root@master1 redis-n2]# kubectl -n redis-clu-9 exec -it redis-cluster-0 -- bash
#登陆redis，查看集群信息和集群节点
root@redis-cluster-0:/data# redis-cli -p 26379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:26379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:3083
cluster_stats_messages_pong_sent:3089
cluster_stats_messages_sent:6172
cluster_stats_messages_ping_received:3084
cluster_stats_messages_pong_received:3083
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:6172
127.0.0.1:26379> cluster node
(error) ERR Unknown subcommand or wrong number of arguments for 'node'. Try CLUSTER HELP.
127.0.0.1:26379> cluster nodes
bda9732d1be2c328b05cb7001392435a40ba55ff 10.244.104.21:26379@36379 slave 83871015d630c2d4cfcd5e4a548c84825dcb9a67 0 1739428953542 1 connected
f5ae65b46ddc9f37e2fdc4e0e06bcde4085964ec 10.244.104.20:26379@36379 master - 0 1739428952739 2 connected 5461-10922
83871015d630c2d4cfcd5e4a548c84825dcb9a67 10.244.135.11:26379@36379 myself,master - 0 1739428952000 1 connected 0-5460
9f262f137f3639f24744ac12ac591cf6697c4bc1 10.244.166.149:26379@36379 slave f5ae65b46ddc9f37e2fdc4e0e06bcde4085964ec 0 1739428953742 2 connected
6dac3649bd15986f5704420434e63d452ee89e60 10.244.135.12:26379@36379 slave c3a58c5fb0691f1a9ff270051e080f339e18a01e 0 1739428953000 3 connected
c3a58c5fb0691f1a9ff270051e080f339e18a01e 10.244.166.148:26379@36379 master - 0 1739428952000 3 connected 10923-163836.业务连接redis cluster地址
6.1 集群地址
redis-service.redis-clu-9.svc.cluster.local6.2 集群内部访问
redis-cluster-0.redis-service.redis-clu-9.svc.cluster.local 26379
redis-cluster-1.redis-service.redis-clu-9.svc.cluster.local 26379
redis-cluster-2.redis-service.redis-clu-9.svc.cluster.local 263796.3 集群外部访问
172.16.4.86 32702
172.16.4.87 32702
172.16.4.89 327027.参考文档
https://blog.csdn.net/qq_40477248/article/details/143323620
分类:
k8s
【推荐】国内首个AI IDE，深度理解中文开发场景，立即下载体验Trae
【推荐】编程新体验，更懂你的AI，立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包，你的智能百科全书，全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell：AI 加持，快人一步
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布：重大改进与新特性概览！
· AI与.NET技术实操系列（二）：开始使用ML.NET
· .NET10 - 预览版1新功能体验（一）