k8s中部署redis集群(三主三从)

1 环境说明

主机名	系统版本	IP地址	cpu/内存/磁盘	用途	软件版本
k8s_nfs	CentOS7.5	172.16.1.60	2核/2GB/60GB	nfs存储	nfs-utils-1.3.0-0.68
k8s-master1	CentOS7.5	172.16.1.81	2核/2GB/60GB	kubernetes master1节点	k8s v1.20.0
k8s-master2	CentOS7.5	172.16.1.82	2核/2GB/60GB	kubernetes master2节点	k8s v1.20.0
k8s-node1	CentOS7.5	172.16.1.83	4核/8GB/60GB	kubernetes node1节点	k8s v1.20.0
k8s-node2	CentOS7.5	172.16.1.84	4核/8GB/60GB	kubernetes node2节点	k8s v1.20.0

补充: kubernetes集群的控制节点我打了污点不能被pod调度使用。

1 nfs服务部署
节点: k8s_nfs
用途: k8s pod 数据持久化存储
说明: nfs服务的搭建过程不再赘述
验证:
[root@k8s_nfs ~]# showmount -e 172.16.1.60
Export list for 172.16.1.60:
/ifs/kubernetes *
​
2 nfs-subdir-external-provisioner插件部署
节点: kubernetes集群
用途: 为中间件pod提供pvc自动供给
说明: nfs pvc自动供给插件的部署过程不再赘述。修改"deployment.yaml"文件中连接nfs服务的地址和nfs共享目录参数；修改"class.yaml"
文件中"archiveOnDelete"(删除时是否存档)参数为 archiveOnDelete: "true"，删除pod时保留pod数据，默认为false时为不保留数据。
注意: 在部署前需要在k8s各个节点上部署nfs的客户端(yum install nfs-utils -y)，否则无法部署成功。
​
补充:
(1) gitlab项目地址: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
(2) 下载 deploy 目录如下文件
class.yaml、deployment.yaml、rbac.yaml
​
查看:
[root@k8s-master1 nfs-subdir-external-provisioner-master]# ls | xargs -i kubectl apply -f {}
[root@k8s-master1 nfs-subdir-external-provisioner-master]# kubectl get deployment,pod,svc,sc -n default

2 redis集群部署(三主三从)

2.1 yaml配置文件

1 配置文件说明
# 部署所在命名空间: redis-cluster

[root@k8s-master1 ~]# cd /root/redis-cluster/
[root@k8s-master1 redis-cluster]# ls -l
total 8
# redis 配置文件使用 configmap 方式进行挂载
# fix-ip.sh 脚本的作用用于当 redis 集群某 pod 重建后 Pod IP 发生变化，在 /data/nodes.conf 中将新的 Pod IP 替换原 Pod IP。不
# 然集群会出问题。
# redis.conf 文件
-rw-r--r-- 1 root root 2389 Feb 14 20:25 redis-cluster-configmap.yml
# StatefulSet、Service 配置文件
-rw-r--r-- 1 root root 1527 Feb 14 20:25 redis-cluster.yml

2 redis-cluster-configmap.yml
[root@k8s-master1 redis-cluster]# cat redis-cluster-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
  namespace: redis-cluster
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |
    bind 0.0.0.0
    protected-mode yes
    port 6379
    tcp-backlog 2048
    timeout 0
    tcp-keepalive 300
    daemonize no
    supervised no
    pidfile /var/run/redis.pid
    loglevel notice
    logfile /data/redis.log
    databases 16
    always-show-logo yes
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    masterauth liuchang@2022
    replica-serve-stale-data yes
    replica-read-only no
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    replica-priority 100
    requirepass liuchang@2022
    maxclients 32768
    #maxmemory 6g
    maxmemory-policy allkeys-lru
    lazyfree-lazy-eviction no
    lazyfree-lazy-expire no
    lazyfree-lazy-server-del no
    replica-lazy-flush no
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    aof-use-rdb-preamble yes
    lua-time-limit 5000
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 15000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    stream-node-max-bytes 4096
    stream-node-max-entries 100
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit replica 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    dynamic-hz yes
    aof-rewrite-incremental-fsync yes
    rdb-save-incremental-fsync yes

3 redis-cluster.yml
[root@k8s-master1 redis-cluster]# cat redis-cluster.yml
---
apiVersion: v1
kind: Service
metadata:
  namespace: redis-cluster
  name: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: redis-cluster
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:5.0.13
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
      #annotations:
      #  volume.beta.kubernetes.io/storage-class: "redis-nfs-storage"
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

2.2 部署

1 创建命名空间
[root@k8s-master1 redis-cluster]# kubectl create namespace redis-cluster
​
2 应用yml文件
[root@k8s-master1 redis-cluster]# ls | xargs -i kubectl apply -f {}
configmap/redis-cluster created
service/redis-cluster created
statefulset.apps/redis-cluster created

3 查看pod
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster

图示:

4 查看pvc
[root@k8s-master1 redis-cluster]# kubectl get pvc -n redis-cluster

图示:

5 查看pv
[root@k8s-master1 redis-cluster]# kubectl get pv

图示:

6 查看svc
[root@k8s-master1 redis-cluster]# kubectl get svc,ep -n redis-cluster

图示:

7 查看 nfs 共享
[root@k8s_nfs ~]# ls -l /ifs/kubernetes/
​
[root@k8s_nfs ~]# ls /ifs/kubernetes/redis-cluster-data-redis-cluster-0-pvc-ab60036d-dde2-4c5c-9b1d-5bf03816be24/
appendonly.aof  nodes.conf  redis.log
[root@k8s_nfs ~]# ls /ifs/kubernetes/redis-cluster-data-redis-cluster-1-pvc-b4b7b35b-de9a-4c57-ad77-73f3c50e891d/
appendonly.aof  nodes.conf  redis.log
注: 未列出的持久化存储文件相同

图示:

8 初始化 Redis Cluster 集群
(1) 注意: 必须使用 ip 进行初始化 redis 集群，使用域名会报如下错误
Node redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local:6379 replied with error:
ERR Invalid node address specified: redis-cluster-0.redis-cluster.redis-cluster.svc.cluster.local:6379
​
(2) 获取 Redis 集群 6 个节点 Pod 的 ip 地址
1) 方式一
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster -o wide
​
2) 方式二
# kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
/ # nslookup redis-cluster.redis-cluster.svc.cluster.local  # 应用连接 redis 集群时使用下面 pod 的域名

图示:

(3) 创建集群
[root@k8s-master1 redis-cluster]# kubectl exec -it pod/redis-cluster-0 -n redis-cluster -- bash
root@redis-cluster-0:/data# 
redis-cli -a liuchang@2022 --cluster create \
172.27.36.87:6379 \
172.27.36.88:6379 \
172.27.36.89:6379 \
172.27.169.155:6379 \
172.27.169.156:6379 \
172.27.169.157:6379 \
--cluster-replicas 1

图示:

9 验证 Redis Cluster 集群
[root@k8s-master1 redis-cluster]# kubectl exec -it pod/redis-cluster-0 -n redis-cluster -- bash
# redis-cli -h redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local -c -a 'liuchang@2022'
redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster info
redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster nodes
​
注: 由下图可以看到 redis 集群状态正常

图示:

2.3 测试

1 查看 redis 集群 pod
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster -o wide
NAME             READY   STATUS   RESTARTS   AGE   IP               NODE     
redis-cluster-0   1/1     Running   0         3h1m   172.27.36.87     k8s-node1
redis-cluster-1   1/1     Running   0         3h1m   172.27.169.155   k8s-node2
redis-cluster-2   1/1     Running   0         3h     172.27.36.88     k8s-node1
redis-cluster-3   1/1     Running   0         3h     172.27.169.156   k8s-node2
redis-cluster-4   1/1     Running   0         3h     172.27.36.89     k8s-node1
redis-cluster-5   1/1     Running   0         3h     172.27.169.157   k8s-node2
​
2 删除任意一个 pod(删除名称为 redis-cluster-3 的 pod)
[root@k8s-master1 redis-cluster]# kubectl delete pod/redis-cluster-3 -n redis-cluster
pod "redis-cluster-3" deleted
​
3 pod 被重新拉起(还占用原来的pvc 和 pv)
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster -o wide
NAME             READY   STATUS   RESTARTS   AGE   IP               NODE     
redis-cluster-0   1/1     Running   0         3h5m   172.27.36.87     k8s-node1
redis-cluster-1   1/1     Running   0         3h4m   172.27.169.155   k8s-node2
redis-cluster-2   1/1     Running   0         3h4m   172.27.36.88     k8s-node1
redis-cluster-3   1/1     Running   0         3s     172.27.169.158   k8s-node2
redis-cluster-4   1/1     Running   0         3h4m   172.27.36.89     k8s-node1
redis-cluster-5   1/1     Running   0         3h4m   172.27.169.157   k8s-node2
​
注:
可以看到名称为 redis-cluster-3 的 pod 启动时长 AGE 为 3s，IP 由原来的 172.27.169.156 变为 172.27.169.158，
/data/nodes.conf 文件中 "myself" 对应的 ip 被 fix-ip.sh 脚本修改，redis 集群修复时会将该 ip 同步到其它 pod
节的 /data/nodes.conf 文件中，从而保证整个 redis 集群的可用性。
​
4 验证集群状态
[root@k8s-master1 redis-cluster]# kubectl exec -it pod/redis-cluster-4 -n redis-cluster -- bash
# redis-cli -h redis-cluster-5.redis-cluster.redis-cluster.svc.cluster.local -c -a 'liuchang@2022'
redis-cluster-5.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster info
redis-cluster-5.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster nodes
​
注: 由下图可以看到集群状态又恢复了正常

图示:

2.4 补充

1 如果整个 redis 集群的 pod 全部都挂掉了，pod自动拉起后，集群不可用，需要重建集群。
​
2 重建集群的方法1: 删除 redis 集群所有的资源，然后重新创建 redis 集群
(1) 删除 redis 集群中所有的 pod
[root@k8s-master1 redis-cluster]# kubectl delete -f redis-cluster.yml
​
(2) 删除 redis 集群中所有的 pvc(pv)
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-0 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-1 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-2 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-3 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-4 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-5 -n redis-cluster
​
(3) 删除 redis 集群中 pod 对应的 nfs 持久化存储目录
[root@k8s_nfs ~]# rm -rf /ifs/kubernetes/archived-redis-cluster-data-redis-cluster-*/
​
(4) 重新创建 redis 集群
[root@k8s-master1 redis-cluster]# kubectl apply -f redis-cluster.yml
​
3 重建集群的方法2: 在原有 redis 集群的基础上进行修复
(1) 删除 redis 集群中所有的 pod
[root@k8s-master1 redis-cluster]# kubectl delete -f redis-cluster.yml
​
(2) 找到 redis 集群中 pod 对应的 nfs 持久化存储目录后删除 nodes.conf
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-0*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-1*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-2*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-3*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-4*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-5*/nodes.conf*
​
(3) 重新创建 redis 集群
[root@k8s-master1 redis-cluster]# kubectl apply -f redis-cluster.yml