kubenetes 搭建redis6.x 集群

kubenetes 搭建redis6.x 集群

1.创建Namespace

kubectl apply -f Namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cluster-redis

2.创建PersistentVolumeClaim #使用nfs做storageclasses，这一步不用操作了

kubectl apply -f PersistentVolumeClaim.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc1
  namespace: cluster-redis
spec:
  accessModes: 
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "storage"
---  
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc2
  namespace: cluster-redis
spec:
  accessModes: 
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "storage"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc3
  namespace: cluster-redis
spec:
  accessModes: 
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "storage"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc4
  namespace: cluster-redis
spec:
  accessModes: 
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "storage"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc5
  namespace: cluster-redis
spec:
  accessModes: 
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "storage"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc6
  namespace: cluster-redis
spec:
  accessModes: 
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "storage"
折叠 

3.创建statefulset

kubectl apply -f ConfigMap.yaml
kubectl apply -f StatefulSet.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-configmap
  namespace: cluster-redis
data:
  update-node.sh: |
    #!/bin/sh
    REDIS_NODES="/data/nodes.conf"
    sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
    exec "$@"
  redis.conf: |+
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 10000
    protected-mode no
    daemonize no
    pidfile /var/run/redis.pid
    port 6379
    tcp-backlog 511
    bind 0.0.0.0
    timeout 3600
    tcp-keepalive 1
    loglevel verbose
    logfile /data/redis.log
    databases 16
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    #requirepass yl123456
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    lua-time-limit 20000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    #rename-command FLUSHALL  ""
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-entries 512
    list-max-ziplist-value 64
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-pod
  namespace: cluster-redis
spec:
  serviceName: redis-cluster-service
  replicas: 6
  selector:
    matchLabels:
      app: redis-pod
  template:
    metadata:
      labels:
        app: redis-pod
    spec:
      containers:
      - name: redis
        image: redis:6.2.1
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/conf/update-node.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster-configmap
          defaultMode: 0755  #此处需注意，报错.configMap.defaultMode: Invalid value: 777: must be a number between 0 and 0777 (octal), 八进制的0755 转换十进制为493
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: "storage" # 注意这个,事先创建好存储卷

4.创建service

kubectl apply -f Service.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-service
  namespace: cluster-redis
spec:
  type: ClusterIP
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-pod

5.初始化 Redis Cluster
注意如下这一步,分开来看:
(1) 命令: kubectl get pods -n cluster-redis -l app=redis-pod -o jsonpath='{range.items[*]}{.status.podIP}:6379 '|awk '{print $1" "$2" "$3" "$4" "$5" "$6" "}'
这个命令是获取到StatefulSet.yaml文件中生成的redis pod ip,最后那个awk是过滤到不需要的信息
比如执行kubectl get pods -n cluster-redis -l app=redis-pod -o jsonpath='{range.items[*]}{.status.podIP}:6379 '得到的结果如下:
10.0.2.122:6379 10.0.0.65:6379 10.0.2.123:6379 10.0.0.66:6379 10.0.0.69:6379 10.0.2.125:6379 :6379
可以看到最后有个多余的:6379是没用的,awk的作用就是去掉这个
(2) 命令:kubectl exec -it redis-pod-0 -n cluster-redis -- redis-cli --cluster create (pod ip) --cluster-replicas 1
这个命令是登录到其中一个pod中创建集群使用的

kubectl exec -it redis-pod-0 -n cluster-redis -- redis-cli --cluster create `kubectl get pods -n cluster-redis -l app=redis-pod -o jsonpath='{range.items[*]}{.status.podIP}:6379 '|awk '{print $1" "$2" "$3" "$4" "$5" "$6" "}'` --cluster-replicas 1

6.验证集群部署

kubectl exec -it redis-pod-0 -n cluster-redis -- redis-cli cluster info

for x in $(seq 0 5); do echo "redis-pod-$x"; kubectl exec redis-pod-$x -n cluster-redis -- redis-cli role; echo; done

7.注意事项
(1)StatefulSet类型的资源不能通过修改'replicas', 'template', and 'updateStrategy'等字段来进行更新,否则就会报错,具体如下:

The StatefulSet "redis-pod" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

(2)生成的pod编号是从0开始的,依次是 redis-pod-0, redis-pod-1, redis-pod-2, redis-pod-3, redis-pod-4, redis-pod-5.
当删除一个pod时则会自动生成一个同名的pod.当然了,新生成的pod ip会发生变化

(3)查看pod对应的DNS域名

# for x in $(seq 0 5); do kubectl exec redis-pod-$x -n cluster-redis -- hostname -f; done
redis-pod-0.redis-cluster-service.cluster-redis.svc.cluster.local
redis-pod-1.redis-cluster-service.cluster-redis.svc.cluster.local
redis-pod-2.redis-cluster-service.cluster-redis.svc.cluster.local
redis-pod-3.redis-cluster-service.cluster-redis.svc.cluster.local
redis-pod-4.redis-cluster-service.cluster-redis.svc.cluster.local
redis-pod-5.redis-cluster-service.cluster-redis.svc.cluster.local

8.使用
项目中代码连接redis集群的话,使用的参数是: ClusterIP:6379

参考资料：https://www.cnblogs.com/sanduzxcvbnm/p/14837419.html