redis-kubernet集群离线安装

环境：centos7.6、kubernet1.17.2

离线包下载：链接: https://pan.baidu.com/s/1B7Rbm2ZVR6P9spbCwhTb0Q 提取码: nhd1

思路：

创建 NFS
创建 PV
创建 PVC
创建 Redis ConfigMap
创建 Redis Headless Service
创建 Redis StatefulSet
初始化 Redis 集群
验证集群是否成功

前提：

#还原镜像并推送到harbor
docker load -i redis.tar  #redis的镜像
docker load -i redis-tribe.tar   #用于配置redis集群的临时镜像
curl -u "admin:harbor123456" -X POST -H "Content-Type: application/json" "172.21.210.20/api/projects" -d "{ \"project_name\": \"redis\", \"public\": 1}"
[root@master1 images]# docker tag redis:5.0.6 172.21.210.20/redis/redis:5.0.6
[root@master1 images]# docker push 172.21.210.20/redis/redis:5.0.6 
[root@master1 images]# docker tag lusiferlee/redis-tribe:latest 172.21.210.20/redis/redis-tribe:latest
[root@master1 images]# docker push 172.21.210.20/redis/redis-tribe:latest

1、创建nfs

我们创建 6 个节点的 Redis 集群，首先需要在 NFS 服务端做好相应的准备工作
创建 Redis 所需共享目录
mkdir -p /data/redis/n1
mkdir -p /data/redis/n2
mkdir -p /data/redis/n3
mkdir -p /data/redis/n4
mkdir -p /data/redis/n5
mkdir -p /data/redis/n6
给目录增加读写权限
chmod a+rw /data/redis
chmod a+rw /data/redis/*
配置 NFS 服务目录
vi /etc/exports

# 底部增加
/data/redis/n1 *(rw,sync,no_subtree_check,no_root_squash)
/data/redis/n2 *(rw,sync,no_subtree_check,no_root_squash)
/data/redis/n3 *(rw,sync,no_subtree_check,no_root_squash)
/data/redis/n4 *(rw,sync,no_subtree_check,no_root_squash)
/data/redis/n5 *(rw,sync,no_subtree_check,no_root_squash)
/data/redis/n6 *(rw,sync,no_subtree_check,no_root_squash)
重启服务，使配置生效
systemctl restart nfs

2、创建 PV

找到 nfs-pv-redis.yml 的配置文件修改
sed -i 's/172.19.163.138/172.21.210.8/g' nfs-pv-redis.yaml  #替换nfs-ip
kubectl apply -f nfs-pv-redis.yaml 

# 输出如下
persistentvolume/nfs-pv-redis-n1 created
persistentvolume/nfs-pv-redis-n2 created
persistentvolume/nfs-pv-redis-n3 created
persistentvolume/nfs-pv-redis-n4 created
persistentvolume/nfs-pv-redis-n5 created
persistentvolume/nfs-pv-redis-n6 created

3、创建 PVC

kubectl apply -f nfs-pvc-redis.yaml  #该文件不用修改

# 输出如下
persistentvolumeclaim/redis-data-redis-app-0 created
persistentvolumeclaim/redis-data-redis-app-1 created
persistentvolumeclaim/redis-data-redis-app-2 created
persistentvolumeclaim/redis-data-redis-app-3 created
persistentvolumeclaim/redis-data-redis-app-4 created
persistentvolumeclaim/redis-data-redis-app-5 created

4、查看 PV 和 PVC

kubectl get pv,pvc

# 输出如下
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                            STORAGECLASS    REASON   AGE
nfs-pv-redis-n1     1Gi        RWX            Retain           Bound       default/redis-data-redis-app-0                            22m
nfs-pv-redis-n2     1Gi        RWX            Retain           Bound       default/redis-data-redis-app-1                            22m
nfs-pv-redis-n3     1Gi        RWX            Retain           Bound       default/redis-data-redis-app-2                            22m
nfs-pv-redis-n4     1Gi        RWX            Retain           Bound       default/redis-data-redis-app-3                            22m
nfs-pv-redis-n5     1Gi        RWX            Retain           Bound       default/redis-data-redis-app-4                            22m
nfs-pv-redis-n6     1Gi        RWX            Retain           Bound       default/redis-data-redis-app-5                            22m
NAME                     STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-data-redis-app-0   Bound    nfs-pv-redis-n1   1Gi        RWX                           22m
redis-data-redis-app-1   Bound    nfs-pv-redis-n2   1Gi        RWX                           22m
redis-data-redis-app-2   Bound    nfs-pv-redis-n3   1Gi        RWX                           22m
redis-data-redis-app-3   Bound    nfs-pv-redis-n4   1Gi        RWX                           22m
redis-data-redis-app-4   Bound    nfs-pv-redis-n5   1Gi        RWX                           22m
redis-data-redis-app-5   Bound    nfs-pv-redis-n6   1Gi        RWX                           22m

5、创建 Redis ConfigMap

我们可以直接将 Redis 的配置文件转化为 Configmap，这是一种更方便的配置读取方式。配置文件 redis.conf 如下：
bind 0.0.0.0
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
创建名为 redis-conf 的 ConfigMap
kubectl create configmap redis-conf --from-file=redis.conf

# 输出如下
configmap/redis-conf created
查看 ConfigMap
kubectl describe cm redis-conf

# 输出如下
Name:         redis-conf
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
redis.conf:
----
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

Events:  <none>

6、创建 Redis Headless Service

kubectl apply -f headless-redis.yaml #该文件不用改
# 输出如下
service/redis-service created

7、创建 Redis StatefulSet

vi statefulset-redis.yaml
----
      - name: redis
        image: 172.21.210.20/redis/redis:5.0.6  #修改为自己的镜像
        command:
          - "redis-server"
-----
kubectl apply -f statefulset-redis.yaml
# 输出如下
statefulset.apps/redis-app created
查看 StatefulSet 创建的 Pods
kubectl get pods -o wide

# 输出如下
# 根据 StatefulSet 的规则，我们生成的 Redis 的 6 个实例的 Hostname
# 会被依次命名为 statefulset名称-序号
NAME                 READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
redis-app-0          1/1     Running   0          31m   10.244.129.88   kubernetes-node1   <none>           <none>
redis-app-1          1/1     Running   0          30m   10.244.81.27    kubernetes-node3   <none>           <none>
redis-app-2          1/1     Running   0          30m   10.244.22.93    kubernetes-node2   <none>           <none>
redis-app-3          1/1     Running   0          30m   10.244.129.89   kubernetes-node1   <none>           <none>
redis-app-4          1/1     Running   0          30m   10.244.22.94    kubernetes-node2   <none>           <none>
redis-app-5          1/1     Running   0          30m   10.244.81.28    kubernetes-node3   <none>           <none>

8、配置集群

以前台交互式方式启动并进入容器
kubectl run -i --tty redis-tribe --image=172.21.210.20/redis/redis-tribe:latest --restart=Never --rm /bin/bash
创建 Master 集群， 命令 dig +short redis-app-0.redis-service.default.svc.cluster.local 用于将 Pod 的域名转化为 IP，这是因为 redis-trib 不支持域名来创建集群
redis-trib.py create \
  `dig +short redis-app-0.redis-service.default.svc.cluster.local`:6379 \
  `dig +short redis-app-1.redis-service.default.svc.cluster.local`:6379 \
  `dig +short redis-app-2.redis-service.default.svc.cluster.local`:6379

# 输出如下
Redis-trib 0.5.1 Copyright (c) HunanTV Platform developers
INFO:root:Instance at 10.244.22.96:6379 checked
INFO:root:Instance at 10.244.81.29:6379 checked
INFO:root:Instance at 10.244.129.91:6379 checked
INFO:root:Add 5462 slots to 10.244.22.96:6379
INFO:root:Add 5461 slots to 10.244.81.29:6379
INFO:root:Add 5461 slots to 10.244.129.91:6379
为每个 Master 添加 Slave
redis-trib.py replicate \
  --master-addr `dig +short redis-app-0.redis-service.default.svc.cluster.local`:6379 \
  --slave-addr `dig +short redis-app-3.redis-service.default.svc.cluster.local`:6379

redis-trib.py replicate \
  --master-addr `dig +short redis-app-1.redis-service.default.svc.cluster.local`:6379 \
  --slave-addr `dig +short redis-app-4.redis-service.default.svc.cluster.local`:6379

redis-trib.py replicate \
  --master-addr `dig +short redis-app-2.redis-service.default.svc.cluster.local`:6379 \
  --slave-addr `dig +short redis-app-5.redis-service.default.svc.cluster.local`:6379

# 输出如下
Redis-trib 0.5.1 Copyright (c) HunanTV Platform developers
INFO:root:Instance at 10.244.129.92:6379 has joined 10.244.129.91:6379; now set replica
INFO:root:Instance at 10.244.129.92:6379 set as replica to 9a2cfdacf9c9dede79ccb96c7ec1c343880445ff

Redis-trib 0.5.1 Copyright (c) HunanTV Platform developers
INFO:root:Instance at 10.244.22.97:6379 has joined 10.244.81.29:6379; now set replica
INFO:root:Instance at 10.244.22.97:6379 set as replica to 8ea9ccc8e9644983185c8fb033c733c9356b73e6

Redis-trib 0.5.1 Copyright (c) HunanTV Platform developers
INFO:root:Instance at 10.244.81.30:6379 has joined 10.244.22.96:6379; now set replica
INFO:root:Instance at 10.244.81.30:6379 set as replica to fa4e37ac4dfc3f0a982ffa248410541d525aafbd

9、集群验证

验证集群是否成功
连接到任意 Reids Pod 检查集群是否安装成功
kubectl exec -it redis-app-0 /bin/bash
/usr/local/bin/redis-cli -c
127.0.0.1:6379> cluster nodes
d584d1198b130a4467128718b5552a7246331707 10.244.22.97:6379@16379 slave 8ea9ccc8e9644983185c8fb033c733c9356b73e6 0 1571066594655 4 connected
0afc8c48fb42ec0f088c2e573f752ecf7c60ec1a 10.244.129.92:6379@16379 slave 9a2cfdacf9c9dede79ccb96c7ec1c343880445ff 0 1571066594000 3 connected
9a2cfdacf9c9dede79ccb96c7ec1c343880445ff 10.244.129.91:6379@16379 myself,master - 0 1571066594000 2 connected 10923-16383
9bd4246dd542b7eed5d1e876e32d9e9b4ed2d3c9 10.244.81.30:6379@16379 slave fa4e37ac4dfc3f0a982ffa248410541d525aafbd 0 1571066593561 5 connected
fa4e37ac4dfc3f0a982ffa248410541d525aafbd 10.244.22.96:6379@16379 master - 0 1571066595161 0 connected 0-5461
8ea9ccc8e9644983185c8fb033c733c9356b73e6 10.244.81.29:6379@16379 master - 0 1571066594150 1 connected 5462-10922
127.0.0.1:6379>  
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_ping_sent:2567
cluster_stats_messages_pong_sent:2651
cluster_stats_messages_meet_sent:6
cluster_stats_messages_sent:5224
cluster_stats_messages_ping_received:2649
cluster_stats_messages_pong_received:2573
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:5224
访问 Redis 集群
在实际生产环境中我们不会将 Redis 端口暴露到公网，故咱们依然采取 kubectl port-forward 的方式暴露服务到主机
kubectl port-forward svc/redis-service 6379:6379 --address 0.0.0.0
直接使用 RedisDesktopManager 等 Redis 客户端工具访问即可