基于StatefulSet部署有状态服务

一、StatefulSet介绍

官网介绍：https://kubernetes.io/zh/docs/concepts/workloads/controllers/statefulset/

• StatefulSet 是用来管理有状态应用的工作负载 API 对象。
• StatefulSet 用来管理某 Pod 集合的部署和扩缩， 并为这些 Pod 提供持久存储和持久标识符。和 Deployment 类似， StatefulSet 管理基于相同容器规约的一组 Pod。但和 Deployment 不同的是， StatefulSet 为它们的每个 Pod 维护了一个有粘性的 ID。这些 Pod 是基于相同的规约来创建的， 但是不能相互替换：无论怎么调度，每个 Pod 都有一个永久不变的 ID。
• 如果希望使用存储卷为工作负载提供持久存储，可以使用 StatefulSet 作为解决方案的一部分。 尽管 StatefulSet 中的单个 Pod 仍可能出现故障， 但持久的 Pod 标识符使得将现有卷与替换已失败 Pod 的新 Pod 相匹配变得更加容易。

使用 StatefulSet
StatefulSet 对于需要满足以下一个或多个需求的应用程序很有价值：

• 稳定的、唯一的网络标识符。
• 稳定的、持久的存储。
• 有序的、优雅的部署和扩缩。
• 有序的、自动的滚动更新。

在上面描述中，“稳定的”意味着 Pod 调度或重调度的整个过程是有持久性的。 如果应用程序不需要任何稳定的标识符或有序的部署、删除或扩缩， 则应该使用由一组无状态的副本控制器提供的工作负载来部署应用程序，比如 Deployment 或者 ReplicaSet 可能更适用于你的无状态应用部署需要。

限制

• 给定 Pod 的存储必须由 PersistentVolume Provisioner 基于所请求的 storage class 来制备，或者由管理员预先制备。
• 删除或者扩缩 StatefulSet 并不会删除它关联的存储卷。 这样做是为了保证数据安全，它通常比自动清除 StatefulSet 所有相关的资源更有价值。
• StatefulSet 当前需要无头服务来负责 Pod 的网络标识。你需要负责创建此服务。
• 当删除一个 StatefulSet 时，该 StatefulSet 不提供任何终止 Pod 的保证。 为了实现 StatefulSet 中的 Pod 可以有序且体面地终止，可以在删除之前将 StatefulSet 缩容到 0。
• 在默认 Pod 管理策略(OrderedReady) 时使用滚动更新， 可能进入需要人工干预才能修复的损坏状态。

使用场景

• Statefulset为了解决有状态服务的集群部署、 集群之间的数据同步问题(MySQL主从等)
• Statefulset所管理的Pod拥有唯一且固定的Pod名称
• Statefulset按照顺序对pod进行启停、 伸缩和回收
• Headless Services(无头服务， 请求的解析直接解析到pod IP)

运行逻辑

二、基于StatefulSet部署有状态服务

2.1、准备nfs 动态存储类

root@easzlab-deploy:~/jiege-k8s/nfs-pv-pvc# cat nfs-deployment.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints","services"]
    verbs: ["get", "list", "watch","create","update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: kube-system 
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner-01
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-provisioner-01
  template:
    metadata:
      labels:
        app: nfs-provisioner-01
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-qingdao.aliyuncs.com/zhangshijie/nfs-subdir-external-provisioner:v4.0.2  
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfs-provisioner-01
            - name: NFS_SERVER
              value: 172.16.88.169
            - name: NFS_PATH
              value: /data
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.16.88.169
            path: /data
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-dynamic-class
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: nfs-provisioner-01
parameters:
  archiveOnDelete: "false"

root@easzlab-deploy:~/jiege-k8s/nfs-pv-pvc# 
root@easzlab-deploy:~/jiege-k8s/nfs-pv-pvc# kubectl apply -f nfs-deployment.yaml 
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
deployment.apps/nfs-provisioner-01 created
storageclass.storage.k8s.io/nfs-dynamic-class created
root@easzlab-deploy:~/jiege-k8s/nfs-pv-pvc# kubectl get pod -n kube-system
NAMESPACE              NAME                                                READY   STATUS    RESTARTS        AGE
kube-system            calico-kube-controllers-5c8bb696bb-5nrbl            1/1     Running   3 (2d18h ago)   3d23h
kube-system            calico-node-68k76                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-bz8pp                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-cfl2g                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-hsrfh                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-j8kgf                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-kdb5j                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-mqmhp                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-x6ctf                                   1/1     Running   2 (2d18h ago)   3d21h
kube-system            calico-node-xh79g                                   1/1     Running   2               3d21h
kube-system            coredns-69548bdd5f-9md6c                            1/1     Running   2 (2d18h ago)   3d23h
kube-system            coredns-69548bdd5f-n6rvg                            1/1     Running   2 (2d18h ago)   3d23h
kube-system            kuboard-7dc6ffdd7c-njhnb                            1/1     Running   2 (2d18h ago)   3d23h
kube-system            metrics-server-686ff776cf-dbgq6                     1/1     Running   4 (2d18h ago)   3d18h
kube-system            nfs-provisioner-01-599b7bfc9d-ldl99                 1/1     Running   0               3s
kubernetes-dashboard   dashboard-metrics-scraper-8c47d4b5d-5cxhb           1/1     Running   2 (2d18h ago)   3d19h
kubernetes-dashboard   kubernetes-dashboard-f778f96c9-s5dk7                1/1     Running   4 (2d18h ago)   3d16h
velero-system          velero-858b9459f9-4jbns                             1/1     Running   4 (2d18h ago)   3d20h
root@easzlab-deploy:~/jiege-k8s/nfs-pv-pvc# kubectl get sc -A
NAME                               PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
ceph-storage-class-k8s (default)   kubernetes.io/rbd    Delete          Immediate           true                   3d19h
nfs-dynamic-class                  nfs-provisioner-01   Delete          Immediate           false                  10s
root@easzlab-deploy:~/jiege-k8s/nfs-pv-pvc# 

2.2、准备有状态部署文件

官网案例:https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/statefulset/

root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# vi StatefulSet-nginx.yaml
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# cat StatefulSet-nginx.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx 
  serviceName: "nginx"
  replicas: 3
  minReadySeconds: 10 
  template:
    metadata:
      labels:
        app: nginx 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: registry.cn-shenzhen.aliyuncs.com/cyh01/nginx-slim:0.8 #这里镜像在国外，只能先想办法上传到自己私仓
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "nfs-dynamic-class"
      resources:
        requests:
          storage: 10Gi
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# 
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# kubectl apply -f StatefulSet-nginx.yaml 
service/nginx created
statefulset.apps/web created
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# 
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# 

root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# kubectl get statefulset -A
NAMESPACE   NAME                            READY   AGE
argocd      argocd-application-controller   1/1     3d18h
argocd      argocd-redis-ha-server          3/3     3d18h
default     web                             3/3     18m
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# 
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# kubectl get pod -owide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE            NOMINATED NODE   READINESS GATES
web-0   1/1     Running   0          78s   10.200.104.238   172.16.88.163   <none>           <none>
web-1   1/1     Running   0          38s   10.200.105.166   172.16.88.164   <none>           <none>
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# kubectl get svc 
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   3d23h
nginx        ClusterIP   None         <none>        80/TCP    87s
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# kubectl get pvc -A
NAMESPACE   NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
default     www-web-0   Bound    pvc-f981dab0-b708-49b9-8b36-a6700cbefa96   10Gi       RWO            nfs-dynamic-class   3m51s
default     www-web-1   Bound    pvc-d3ccbf91-3541-4b33-b7fc-3f54e27f2759   10Gi       RWO            nfs-dynamic-class   3m11s
default     www-web-2   Bound    pvc-e5c7d9f2-2808-4086-bc7c-3aefd6c75012   10Gi       RWO            nfs-dynamic-class   2m31s
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7# kubectl get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS        REASON   AGE
pvc-d3ccbf91-3541-4b33-b7fc-3f54e27f2759   10Gi       RWO            Delete           Bound    default/www-web-1   nfs-dynamic-class            3m15s
pvc-e5c7d9f2-2808-4086-bc7c-3aefd6c75012   10Gi       RWO            Delete           Bound    default/www-web-2   nfs-dynamic-class            2m34s
pvc-f981dab0-b708-49b9-8b36-a6700cbefa96   10Gi       RWO            Delete           Bound    default/www-web-0   nfs-dynamic-class            3m55s
root@easzlab-deploy:~/jiege-k8s/pod-test/case-yaml/case7#

上述例子中：

• 名为 nginx 的 Headless Service 用来控制网络域名。
• 名为 web 的 StatefulSet 有一个 Spec，它表明将在独立的 3 个 Pod 副本中启动 nginx 容器。
• volumeClaimTemplates 将通过 PersistentVolume 制备程序所准备的 PersistentVolumes 来提供稳定的存储。