Kubernetes中StatefulSet
root@ubuntu:~/tenant# cat stateful.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi
root@ubuntu:~/tenant# kubectl create -f stateful.yaml statefulset.apps/web created root@ubuntu:~/tenant# kubectl get statefulsets.apps NAME READY AGE web 0/2 7s root@ubuntu:~/tenant# kubectl get pods NAME READY STATUS RESTARTS AGE example-foo-54dc4db9fc-lqz9j 1/1 Running 0 18d job-1-nginx-0 0/1 Completed 0 21d test-job-default-nginx-0 1/1 Running 0 14d test-job-default-nginx-1 1/1 Running 0 14d test-job-default-nginx-2 1/1 Running 0 14d test-job-default-nginx-3 1/1 Running 0 14d test-job-default-nginx-4 1/1 Running 0 14d test-job-default-nginx-5 1/1 Running 0 14d test-job-v2-default-nginx-v2-0 1/1 Running 0 14d web-0 0/1 Pending 0 11s root@ubuntu:~/tenant# kubectl get persistentvolumes No resources found in default namespace. root@ubuntu:~/tenant# kubectl get persistentvolumeclaims NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE www-web-0 Pending 2m13s root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl describe pods web-0 Name: web-0 Namespace: default Priority: 0 Node: <none> Labels: app=nginx controller-revision-hash=web-744989db8c statefulset.kubernetes.io/pod-name=web-0 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: StatefulSet/web Containers: nginx: Image: nginx Port: <none> Host Port: <none> Environment: <none> Mounts: /usr/share/nginx/html from www (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-cfr6q (ro) Conditions: Type Status PodScheduled False Volumes: www: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: www-web-0 ReadOnly: false default-token-cfr6q: Type: Secret (a volume populated by a Secret) SecretName: default-token-cfr6q Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaims Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaims
root@ubuntu:~/tenant# kubectl get sc No resources found in default namespace. root@ubuntu:~/tenant#
创建存储类
root@ubuntu:~/tenant# kubectl get sc No resources found in default namespace.
本地卷
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer
本地卷还不支持动态制备，然而还是需要创建 StorageClass 以延迟卷绑定， 直到完成 Pod 的调度。这是由 WaitForFirstConsumer 卷绑定模式指定的。
延迟卷绑定使得调度器在为 PersistentVolumeClaim 选择一个合适的 PersistentVolume 时能考虑到所有 Pod 的调度限制
cat >nginx-store-class.yaml<<-EOF kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: manual provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer EOF
root@ubuntu:~/tenant# kubectl create -f nginx-store-class.yaml storageclass.storage.k8s.io/manual created root@ubuntu:~/tenant# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE manual kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 8s root@ubuntu:~/tenant#
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaims Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaims root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 115m www-web-0 Pending 155m root@ubuntu:~/tenant# kubectl delete www-web-0 error: the server doesn't have a resource type "www-web-0" root@ubuntu:~/tenant# kubectl delete -f stateful.yaml statefulset.apps "web" deleted root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 116m www-web-0 Pending 155m root@ubuntu:~/tenant# kubectl pvc delete www-web-0 Error: unknown command "pvc" for "kubectl" Run 'kubectl --help' for usage.
删除掉旧的pvc重新创建即可 root@ubuntu:~/tenant# kubectl delete pvc www-web-0 persistentvolumeclaim "www-web-0" deleted root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 116m root@ubuntu:~/tenant# kubectl create -f stateful.yaml statefulset.apps/web created
node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 4 node(s) didn't find available persistent volumes to bind. Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 4 node(s) didn't find available persistent volumes to bind. root@ubuntu:~/tenant# cat stateful.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi storageClassName: manual root@ubuntu:~/tenant# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Bound default/task-pv-claim manual 119m root@ubuntu:~/tenant# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE manual (default) kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 73m root@ubuntu:~/tenant# kubectl get pvs error: the server doesn't have a resource type "pvs" root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 119m www-web-0 Pending manual 3m4s root@ubuntu:~/tenant#
删除所有重建
root@ubuntu:~/tenant# cat stateful.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi storageClassName: manual
root@ubuntu:~/tenant# kubectl delete stateful.yaml nginx-pvc.yaml nginx-pv.yaml nginx-store-class.yaml error: the server doesn't have a resource type "stateful" root@ubuntu:~/tenant# kubectl delete -f stateful.yaml nginx-pvc.yaml nginx-pv.yaml nginx-store-class.yaml error: when paths, URLs, or stdin is provided as input, you may not specify resource arguments as well root@ubuntu:~/tenant# kubectl delete -f stateful.yaml statefulset.apps "web" deleted root@ubuntu:~/tenant# kubectl create -f nginx-store-class.yaml Error from server (AlreadyExists): error when creating "nginx-store-class.yaml": storageclasses.storage.k8s.io "manual" already exists root@ubuntu:~/tenant# kubectl delete -f nginx-store-class.yaml storageclass.storage.k8s.io "manual" deleted root@ubuntu:~/tenant# kubectl delete -f nginx-pvc.yaml persistentvolumeclaim "task-pv-claim" deleted root@ubuntu:~/tenant# kubectl delete -f nginx-pv.yaml persistentvolume "task-pv-volume" deleted root@ubuntu:~/tenant# kubectl delete -f nginx-store-class.yaml Error from server (NotFound): error when deleting "nginx-store-class.yaml": storageclasses.storage.k8s.io "manual" not found root@ubuntu:~/tenant# kubectl create -f nginx-store-class.yaml storageclass.storage.k8s.io/manual created root@ubuntu:~/tenant# kubectl create -f nginx-pv.yaml persistentvolume/task-pv-volume created root@ubuntu:~/tenant# kubectl create -f nginx-pvc.yaml persistentvolumeclaim/task-pv-claim created root@ubuntu:~/tenant# kubectl create -f stateful.yaml statefulset.apps/web created root@ubuntu:~/tenant# kubectl get pods web-0 NAME READY STATUS RESTARTS AGE web-0 0/1 ContainerCreating 0 6s root@ubuntu:~/tenant# kubectl get pods web-0 NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 22s root@ubuntu:~/tenant# kubectl get pods web-0
root@ubuntu:~/tenant# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE manual kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 2m11s root@ubuntu:~/tenant# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Bound default/www-web-0 manual 2m3s root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Pending manual 119s www-web-0 Bound task-pv-volume 10Gi RWO manual 19m root@ubuntu:~/tenant#
创建2个
root@ubuntu:~# cat tenant/stateful.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi storageClassName: manual
root@ubuntu:~/tenant# kubectl create -f stateful.yaml statefulset.apps/web created root@ubuntu:~/tenant# kubectl get pods web-0 NAME READY STATUS RESTARTS AGE web-0 0/1 ContainerCreating 0 5s root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 1/1 Running 0 31s 10.244.129.136 centos7 <none> <none> web-1 0/1 Pending 0 20s <none> <none> <none> <none> root@ubuntu:~/tenant# kubectl describe pods web-1 Name: web-1 Namespace: default Priority: 0 Node: <none> Labels: app=nginx controller-revision-hash=web-744989db8c statefulset.kubernetes.io/pod-name=web-1 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: StatefulSet/web Containers: nginx: Image: nginx Port: <none> Host Port: <none> Environment: <none> Mounts: /usr/share/nginx/html from www (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-cfr6q (ro) Conditions: Type Status PodScheduled False Volumes: www: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: www-web-1 ReadOnly: false default-token-cfr6q: Type: Secret (a volume populated by a Secret) SecretName: default-token-cfr6q Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 4 node(s) didn't find available persistent volumes to bind. Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 4 node(s) didn't find available persistent volumes to bind. root@ubuntu:~/tenant#
再创建一个pv
root@ubuntu:~/tenant# cat nginx-pv.yaml kind: PersistentVolume apiVersion: v1 metadata: name: task-pv-volume2 labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: "/data2"
root@ubuntu:~/tenant# kubectl create -f nginx-pv.yaml persistentvolume/task-pv-volume2 created root@ubuntu:~/tenant# kubectl create -f stateful.yaml statefulset.apps/web created root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 0/1 ContainerCreating 0 3s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 0/1 ContainerCreating 0 5s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 0/1 ContainerCreating 0 7s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 0/1 ContainerCreating 0 9s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 0/1 ContainerCreating 0 11s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 1/1 Running 0 13s 10.244.129.138 centos7 <none> <none> web-1 0/1 ContainerCreating 0 2s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 1/1 Running 0 17s 10.244.129.138 centos7 <none> <none> web-1 0/1 ContainerCreating 0 6s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 1/1 Running 0 18s 10.244.129.138 centos7 <none> <none> web-1 0/1 ContainerCreating 0 7s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 1/1 Running 0 21s 10.244.129.138 centos7 <none> <none> web-1 0/1 ContainerCreating 0 10s <none> centos7 <none> <none> root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> test-job-default-nginx-0 1/1 Running 0 14d 10.244.29.3 bogon <none> <none> test-job-default-nginx-1 1/1 Running 0 14d 10.244.29.9 bogon <none> <none> test-job-default-nginx-2 1/1 Running 0 14d 10.244.29.19 bogon <none> <none> test-job-default-nginx-3 1/1 Running 0 14d 10.244.29.63 bogon <none> <none> test-job-default-nginx-4 1/1 Running 0 14d 10.244.29.1 bogon <none> <none> test-job-default-nginx-5 1/1 Running 0 14d 10.244.29.2 bogon <none> <none> test-job-v2-default-nginx-v2-0 1/1 Running 0 14d 10.244.29.20 bogon <none> <none> web-0 1/1 Running 0 27s 10.244.129.138 centos7 <none> <none> web-1 1/1 Running 0 16s 10.244.129.139 centos7 <none> <none> root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE manual kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 10m root@ubuntu:~/tenant# kubectl get pv --2个 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Bound default/www-web-0 manual 10m task-pv-volume2 10Gi RWO Retain Bound default/www-web-1 manual 3m26s root@ubuntu:~/tenant# kubectl get pvc --多个 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Pending manual 10m www-web-0 Bound task-pv-volume 10Gi RWO manual 28m www-web-1 Bound task-pv-volume2 10Gi RWO manual 7m11s root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl exec -it web-1 -- bash root@web-1:/# ls /usr/share/nginx/html root@web-1:/# touch /usr/share/nginx/html/web-1.txt root@web-1:/# ls bin boot dev docker-entrypoint.d docker-entrypoint.sh etc home lib media mnt opt proc root run sbin srv sys tmp usr var root@web-1:/# ls /usr/share/nginx/html web-1.txt root@web-1:/# exit exit
本地volume
[root@centos7 ~]# ls /data1/ html job.yaml test.txt [root@centos7 ~]# ls /data1/ html job.yaml test.txt [root@centos7 ~]# ls anaconda-ks.cfg calicoctl dpdk-19.11 dpdk-19.11.kunpeng.tgz filter.txt gcc-7.5.0 iptables2.bak iptables3.bak iptables.bak ndn-cxx no_filter.txt [root@centos7 ~]# ls /data2 [root@centos7 ~]# ls /data1 html job.yaml test.txt [root@centos7 ~]# ls /data1 html job.yaml test.txt [root@centos7 ~]# ls /data2 web-1.txt [root@centos7 ~]#
删除一个stateful pod
root@ubuntu:~/c++# kubectl get pods NAME READY STATUS RESTARTS AGE example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d job-1-nginx-0 0/1 Completed 0 22d test-job-default-nginx-0 1/1 Running 0 14d test-job-default-nginx-1 1/1 Running 0 14d test-job-default-nginx-2 1/1 Running 0 14d test-job-default-nginx-3 1/1 Running 0 14d test-job-default-nginx-4 1/1 Running 0 14d test-job-default-nginx-5 1/1 Running 0 14d test-job-v2-default-nginx-v2-0 1/1 Running 0 14d web-0 1/1 Running 0 16h web-1 1/1 Running 0 16h root@ubuntu:~/c++# kubectl delete pod web-0 pod "web-0" deleted root@ubuntu:~/c++# kubectl get pods NAME READY STATUS RESTARTS AGE example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d job-1-nginx-0 0/1 Completed 0 22d test-job-default-nginx-0 1/1 Running 0 14d test-job-default-nginx-1 1/1 Running 0 14d test-job-default-nginx-2 1/1 Running 0 14d test-job-default-nginx-3 1/1 Running 0 14d test-job-default-nginx-4 1/1 Running 0 14d test-job-default-nginx-5 1/1 Running 0 14d test-job-v2-default-nginx-v2-0 1/1 Running 0 14d web-0 1/1 Running 0 89s web-1 1/1 Running 0 16h root@ubuntu:~/c++#
StatefulSet存储状态
上面说了StatefulSet可以通过PVC做持久化存储，保证Pod重新调度后还是能访问到相同的持久化数据，在删除Pod时，PVC不会被删除。
root@ubuntu:~/c++# kubectl exec web-1 -- sh -c 'echo hello world > /usr/share/nginx/html/index.html'
修改完后，如果在Pod中访问“http://localhost”，那就会返回“hello world”。
root@ubuntu:~/c++# kubectl exec -it web-1 -- curl localhost hello world root@ubuntu:~/c++#
此时如果手动删除web-1这个Pod，然后再次查询Pod，可以看到StatefulSet重新创建了一个名称相同的Pod，通过创建时间4s可以看出web-1是刚刚创建的。
再次访问该Pod的index.html页面，会发现仍然返回“hello world”，这说明这个Pod仍然是访问相同的存储。
root@ubuntu:~/c++# kubectl exec web-1 -- sh -c 'echo hello world > /usr/share/nginx/html/index.html' root@ubuntu:~/c++# kubectl exec -it web-1 -- curl localhost hello world root@ubuntu:~/c++# kubectl delete pod web-1 pod "web-1" deleted root@ubuntu:~/c++# kubectl get pods NAME READY STATUS RESTARTS AGE example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d job-1-nginx-0 0/1 Completed 0 22d test-job-default-nginx-0 1/1 Running 0 14d test-job-default-nginx-1 1/1 Running 0 14d test-job-default-nginx-2 1/1 Running 0 14d test-job-default-nginx-3 1/1 Running 0 14d test-job-default-nginx-4 1/1 Running 0 14d test-job-default-nginx-5 1/1 Running 0 14d test-job-v2-default-nginx-v2-0 1/1 Running 0 14d web-0 1/1 Running 0 5m49s web-1 0/1 ContainerCreating 0 10s root@ubuntu:~/c++# kubectl get pods NAME READY STATUS RESTARTS AGE example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d job-1-nginx-0 0/1 Completed 0 22d test-job-default-nginx-0 1/1 Running 0 14d test-job-default-nginx-1 1/1 Running 0 14d test-job-default-nginx-2 1/1 Running 0 14d test-job-default-nginx-3 1/1 Running 0 14d test-job-default-nginx-4 1/1 Running 0 14d test-job-default-nginx-5 1/1 Running 0 14d test-job-v2-default-nginx-v2-0 1/1 Running 0 14d web-0 1/1 Running 0 129m web-1 1/1 Running 0 124m root@ubuntu:~/c++# kubectl exec -it web-1 -- curl localhost hello world root@ubuntu:~/c++#
StatefulSet的网络标识
StatefulSet创建后，可以看下Pod是有固定名称的，那Headless Service是如何起作用的呢，那就是使用DNS，为Pod提供固定的域名，这样Pod间就可以使用域名访问，即便Pod被重新创建而导致Pod的IP地址发生变化，这个域名也不会发生变化。
Headless Service创建后，每个Pod的IP都会有下面格式的域名。
<pod-name>.<svc-name>.<namespace>.svc.cluster.local
root@ubuntu:~/tenant# ping web-0.nginx.default.svc.cluster.local ping: web-0.nginx.default.svc.cluster.local: Name or service not known root@ubuntu:~/tenant# ping web-1.nginx.default.svc.cluster.local ping: web-1.nginx.default.svc.cluster.local: Name or service not known root@ubuntu:~/tenant#
没有创建headless service
创建Headless Service
如前所述，创建Statefulset需要一个Headless Service用于Pod访问，Service的概念会在Service中详细介绍，这里先介绍Headless Service的创建方法。
使用如下文件描述Headless Service，其中：
- spec.clusterIP：必须设置为None，表示Headless Service。
- spec.ports.port：Pod间通信端口号。
- spec.ports.name：Pod间通信端口名称。
root@ubuntu:~/tenant# kubectl create -f head_less-svc.yaml service/nginx created root@ubuntu:~/tenant# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 27d nginx ClusterIP None <none> 80/TCP 6s test-job ClusterIP None <none> 1/TCP 14d test-job-v2 ClusterIP None <none> 1/TCP 14d root@ubuntu:~/tenant# kubectl describe svc nginx Name: nginx Namespace: default Labels: app=nginx Annotations: <none> Selector: app=nginx Type: ClusterIP IP: None Port: nginx 80/TCP TargetPort: 80/TCP Endpoints: 10.244.129.142:80,10.244.129.143:80,10.244.29.26:80 Session Affinity: None Events: <none> root@ubuntu:~/tenant# ping web-1.nginx.default.svc.cluster.local ping: web-1.nginx.default.svc.cluster.local: Name or service not known root@ubuntu:~/tenant# ping web-0.nginx.default.svc.cluster.local ping: web-0.nginx.default.svc.cluster.local: Name or service not known root@ubuntu:~/tenant# nslookup nginx.default.svc.cluster.local Server: 8.8.8.8 Address: 8.8.8.8#53 ** server can't find nginx.default.svc.cluster.local: NXDOMAIN root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl create -f head_less-svc.yaml service/nginx created root@ubuntu:~/tenant# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 27d nginx ClusterIP None <none> 80/TCP 6s test-job ClusterIP None <none> 1/TCP 14d test-job-v2 ClusterIP None <none> 1/TCP 14d root@ubuntu:~/tenant# kubectl describe svc nginx Name: nginx Namespace: default Labels: app=nginx Annotations: <none> Selector: app=nginx Type: ClusterIP IP: None Port: nginx 80/TCP TargetPort: 80/TCP Endpoints: 10.244.129.142:80,10.244.129.143:80,10.244.29.26:80 Session Affinity: None Events: <none> root@ubuntu:~/tenant# ping web-1.nginx.default.svc.cluster.local ping: web-1.nginx.default.svc.cluster.local: Name or service not known root@ubuntu:~/tenant# ping web-0.nginx.default.svc.cluster.local ping: web-0.nginx.default.svc.cluster.local: Name or service not known root@ubuntu:~/tenant# nslookup nginx.default.svc.cluster.local Server: 8.8.8.8 Address: 8.8.8.8#53 ** server can't find nginx.default.svc.cluster.local: NXDOMAIN root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl exec -it busybox -- nslookup web-0.nginx.default.svc.cluster.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: web-0.nginx.default.svc.cluster.local Address 1: 10.244.129.142 web-0.nginx.default.svc.cluster.local root@ubuntu:~/tenant# kubectl exec -it busybox -- nslookup web-1.nginx.default.svc.cluster.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: web-1.nginx.default.svc.cluster.local Address 1: 10.244.129.143 web-1.nginx.default.svc.cluster.local root@ubuntu:~/tenant#
删除headless service
root@ubuntu:~/tenant# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 27d nginx ClusterIP None <none> 80/TCP 36m test-job ClusterIP None <none> 1/TCP 14d test-job-v2 ClusterIP None <none> 1/TCP 14d root@ubuntu:~/tenant# kubectl delete svc nginx service "nginx" deleted root@ubuntu:~/tenant# kubectl exec -it busybox -- nslookup nginx.default.svc.cluster.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local nslookup: can't resolve 'nginx.default.svc.cluster.local' command terminated with exit code 1 root@ubuntu:~/tenant# kubectl exec -it busybox -- nslookup web-1.nginx.default.svc.cluster.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local nslookup: can't resolve 'web-1.nginx.default.svc.cluster.local' command terminated with exit code 1 root@ubuntu:~/tenant#
coredns
cat >busy.yaml<<-EOF apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: busybox:1.28 command: - sleep - "3600" imagePullPolicy: IfNotPresent restartPolicy: Always EOF
root@ubuntu:~/tenant# kubectl get pods NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 2m12s example-foo-54dc4db9fc-lqz9j 1/1 Running 0 19d job-1-nginx-0 0/1 Completed 0 22d test-job-default-nginx-0 1/1 Running 0 14d test-job-default-nginx-1 1/1 Running 0 14d test-job-default-nginx-2 1/1 Running 0 14d test-job-default-nginx-3 1/1 Running 0 14d test-job-default-nginx-4 1/1 Running 0 14d test-job-default-nginx-5 1/1 Running 0 14d test-job-v2-default-nginx-v2-0 1/1 Running 0 14d web-0 1/1 Running 0 157m web-1 1/1 Running 0 152m root@ubuntu:~/tenant# kubectl exec -it busybox -- ping kubernetes.default ping: bad address 'kubernetes.default' command terminated with exit code 1 root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl exec -it busybox -- sh / # ls bin dev etc home proc root run sys tmp usr var / # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 / # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if41: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue link/ether 2a:0c:12:56:87:7f brd ff:ff:ff:ff:ff:ff inet 10.244.129.145/32 brd 10.244.129.145 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::280c:12ff:fe56:877f/64 scope link valid_lft forever preferred_lft forever / # ping 10.96.0.10 PING 10.96.0.10 (10.96.0.10): 56 data bytes 64 bytes from 10.96.0.10: seq=0 ttl=64 time=954.020 ms 64 bytes from 10.96.0.10: seq=1 ttl=64 time=445.391 ms 64 bytes from 10.96.0.10: seq=2 ttl=64 time=445.277 ms ^C --- 10.96.0.10 ping statistics --- 4 packets transmitted, 3 packets received, 25% packet loss round-trip min/avg/max = 445.277/614.896/954.020 ms / #
解析异常排查
k8s默认在pod中的resolv.conf中配置的nameserver的ip是kube-dns的svc的ip 使用这个svc ip进行解析域名的时候会提示解析失败 手动把pod中的resolv.conf中的nameserver的ip配置成core-dns pod的ip即可成功解析
1.查看core-dns的service ip kubectl get svc -n kube-system service ip是一个虚拟IP 它是不能够被ping通的
root@ubuntu:~/tenant# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 27d root@ubuntu:~/tenant#
2.查看core dns的pod ip
root@ubuntu:~/tenant# kubectl get pods -n kube-system -o wide | grep core coredns-546565776c-82jfw 1/1 Running 0 27d 10.244.243.193 ubuntu <none> <none> coredns-546565776c-px8bd 1/1 Running 0 27d 10.244.243.194 ubuntu <none> <none> root@ubuntu:~/tenant#
3.查看业务pod中配置的dns服务器地址
root@ubuntu:~/tenant# kubectl exec -it busybox -- sh / # ls bin dev etc home proc root run sys tmp usr var / # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5
/ # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if41: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue link/ether 2a:0c:12:56:87:7f brd ff:ff:ff:ff:ff:ff inet 10.244.129.145/32 brd 10.244.129.145 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::280c:12ff:fe56:877f/64 scope link valid_lft forever preferred_lft forever / # ping 10.96.0.10 PING 10.96.0.10 (10.96.0.10): 56 data bytes 64 bytes from 10.96.0.10: seq=0 ttl=64 time=954.020 ms 64 bytes from 10.96.0.10: seq=1 ttl=64 time=445.391 ms 64 bytes from 10.96.0.10: seq=2 ttl=64 time=445.277 ms ^C --- 10.96.0.10 ping statistics --- 4 packets transmitted, 3 packets received, 25% packet loss round-trip min/avg/max = 445.277/614.896/954.020 ms
4、pod 是否可以访问coredns
coredns所在node无法访问pod
root@ubuntu:~# ping 10.244.129.145 PING 10.244.129.145 (10.244.129.145) 56(84) bytes of data. ^C --- 10.244.129.145 ping statistics --- 9 packets transmitted, 0 received, 100% packet loss, time 8193ms
5、查看防火墙
root@ubuntu:~# kubectl get networkpolicy NAME POD-SELECTOR AGE test-job volcano.sh/job-name=test-job,volcano.sh/job-namespace=default 14d test-job-v2 volcano.sh/job-name=test-job-v2,volcano.sh/job-namespace=default 14d root@ubuntu:~# ./calicoctl get networkpolicy NAME calico1-allow-same-org root@ubuntu:~# ./calicoctl delete networkpolicy calico1-allow-same-org Successfully deleted 1 'NetworkPolicy' resource(s)
root@ubuntu:~# ping 10.244.129.145 PING 10.244.129.145 (10.244.129.145) 56(84) bytes of data. 64 bytes from 10.244.129.145: icmp_seq=1 ttl=63 time=1208 ms 64 bytes from 10.244.129.145: icmp_seq=2 ttl=63 time=784 ms 64 bytes from 10.244.129.145: icmp_seq=3 ttl=63 time=576 ms 64 bytes from 10.244.129.145: icmp_seq=4 ttl=63 time=576 ms ^C --- 10.244.129.145 ping statistics --- 5 packets transmitted, 4 received, 20% packet loss, time 4003ms rtt min/avg/max/mdev = 576.779/786.698/1208.463/257.874 ms, pipe 2 root@ubuntu:~#
networkpolicy有问题
/ # ping kubernetes.default PING kubernetes.default (10.96.0.1): 56 data bytes 64 bytes from 10.96.0.1: seq=0 ttl=64 time=950.276 ms 64 bytes from 10.96.0.1: seq=1 ttl=64 time=444.556 ms ^C --- kubernetes.default ping statistics --- 3 packets transmitted, 2 packets received, 33% packet loss round-trip min/avg/max = 444.556/697.416/950.276 ms / #
root@ubuntu:~/tenant# kubectl exec -it busybox -- ping kubernetes.default PING kubernetes.default (10.96.0.1): 56 data bytes 64 bytes from 10.96.0.1: seq=0 ttl=64 time=950.287 ms 64 bytes from 10.96.0.1: seq=1 ttl=64 time=444.355 ms ^C --- kubernetes.default ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 444.355/697.321/950.287 ms root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl exec -it busybox -- nslookup nginx.default.svc.cluster.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: nginx.default.svc.cluster.local Address 1: 10.244.129.142 web-0.nginx.default.svc.cluster.local Address 2: 10.244.29.26 10-244-29-26.nginx.default.svc.cluster.local Address 3: 10.244.129.143 web-1.nginx.default.svc.cluster.local root@ubuntu:~/tenant# nslookup nginx.default.svc.cluster.local Server: 8.8.8.8 Address: 8.8.8.8#53 ** server can't find nginx.default.svc.cluster.local: NXDOMAIN root@ubuntu:~/tenant#
Kubernetes里的PersistentVolumeClaim
Kubernetes的pod本身是无状态的（stateless）,生命周期通常比较短，只要出现了异常，Kubernetes就会自动创建一个新的Pod来代替它。
而容器产生的数据，会随着Pod消亡而自动消失。
为了实现Pod内数据的存储管理，Kubernetes引入了两个API资源：Persistent Volume（持久卷，以下简称PV）和Persistent Volume Claim（持久卷申请，以下简称PVC）。
PV是Kubernetes集群中的一种网络存储实现，跟Node一样，也是属于集群的资源。
PV跟Docker里的Volume(卷)类似，不过会有独立于Pod的生命周期。
使用kubectl get pv查看列表：
而PVC是用户的一个请求，跟Pod类似。Pod消费Node的资源，PVC消费PV的资源。
Pod 能够申请特定的资源（CPU和内存）；PVC能够申请特定的尺寸和访问模式，例如可以加载一个读写实例或者多个只读实例，（就是上图kubectl get pvc返回结果的Access Mode这一列的值RWO, ROX等等）而无须感知这些实例背后具体的存储实现。
Pod配置PersistentVolumeClaim
1，创建PersistentVolume
cat >nginx-pv.yaml<<-EOF kind: PersistentVolume apiVersion: v1 metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: "/mnt/data" EOF
root@ubuntu:~/tenant# kubectl create -f nginx-pv.yaml persistentvolume/task-pv-volume created root@ubuntu:~/tenant# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Available manual 11s root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE www-web-0 Pending 24m root@ubuntu:~/tenant#
2，创建PersistentVolumeClaim
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 3Gi
root@ubuntu:~/tenant# kubectl create -f nginx-pvc.yaml persistentvolumeclaim/task-pv-claim created root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 47s www-web-0 Pending 26m root@ubuntu:~/tenant#
3，创建Pod使用PersistentVolumeClaim
kind: Pod
apiVersion: v1
metadata:
name: task-pv-pod
spec:
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: task-pv-claim
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: task-pv-storage
容器目录/usr/share/nginx/html已挂载到宿主机/mnt/data
使用kubectl describe pvc 查看这个persistent volume claim对应生成的persistent volume：
root@ubuntu:~/tenant# kubectl describe pvc task-pv-claim Name: task-pv-claim Namespace: default StorageClass: manual Status: Bound Volume: task-pv-volume Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes Finalizers: [kubernetes.io/pvc-protection] Capacity: 10Gi Access Modes: RWO VolumeMode: Filesystem Mounted By: task-pv-pod Events: <none>
root@ubuntu:~/tenant# cat nginx-pv.yaml kind: PersistentVolume apiVersion: v1 metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: "/data"
root@ubuntu:~/tenant# ls /data
k8s
root@ubuntu:~/tenant#
root@ubuntu:~/tenant# ls /data1/ a.out barrier.c long office.21-30.flv test10 test10.ii test10.s test11.cpp test2 test3 test4 test5 test6 test7 test8 test9 barrier core long.c test test10.cpp test10.o test11 test1.c test2.c test3.c test4.c test5.c test6.cpp test7.cpp test8.cpp test9.cpp root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl apply -f nginx-pv.yaml Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply The PersistentVolume "task-pv-volume" is invalid: spec.persistentvolumesource: Forbidden: is immutable after creation root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl create -f nginx-pv.yaml persistentvolume/task-pv-volume created root@ubuntu:~/tenant# kubectl create -f nginx-pvc.yaml persistentvolumeclaim/task-pv-claim created root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 11s www-web-0 Pending 39m root@ubuntu:~/tenant# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Bound default/task-pv-claim manual 16s root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES example-foo-54dc4db9fc-lqz9j 1/1 Running 0 18d 10.244.29.26 bogon <none> <none> job-1-nginx-0 0/1 Completed 0 21d 10.244.29.19 bogon <none> <none> task-pv-pod 1/1 Running 0 35s 10.244.129.133 centos7 <none> <none>
root@ubuntu:~# kubectl describe pvc task-pv-claim Name: task-pv-claim Namespace: default StorageClass: manual Status: Bound Volume: task-pv-volume Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes Finalizers: [kubernetes.io/pvc-protection] Capacity: 10Gi Access Modes: RWO VolumeMode: Filesystem Mounted By: task-pv-pod Events: <none>
root@ubuntu:~# kubectl cp job.yaml task-pv-pod:/usr/share/nginx/html
root@ubuntu:~/tenant# kubectl exec -it task-pv-pod -- bash root@task-pv-pod:/# ls /usr/share/nginx/html root@task-pv-pod:/# ls /usr/share/nginx/ html root@task-pv-pod:/# ls /usr/share/ X11 base-files bash-completion ca-certificates debconf dict doc-base fontconfig gcc-8 info keyrings lintian man misc pam perl5 polkit-1 sensible-utils terminfo zoneinfo adduser base-passwd bug common-licenses debianutils doc dpkg fonts gdb java libc-bin locale menu nginx pam-configs pixmaps readline tabset xml zsh root@task-pv-pod:/# ls /usr/ bin games include lib local sbin share src root@task-pv-pod:/# ls /usr/share/nginx/html root@task-pv-pod:/# cd /usr/share/nginx/html root@task-pv-pod:/usr/share/nginx/html# ls root@task-pv-pod:/usr/share/nginx/html# touch test.txt root@task-pv-pod:/usr/share/nginx/html# ls test.txt root@task-pv-pod:/usr/share/nginx/html# touch html root@task-pv-pod:/usr/share/nginx/html# echo "hello" >> html root@task-pv-pod:/usr/share/nginx/html# ls html job.yaml test.txt root@task-pv-pod:/usr/share/nginx/html#
在pod所在的node查看
[root@centos7 ~]# ls /data1/
html job.yaml test.txt
[root@centos7 ~]#
再创建一个pod
root@ubuntu:~/tenant# kubectl create -f nginx-another-pod.yaml pod/task-pv-another-pod created root@ubuntu:~/tenant# kubectl exec -it task-pv-another-pod -- bash root@task-pv-another-pod:/# ls bin boot dev docker-entrypoint.d docker-entrypoint.sh etc home lib media mnt opt proc root run sbin srv sys tmp usr var root@task-pv-another-pod:/# ls /usr/share/nginx/html html job.yaml test.txt root@task-pv-another-pod:/#
root@ubuntu:~/tenant# cat nginx-another-pod.yaml kind: Pod apiVersion: v1 metadata: name: task-pv-another-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: "http-server" volumeMounts: - mountPath: "/usr/share/nginx/html" name: task-pv-storage root@ubuntu:~/tenant#
root@ubuntu:~/tenant# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE task-pv-volume 10Gi RWO Retain Bound default/task-pv-claim manual 16m root@ubuntu:~/tenant# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE task-pv-claim Bound task-pv-volume 10Gi RWO manual 16m www-web-0 Pending 56m root@ubuntu:~/tenant#
这个例子说明persistent volume claim能够用于在多个pod间共享持久化数据。
删除两个pod
root@ubuntu:~/tenant# kubectl delete -f nginx-another-pod.yaml pod "task-pv-another-pod" deleted root@ubuntu:~/tenant# kubectl delete -f nginx-pod.yaml pod "task-pv-pod" deleted root@ubuntu:~/tenant#
data1目录还在
[root@centos7 ~]# ls /data1/
html job.yaml test.txt
[root@centos7 ~]#
StatefulSet
详解 Kubernetes StatefulSet 实现原理
PV、PVC、StorageClass讲解
