kubernetes之数据管理
volume
emptyDir
[machangwei@mcwk8s-master ~]$ kubectl apply -f mcwVolume1.yml #部署emptydir pod/producer-consumer created [machangwei@mcwk8s-master ~]$ cat mcwVolume1.yml apiVersion: v1 kind: Pod metadata: name: producer-consumer spec: containers: - image: busybox name: producer volumeMounts: - mountPath: /producer_dir name: shared-volume args: - /bin/sh - -c - echo "hello world" >/producer_dir/hello ; sleep 30000 - image: busybox name: consumer volumeMounts: - mountPath: /consumer_dir name: shared-volume args: - /bin/sh - -c - cat /consumer_dir/hello ; sleep 30000 volumes: - name: shared-volume emptyDir: {}
######定义一个volumes，使用空字典代表它是个空目录，起个名字。规格容器下，有两个镜像代表两个容器。这是pod种类资源。
#元数据名称就是运行的pod的名称。pod下两个容器，下面这行结果，ready下应该是两个容器，但是暂时是0个容器准备好。容器正在创建中
#容器下有镜像。每个镜像下做配置，一个镜像的配置就是一个容器。镜像下的名称就是容器的名称。指定逻辑卷挂载，指定这个容器的逻辑卷
#挂载路径是容器里哪个目录，需要挂载的逻辑卷名称是哪个。后面定义逻辑卷的时候有定义它的名称。然后这个容器的args参数，就是启动容器后，
#要运行的命令，这里生产者是每隔睡眠时间个单位，就写入数据到挂载目录下的某个文件。消费者就是每个睡眠个时间单位，去查看（消费）被挂载目录的文件
#由于两者是同一个volume挂载上去的，所以这个pod中的两个容器共享这一个volume。而这个volume实质就是一个目录，请往下看 [machangwei@mcwk8s-master ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE producer-consumer 0/2 ContainerCreating 0 84s [machangwei@mcwk8s-master ~]$ kubectl describe pod producer-consumer Name: producer-consumer Namespace: default Priority: 0 Node: mcwk8s-node1/10.0.0.5 Start Time: Fri, 18 Feb 2022 20:20:38 +0800 Labels: <none> Annotations: <none> Status: Running IP: 10.244.1.15 IPs: IP: 10.244.1.15 Containers: producer: Container ID: docker://e9a06c83f73861d15a115cc95665d19d8b6ef024546d339cc049d1571840cad7 Image: busybox Image ID: docker-pullable://busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678 Port: <none> Host Port: <none> Args: /bin/sh -c echo "hello world" >/producer_dir/hello ; sleep 30000 State: Running Started: Fri, 18 Feb 2022 20:21:54 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /producer_dir from shared-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-spkgq (ro) consumer: Container ID: docker://634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40 Image: busybox Image ID: docker-pullable://busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678 Port: <none> Host Port: <none> Args: /bin/sh -c cat /consumer_dir/hello ; sleep 30000 State: Running Started: Fri, 18 Feb 2022 20:22:11 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /consumer_dir from shared-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-spkgq (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: shared-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-spkgq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m39s default-scheduler Successfully assigned default/producer-consumer to mcwk8s-node1 Normal Pulling 105s kubelet Pulling image "busybox" Normal Pulled 89s kubelet Successfully pulled image "busybox" in 16.349119305s Normal Created 89s kubelet Created container producer Normal Started 83s kubelet Started container producer Normal Pulling 83s kubelet Pulling image "busybox" Normal Pulled 67s kubelet Successfully pulled image "busybox" in 16.122373015s Normal Created 67s kubelet Created container consumer Normal Started 66s kubelet Started container consumer [machangwei@mcwk8s-master ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE producer-consumer 2/2 Running 0 2m55s [machangwei@mcwk8s-master ~]$ kubectl logs producer-consumer error: a container name must be specified for pod producer-consumer, choose one of: [producer consumer] [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ kubectl logs producer-consumer consumer hello world [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide #查看pod所在的host NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES producer-consumer 2/2 Running 0 6m19s 10.244.1.15 mcwk8s-node1 <none> <none> [root@mcwk8s-node1 ~]$ docker ps|grep producer-consumer #pod部署主机上查看跟这个pod有关的容器。两个容器和pod在容一个host 634dc8fe053c busybox "/bin/sh -c 'cat /co…" 13 minutes ago Up 13 minutes k8s_consumer_producer-consume_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0 e9a06c83f738 busybox "/bin/sh -c 'echo \"h…" 13 minutes ago Up 13 minutes k8s_producer_producer-consume_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0 173864534fcb registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 14 minutes ago Up 13 minutes k8s_POD_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0 #它们的命名时部署配置中镜像下的名字是容器名，pod的POD,然后三者后面都是接pod名称producer-consumer
#看下面的，可以看到生产这容器和消费者容器源目录都是一样的，挂载的目标目录是各自容器中指定的目录。有的时候，可能容器改变，可能源目录和目标目录
发生了变化，当容器好了后，应该就都是如下类似。的 [root@mcwk8s-node1 ~]$ docker inspect e9a06c8|grep -iA 8 "mounts" "Mounts": [ { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume", "Destination": "/producer_dir", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, [root@mcwk8s-node1 ~]$ docker inspect 634dc8|grep -iA 8 "mounts" "Mounts": [ { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume", "Destination": "/consumer_dir", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, [root@mcwk8s-node1 ~]$
查看逻辑卷在物理机中的实际位置。以及两个容器操作后的数据
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume
hello
[root@mcwk8s-node1 ~]$ cat /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume/hello
hello world
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/
kubernetes.io~empty-dir kubernetes.io~projected
[root@mcwk8s-node1 ~]$ docker inspect 634dc8 [ { "Id": "634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40", "Created": "2022-02-18T12:22:10.65373133Z", "Path": "/bin/sh", "Args": [ "-c", "cat /consumer_dir/hello ; sleep 30000" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 91494, "ExitCode": 0, "Error": "", "StartedAt": "2022-02-18T12:22:11.920497543Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a", "ResolvConfPath": "/var/lib/docker/containers/173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2/resolv.conf", "HostnamePath": "/var/lib/docker/containers/173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2/hostname", "HostsPath": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/etc-hosts", "LogPath": "/var/lib/docker/containers/634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40/634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40-json.log", "Name": "/k8s_consumer_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume:/consumer_dir:Z", "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~projected/kube-api-access-spkgq:/var/run/secrets/kubernetes.io/serviceaccount:ro,Z", "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/etc-hosts:/etc/hosts:Z", "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/containers/consumer/61a05b66:/dev/termination-log:Z" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "container:173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2", "PortBindings": null, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "CgroupnsMode": "host", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "container:173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2", "Cgroup": "", "Links": null, "OomScoreAdj": 1000, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "seccomp=unconfined" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 2, "Memory": 0, "NanoCpus": 0, "CgroupParent": "kubepods-besteffort-podd32da9de_8574_458a_97c7_31cf9107f0a1.slice", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 100000, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/asound", "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d-init/diff:/var/lib/docker/overlay2/5c9e2add80e4eb9b40376cc60407679fdc4b510a0c146356397e8a769c1a307c/diff", "MergedDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d/merged", "UpperDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d/diff", "WorkDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume", "Destination": "/consumer_dir", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~projected/kube-api-access-spkgq", "Destination": "/var/run/secrets/kubernetes.io/serviceaccount", "Mode": "ro,Z", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/etc-hosts", "Destination": "/etc/hosts", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/containers/consumer/61a05b66", "Destination": "/dev/termination-log", "Mode": "Z", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "producer-consumer", "Domainname": "", "User": "0", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "KUBERNETES_PORT_443_TCP_PROTO=tcp", "KUBERNETES_PORT_443_TCP_PORT=443", "KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1", "KUBERNETES_SERVICE_HOST=10.96.0.1", "KUBERNETES_SERVICE_PORT=443", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT=tcp://10.96.0.1:443", "KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/bin/sh", "-c", "cat /consumer_dir/hello ; sleep 30000" ], "Healthcheck": { "Test": [ "NONE" ] }, "Image": "busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "annotation.io.kubernetes.container.hash": "2c893f27", "annotation.io.kubernetes.container.restartCount": "0", "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy": "File", "annotation.io.kubernetes.pod.terminationGracePeriod": "30", "io.kubernetes.container.logpath": "/var/log/pods/default_producer-consumer_d32da9de-8574-458a-97c7-31cf9107f0a1/consumer/0.log", "io.kubernetes.container.name": "consumer", "io.kubernetes.docker.type": "container", "io.kubernetes.pod.name": "producer-consumer", "io.kubernetes.pod.namespace": "default", "io.kubernetes.pod.uid": "d32da9de-8574-458a-97c7-31cf9107f0a1", "io.kubernetes.sandbox.id": "173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": {} } } ] [root@mcwk8s-node1 ~]$
[root@mcwk8s-node1 ~]$ docker ps|grep producer-consumer 634dc8fe053c busybox "/bin/sh -c 'cat /co…" 3 hours ago Up 3 hours k8s_consumer_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0 e9a06c83f738 busybox "/bin/sh -c 'echo \"h…" 3 hours ago Up 3 hours k8s_producer_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0 173864534fcb registry.aliyuncs.com/google_containers/pause:3.6 "/pause" 3 hours ago Up 3 hours k8s_POD_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0 [root@mcwk8s-node1 ~]$ [root@mcwk8s-node1 ~]$ [root@mcwk8s-node1 ~]$ docker inspect 634dc8|grep -iA 8 "mounts" "Mounts": [ { "Type": "bind", "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume", "Destination": "/consumer_dir", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, [root@mcwk8s-node1 ~]$ docker exec -it 634d cat /consumer_dir/hello hello world [root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume hello [root@mcwk8s-node1 ~]$ cat /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume/hello hello world [root@mcwk8s-node1 ~]$ echo -e " mcw">>/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume/hello [root@mcwk8s-node1 ~]$ docker exec -it 634d cat /consumer_dir/hello #修改host目录，容器中显示被修改后的 hello world mcw [root@mcwk8s-node1 ~]$ docker exec -it 634d touch /consumer_dir/mcw.txt #容器中修改，host中也是修改后的 [root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume hello mcw.txt [root@mcwk8s-node1 ~]$
hostPath
[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces|grep apiserver kube-system kube-apiserver-mcwk8s-master 1/1 Running 15 (110m ago) 28d [machangwei@mcwk8s-master ~]$ kubectl edit --namespace=kube-system pod kube-apiserver-mcwk8s-master ...... volumeMounts: #我们查看apiserver服务的配置。 - mountPath: /etc/ssl/certs #可以看到挂载容器中的三个目录，使用某个名称的volume，是否只读挂载 name: ca-certs #下面有定义用来挂载的volume readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostNetwork: true nodeName: mcwk8s-master preemptionPolicy: PreemptLowerPriority priority: 2000001000 priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: seccompProfile: type: RuntimeDefault terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute operator: Exists volumes: - hostPath: #host主机path目录被映射到使用这个容器的目录中，起个名字后面通过名字使用这个host上的目录 path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs
外部Storage Provider
云硬盘存储
PersistentVolume & PersistentVolumeClaim
| persistent | 英[pəˈsɪstənt] | 美[pərˈsɪstənt] |
| adj. | 持久的; 持续的; 坚持不懈的; 执著的; 不屈不挠的; 连绵的; 反复出现的; |
| claim | 英[kleɪm] | 美[kleɪm] |
| v. | 宣称; 声称; 断言; 要求(拥有); 索取; 认领; 索要; 引起（注意）; 获得; 夺走，夺去（生命）; |
| n. | 声明; 宣称; 断言; (尤指对财产、土地等要求拥有的)所有权; (尤指向公司、政府等)索款，索赔; |
PersistentVolume & PersistentVolumeClaim pv &pvc
pv持久化volume，pvc持久化volume申请。
NFS PersistentVolume
nfs服务器
参考：https://www.cnblogs.com/machangwei-8/articles/15487295.html#_label5
[root@mcwk8s-master ~]$ showmount -e 10.0.0.4 #先部署好nfs服务器，这里nfs的共享目录是/nfsdata Export list for 10.0.0.4: /nfsdata * [root@mcwk8s-master ~]$
要使用外部存储，这里使用nfs挂载存储，得有nfs服务器。上面已经部署好了。
pv
[machangwei@mcwk8s-master ~]$ vim nfs-pv1.yml [machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pv1.yml persistentvolume/mypv1 created [machangwei@mcwk8s-master ~]$ cat nfs-pv1.yml apiVersion: v1 kind: PersistentVolume metadata: name: mypv1 spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: nfs nfs: path: /nfsdata/pv1 server: 10.0.0.4 [machangwei@mcwk8s-master ~]$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Recycle Available nfs 21s
配置pv，配置api版本，种类是持久化逻辑卷。元数据名称就是逻辑卷名称。
规格中容量存储设置，访问模式是什么，这里设置该pv只能读写模式挂载一个节点。模式还有ReadWriteMany 能以读写模式挂载到多个节点；ReadOnlyMany 能以只读模式挂载到多个节点。
定义持久化逻辑卷重新声明策略，即回收策略。Reain 需要管理员手工回；Recycle清除PV中的数据，；Delete删除存储提供者上对应的资源
定义存储类名称，这里是使用nfs存储。nfs的信息需要设置，提供的挂载path是什么，访问服务地址是什么
pvc
种类是持久化逻辑卷声明；api版本多少，元数据名称设置逻辑卷名称；规则里设置访问模式，这里是以读写模式可以挂载到单个节点；资源是多少，请求存储大小是多少设置。存储类的名称是nfs
[machangwei@mcwk8s-master ~]$ cat nfs-pvc1.yml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mypvc1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: nfs [machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pvc1.yml persistentvolumeclaim/mypvc1 created [machangwei@mcwk8s-master ~]$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc1 Bound mypv1 1Gi RWO nfs 8s [machangwei@mcwk8s-master ~]$ kubectl get pvc -o wide #查看到pvc绑定的pv，声明之后就绑定了逻辑卷 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE mypvc1 Bound mypv1 1Gi RWO nfs 18s Filesystem [machangwei@mcwk8s-master ~]$ kubectl get pv -o wide #查看到pv被pvc绑定了， NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE mypv1 1Gi RWO Recycle Bound default/mypvc1 nfs 2m40s Filesystem
pod还有报错了
[machangwei@mcwk8s-master ~]$ cat pod1.yml kind: Pod apiVersion: v1 metadata: name: mypod1 spec: containers: - name: mypod1 image: busybox args: - /bin/sh - -c - sleep 30000 volumeMounts: - mountPath: "/mydata" name: mydata volumes: - name: mydata persistentVolumeClaim: claimName: mypvc1
#pod中使用创建的pvc。种类是pod，元数据名称设置pod名称。规格里设置容器信息。容器逻辑卷和镜像是同一级下的，挂载容器中指定路径是什么，使用哪个逻辑卷名称；下面定义了这个
#使用的逻辑卷，这个逻辑卷怎么定义的？逻辑卷定义和容器同级，定义逻辑卷名称，持久化逻辑卷声明的名字是哪个，就是之前创建好的哪个pvc。这样修改容器中的那个目录，容器对应该主机的
#某个逻辑卷目录，而该目录又被挂载nfs挂载着，这样容器就间接访问到了nfs上的目录 [machangwei@mcwk8s-master ~]$ kubectl apply -f pod1.yml pod/mypod1 created [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mypod1 0/1 ContainerCreating 0 16s <none> mcwk8s-node1 <none> <none> [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mypod1 0/1 ContainerCreating 0 33s <none> mcwk8s-node1 <none> <none> [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mypod1 0/1 ContainerCreating 0 62s <none> mcwk8s-node1 <none> <none> [machangwei@mcwk8s-master ~]$ kubectl describe pod mypod1 Name: mypod1 Namespace: default Priority: 0 Node: mcwk8s-node1/10.0.0.5 Start Time: Sat, 19 Feb 2022 01:21:32 +0800 Labels: <none> Annotations: <none> Status: Pending IP: IPs: <none> Containers: mypod1: Container ID: Image: busybox Image ID: Port: <none> Host Port: <none> Args: /bin/sh -c sleep 30000 State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /mydata from mydata (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-28s6l (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: mydata: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: mypvc1 ReadOnly: false kube-api-access-28s6l: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 80s default-scheduler Successfully assigned default/mypod1 to mcwk8s-node1 Warning FailedMount 13s (x8 over 77s) kubelet MountVolume.SetUp failed for volume "mypv1" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 10.0.0.4:/nfsdata/pv1 /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1 Output: mount: wrong fs type, bad option, bad superblock on 10.0.0.4:/nfsdata/pv1, missing codepage or helper program, or other error (for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program) In some cases useful info is found in syslog - try dmesg | tail or so. [machangwei@mcwk8s-master ~]$ #由上可知，就是将网络存储挂载到host本地某个目录
解决上面的pod部署挂载问题
缺少包
上面的原因是，由于部署pod在节点上，从节点上把nfs服务(在主上)挂载到节点1， 由于缺少包，无法挂载，导致pod无法在节点上做挂载。 在节点1无法挂载nfs [root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/ /root/mcw/ mount: wrong fs type, bad option, bad superblock on 10.0.0.4:/nfsdata/, missing codepage or helper program, or other error (for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program) In some cases useful info is found in syslog - try dmesg | tail or so. 原因，缺少包，安装一下。 [root@mcwk8s-node1 ~]$ yum install nfs-utils ^C [root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/ /root/mcw/ [root@mcwk8s-node1 ~]$ df -h|tail -1 #可以正常挂载了 10.0.0.4:/nfsdata 19G 3.9G 15G 22% /root/mcw [root@mcwk8s-node1 ~]$ df -h #过一会看pv1已经挂载上了 Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 19G 2.6G 16G 14% / devtmpfs 478M 0 478M 0% /dev tmpfs 489M 0 489M 0% /dev/shm tmpfs 489M 51M 438M 11% /run tmpfs 489M 0 489M 0% /sys/fs/cgroup /dev/sda1 797M 125M 673M 16% /boot tmpfs 877M 12K 877M 1% /var/lib/kubelet/pods/9b74614a-6d0b-439c-8ea5-cf1583be7a5a/volumes/kubernetes.io~projected/kube-api-access-6hk5l tmpfs 877M 12K 877M 1% /var/lib/kubelet/pods/57f73623-751b-41a3-883e-62c025c49f92/volumes/kubernetes.io~projected/kube-api-access-wv8cc overlay 19G 2.6G 16G 14% /var/lib/docker/overlay2/c3a132cb506fe3db901260f4e8e40ab73c20b6dbb8904c545bd6c12e45af2ce9/merged overlay 19G 2.6G 16G 14% /var/lib/docker/overlay2/9c92a061f975e8f573fc9b895664adeeec375d7609c6e00fd5facada9a525a13/merged shm 64M 0 64M 0% /var/lib/docker/containers/fc176c43633f8e6362d47f6bbec6f2f556685bbe51564fe189cf58cd9bf69fa5/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/3bbd8916f1ae8819e27eb1269afa524eca1c60a5d0b8df56dbbf314217c063a0/mounts/shm overlay 19G 2.6G 16G 14% /var/lib/docker/overlay2/914fc6440a30892ef51129fb445349c082d79ffb0fd79d62769384c7920e983a/merged overlay 19G 2.6G 16G 14% /var/lib/docker/overlay2/fe20b74227debccec19071237760ba6c37be72eadccc3999490a8aeae432919a/merged tmpfs 98M 0 98M 0% /run/user/0 tmpfs 877M 12K 877M 1% /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~projected/kube-api-access-28s6l 10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1 overlay 19G 2.6G 16G 14% /var/lib/docker/overlay2/b316299270c3541708659a9bbd6c65bb6d1fe4fa713460ad298866e1096adb0a/merged shm 64M 0 64M 0% /var/lib/docker/containers/b35f15c56b2e6f635dd346a9352fa0cbcb77ec9ba3f9a53b62f7bb53cb249e30/mounts/shm overlay 19G 2.6G 16G 14% /var/lib/docker/overlay2/bda202921f17c4ec866e22f2517af01a6df6434e4107b05588f4a001d8b5ec49/merged 但是验证的时候，报错只读 [machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch /mydata/hello kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. touch: /mydata/hello: Read-only file system command terminated with exit code 1 去节点上查看pod写入不了 [root@mcwk8s-node1 ~]$ docker exec -it e09 touch /mydata/mcw.txt touch: /mydata/mcw.txt: Read-only file system
这个无法从容器中写入文件到nfs存储里，跟容器有关吗。下面直接通过挂载的方式，将某个目录挂载上去了，但是写入文件容易显示只读。说明这是nfs服务端的问题。
[root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/pv1 /root/mcw/
[root@mcwk8s-node1 ~]$ df -h|grep mcw
10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /root/mcw
[root@mcwk8s-node1 ~]$ touch /root/mcw/test.txt
touch: cannot touch ‘/root/mcw/test.txt’: Read-only file system
[root@mcwk8s-node1 ~]$
nfs问题导致的容器无法写入数据到外部存储的问题解决
这个无法从容器中写入文件到nfs存储里，跟容器有关吗。下面直接通过挂载的方式， 将某个目录挂载上去了，但是写入文件容易显示只读。说明这是nfs服务端的问题。 [root@mcwk8s-master ~]$ cat /etc/exports /nfsdata * /nfsdata/pv1 * [root@mcwk8s-master ~]$ vim /etc/exports [root@mcwk8s-master ~]$ cat /etc/exports /nfsdata * (rw,sync,all_squash) /nfsdata/pv1 * (rw,sync,all_squash) [root@mcwk8s-master ~]$ systemctl restart nfs [root@mcwk8s-master ~]$ showmount -e 10.0.0.4 Export list for 10.0.0.4: /nfsdata/pv1 * /nfsdata * 然后再去节点1上挂载发现还是不能写入文件 [root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/pv1 /root/mcw/ [root@mcwk8s-node1 ~]$ df -h|grep mcw 10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /root/mcw [root@mcwk8s-node1 ~]$ touch /root/mcw/test.txt touch: cannot touch ‘/root/mcw/test.txt’: Read-only file system 再次修改nfs配置，*代表所有ip网段，后面将小括号里的设置，弄成和ip间是没有空格的，这样才生效了 [root@mcwk8s-master ~]$ vim /etc/exports [root@mcwk8s-master ~]$ cat /etc/exports /nfsdata *(rw,sync,all_squash) /nfsdata/pv1 *(rw,sync,all_squash) [root@mcwk8s-master ~]$ systemctl restart nfs [root@mcwk8s-master ~]$ showmount -e 10.0.0.4 Export list for 10.0.0.4: /nfsdata/pv1 * /nfsdata * [root@mcwk8s-master ~]$ 再次去节点1上访问，发现成功创建 [root@mcwk8s-node1 ~]$ touch /root/mcw/test.txt [root@mcwk8s-node1 ~]$ ls /root/mcw/ test.txt [root@mcwk8s-node1 ~]$ df -h|grep mcw 10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /root/mcw 主节点上也可以看到这个文件 [root@mcwk8s-master ~]$ ls /nfsdata/pv1/ test.txt 也可以正常从容器中写入文件到nfs服务器 [machangwei@mcwk8s-master ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE mypod1 1/1 Running 0 177m [machangwei@mcwk8s-master ~]$ kubectl exec mypod1 ls /mydata kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. test.txt [machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch /mydata/mcw.txt kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ mcw.txt test.txt 节点1上看挂载目录，首先是10.0.0.4:/nfsdata/pv1这个目录重新被挂载了， 虽然df查看只显示/root/mcw，但是也是可以通过之前那个mypv1目录去访问到nfs服务器上的文件的 当把/root/mcw卸载后，那么之前那个mypv1就显示出来了。并且可以正常使用 [root@mcwk8s-node1 ~]$ df -h|grep pv1 10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /root/mcw [root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1 mcw.txt test.txt [root@mcwk8s-node1 ~]$ unmout /root/mcw/ -bash: unmout: command not found [root@mcwk8s-node1 ~]$ unmount /root/mcw/ -bash: unmount: command not found [root@mcwk8s-node1 ~]$ unmount /root/mcw/ unalias unexpand unicode_stop unix2dos unix_chkpwd unlink unshare unxz uname unicode_start uniq unix2mac unix_update unset until [root@mcwk8s-node1 ~]$ umount /root/mcw/ [root@mcwk8s-node1 ~]$ df -h|grep pv1 10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1 [root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1 mcw.txt test.txt
回收PV
会删除nfs数据的回收pv
[machangwei@mcwk8s-master ~]$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc1 Bound mypv1 1Gi RWO nfs 3h31m [machangwei@mcwk8s-master ~]$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Recycle Bound default/mypvc1 nfs 3h33m [machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1 #删除pvc失败，pvc状态也变成终止，而不是绑定 persistentvolumeclaim "mypvc1" deleted ^C [machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1 persistentvolumeclaim "mypvc1" deleted ^C [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ kubectl get pod #删除pod，再删除pvc,pvc成功删除，网上说是删除pod，再删pvc,pv,应该就是这个顺序吧 NAME READY STATUS RESTARTS AGE mypod1 1/1 Running 0 3h28m [machangwei@mcwk8s-master ~]$ kubectl delete pod mypod1 pod "mypod1" deleted [machangwei@mcwk8s-master ~]$ kubectl get pvc No resources found in default namespace. [machangwei@mcwk8s-master ~]$ kubectl get pod #删除pvc的时候创建了回收pv的容器 NAME READY STATUS RESTARTS AGE recycler-for-mypv1 0/1 ContainerCreating 0 33s [machangwei@mcwk8s-master ~]$ kubectl get pv #状态现在是空闲，pv可以重新被其它pvc声明绑定。在删除pvc的过程中应该是被释放的状态，那时是不能被其它pvc绑定 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Recycle Available nfs 3h41m [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ #删除pvc后，pv虽然空闲了，但是pv对应的 [machangwei@mcwk8s-master ~]$ #nfs服务器存储数据都被删除了。这是回收策略决定的，所以当我们要 [machangwei@mcwk8s-master ~]$ #删除pvc时，一定要知道pv使用的什么回收策略，防止本来不想删除pv数据的，但是被删掉的情况发生 [machangwei@mcwk8s-master ~]$ #删除pvc，但是不删除外部存储nfs上的数据，得用Retain的回收策略
不会删除nfs服务端数据的回收pv
[machangwei@mcwk8s-master ~]$ cat nfs-pvc1.yml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mypvc1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: nfs [machangwei@mcwk8s-master ~]$ cat nfs-pv1.yml #将pv的回收策略修改为Retain，这样删除pvc就不会删除掉nfs服务端数据了 apiVersion: v1 kind: PersistentVolume metadata: name: mypv1 spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: nfs nfs: path: /nfsdata/pv1 server: 10.0.0.4 [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pv1.yml persistentvolume/mypv1 configured [machangwei@mcwk8s-master ~]$ kubectl get pv #查看可知，现在回收策略已经修改了 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Retain Available nfs 3h56m [machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pvc1.yml persistentvolumeclaim/mypvc1 created [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc1 Bound mypv1 1Gi RWO nfs 10s [machangwei@mcwk8s-master ~]$ cat pod1.yml kind: Pod apiVersion: v1 metadata: name: mypod1 spec: containers: - name: mypod1 image: busybox args: - /bin/sh - -c - sleep 30000 volumeMounts: - mountPath: "/mydata" name: mydata volumes: - name: mydata persistentVolumeClaim: claimName: mypvc1 [machangwei@mcwk8s-master ~]$ kubectl apply -f pod1.yml #部署pod pod/mypod1 created [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mypod1 1/1 Running 0 3m16s 10.244.2.16 mcwk8s-node2 <none> <none> [machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch ls /mydata kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ #查看nfs服务端没有数据 [machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch /mydata/mcw.txt #执行容器名字，后接命令。这样就相当于进入容器执行命令 kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ #容器中创建了文件，可以看到nfs服务端已经存在这个文件了 mcw.txt [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc1 Bound mypv1 1Gi RWO nfs 7m10s [machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1 #此时删除pvc，卡住很长时间不动 再开一个xshell会话，将使用pvc的pod删除掉， [machangwei@mcwk8s-master ~]$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Retain Bound default/mypvc1 nfs 4h5m [machangwei@mcwk8s-master ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE mypod1 1/1 Running 0 8m31s [machangwei@mcwk8s-master ~]$ kubectl delete pod mypod1 #当删除掉pod后，pvc也立即删除了，pv不再是绑定状态，暂时是被释放的状态 pod "mypod1" deleted [machangwei@mcwk8s-master ~]$ pod删除后,pvc也被删除了，应该是需要删除pod，才能删除掉pvc吧，目前看是这个样子。 [machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1 persistentvolumeclaim "mypvc1" deleted [machangwei@mcwk8s-master ~]$ kubectl get pv #删除掉pvc后，发现状态一直是被释放的，只有状态是空闲的，才能重新部署pvc使用这个pv，也就是才能被其它pvc申请 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Retain Released default/mypvc1 nfs 4h7m [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ #查看到即使删除了pvc,但是外部存储数据nfs服务端的数据还是存在的。 mcw.txt [machangwei@mcwk8s-master ~]$
让pv中的数据重新被使用
[machangwei@mcwk8s-master ~]$ #retain回收策略，被释放的状态，对应的pvc不存在了， [machangwei@mcwk8s-master ~]$ kubectl get pv #pv下面的数据还想被重新使用， NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Retain Released default/mypvc1 nfs 4h16m [machangwei@mcwk8s-master ~]$ kubectl get pvc No resources found in default namespace. [machangwei@mcwk8s-master ~]$ kubectl delete pv mypv1 #那么需要删除pv, persistentvolume "mypv1" deleted [machangwei@mcwk8s-master ~]$ [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ #这种情况的pv删除是不会将外部数据删除掉的 mcw.txt [machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pv1.yml #然后重新部署pv,pvc,pod persistentvolume/mypv1 created [machangwei@mcwk8s-master ~]$ kubectl get pv #重新部署了pv，pv空闲可被pvc声明使用 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE mypv1 1Gi RWO Retain Available nfs 12s [machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pvc1.yml persistentvolumeclaim/mypvc1 created [machangwei@mcwk8s-master ~]$ kubectl apply -f pod1.yml pod/mypod1 created [machangwei@mcwk8s-master ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE mypod1 1/1 Running 0 4m26s [machangwei@mcwk8s-master ~]$ kubectl exec mypod1 ls /mydata #然后发现pod下容器重新使用这个数据了。也就是这种情况下数据是能保留下来的 kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. mcw.txt [machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/ mcw.txt [machangwei@mcwk8s-master ~]$
PV动态供给
AWS EBS 支持pv动态供给，以后补充
https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner
mysql数据库持久化逻辑卷使用案例(nfs)
查看部署配置文件
[machangwei@mcwk8s-master ~]$ cat mysql-pv.yml apiVersion: v1 kind: PersistentVolume metadata: name: mysql-pv spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi persistentVolumeReclaimPolicy: Retain storageClassName: nfs nfs: path: /nfsdata/mysql-pv server: 10.0.0.4 [machangwei@mcwk8s-master ~]$ cat mysql-pvc.yml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mysql-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: nfs [machangwei@mcwk8s-master ~]$ cat mysql.yml apiVersion: v1 kind: Service metadata: name: mysql spec: ports: - port: 3306 selector: app: mysql --- apiVersion: apps/v1 kind: Deployment metadata: name: mysql spec: selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD value: password ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc [machangwei@mcwk8s-master ~]$
详解配置文件
[machangwei@mcwk8s-master ~]$ cat mysql-pv.yml apiVersion: v1 kind: PersistentVolume metadata: name: mysql-pv spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi persistentVolumeReclaimPolicy: Retain #使用Reatin回收策略，即使删除pv,pvc，数据也不清除 storageClassName: nfs nfs: path: /nfsdata/mysql-pv server: 10.0.0.4 [machangwei@mcwk8s-master ~]$ cat mysql-pvc.yml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mysql-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: nfs [machangwei@mcwk8s-master ~]$ cat mysql.yml apiVersion: v1 kind: Service #服务种类 metadata: name: mysql #元数据名称是服务名称 spec: #查看规则 ports: #服务使用端口是什么 - port: 3306 selector: #使用哪个选择器这里是使用app mysql的选择器。应该是创建app mysql 标签，后面选择器匹配的时候用把 app: mysql --- apiVersion: apps/v1 kind: Deployment metadata: name: mysql spec: selector: #选择器匹配标签带有app mysql 的 matchLabels: app: mysql template: #查看模板 metadata: 元数据标签是app mysql 模板规则如下 labels: app: mysql spec: containers: - image: mysql:5.6 #容器使用哪个镜像 name: mysql #容器的名称是什么 env: - name: MYSQL_ROOT_PASSWORD #容器内环境变量咋样，这里定义了mysql用户密码 value: password ports: - containerPort: 3306 #端口下，容器端口用的是哪个，并且是什么名字 name: mysql volumeMounts: - name: mysql-persistent-storage #容器逻辑卷挂载吗，和容器同级。 mountPath: /var/lib/mysql #容器内部挂载目录 volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc [machangwei@mcwk8s-master ~]$
部署pv，pvc ,service，deployment
[machangwei@mcwk8s-master ~]$ kubectl apply -f mysql-pv.yml persistentvolume/mysql-pv unchanged [machangwei@mcwk8s-master ~]$ kubectl apply -f mysql-pvc.yml persistentvolumeclaim/mysql-pvc created [machangwei@mcwk8s-master ~]$ kubectl get pv,pvc NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/mysql-pv 1Gi RWO Retain Bound default/mysql-pvc nfs 4m22s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/mysql-pvc Bound mysql-pv 1Gi RWO nfs 11s [machangwei@mcwk8s-master ~]$ kubectl apply -f mysql.yml service/mysql unchanged deployment.apps/mysql created [machangwei@mcwk8s-master ~]$ kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d mysql ClusterIP 10.97.19.121 <none> 3306/TCP 9m22s [machangwei@mcwk8s-master ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE mysql-dbffc69d-trvvb 0/1 ContainerCreating 0 8m25s 查看是卡住了，mysq-pv目录没有，挂载上又卸载不下来了 Normal Scheduled 8m7s default-scheduler Successfully assigned default/mysql-dbffc69d-trvvb to mcwk8s-node2 Warning FailedMount <invalid> (x11 over <invalid>) kubelet MountVolume.SetUp failed for volume "mysql-pv" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 10.0.0.4:/nfsdata/mysql-pv /var/lib/kubelet/pods/6392d0d9-15be-4ba9-a338-61f290e6f999/volumes/kubernetes.io~nfs/mysql-pv Output: mount.nfs: mounting 10.0.0.4:/nfsdata/mysql-pv failed, reason given by server: No such file or directory Warning FailedMount <invalid> (x3 over <invalid>) kubelet Unable to attach or mount volumes: unmounted volumes=[mysql-persistent-storage], unattached volumes=[mysql-persistent-storage kube-api-access-v6lfr]: timed out waiting for the condition 我删除这个pod也卡在删不掉，找到pod所在节点，然后把这个挂载上的目录，手动执行命令umount卸载掉，然后就把这pod删除了，随后生成新的pod，运行起来了 [machangwei@mcwk8s-master ~]$ kubectl delete pod mysql-dbffc69d-trvvb pod "mysql-dbffc69d-trvvb" deleted
后面重新部署
[machangwei@mcwk8s-master ~]$ ls mysql-pvc.yml mysql-pv.yml mysql.yml [machangwei@mcwk8s-master ~]$ kubectl apply -f mysql-pv.yml persistentvolume/mysql-pv created [machangwei@mcwk8s-master ~]$ kubectl apply -f mysql-pvc.yml persistentvolumeclaim/mysql-pvc created [machangwei@mcwk8s-master ~]$ kubectl apply -f mysql.yml service/mysql created deployment.apps/mysql created 报错 ---- ------ ---- ---- ------- Normal Scheduled 13m default-scheduler Successfully assigned default/mysql-dbffc69d-292lp to mcwk8s-node2 Normal Pulling 13m kubelet Pulling image "mysql:5.6" Normal Pulled 10m kubelet Successfully pulled image "mysql:5.6" in 2m17.834341538s Normal Created 9m24s (x5 over 10m) kubelet Created container mysql Normal Started 9m24s (x5 over 10m) kubelet Started container mysql Normal Pulled 9m24s (x4 over 10m) kubelet Container image "mysql:5.6" already present on machine Warning BackOff 3m13s (x37 over 10m) kubelet Back-off restarting failed container [root@mcwk8s-node2 ~]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 75829eceaae7 dd3b2a5dcb48 "docker-entrypoint.s…" About a minute ago Exited (1) About a minute ago k8s_mysql_mysql-dbffc69d-292lp_default_e9825a68-47b0-471d-8280-b95bd51c4068_6 去节点上查看MySQL容器日志，报错 [root@mcwk8s-node2 ~]$ docker logs 758 2022-02-19 11:16:19+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.51-1debian9 started. chown: changing ownership of '/var/lib/mysql/': Operation not permitted 解决方法，给nfs添加配置no_root_squash [root@mcwk8s-master ~]$ cat /etc/exports /nfsdata *(rw,sync) [root@mcwk8s-master ~]$ vim /etc/exports [root@mcwk8s-master ~]$ cat /etc/exports /nfsdata *(rw,sync,no_root_squash) [root@mcwk8s-master ~]$ systemctl restart nfs [root@mcwk8s-master ~]$ showmount -e localhost Export list for localhost: /nfsdata * [root@mcwk8s-master ~]$
进入MySQL并创建数据
[machangwei@mcwk8s-master ~]$ kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-mcwclient -- mysql -h 10.103.171.207 -P3306 -ppassword If you don't see a command prompt, try pressing enter. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0.03 sec) mysql> create database mcwtest; Query OK, 1 row affected (0.08 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mcwtest | | mysql | | performance_schema | +--------------------+ 4 rows in set (0.03 sec) mysql> use mcwtest; Database changed mysql> create table my_id(id int(4)) -> ; Query OK, 0 rows affected (0.25 sec) mysql> insert my_i values(111); ERROR 1146 (42S02): Table 'mcwtest.my_i' doesn't exist mysql> insert my_id values(111); Query OK, 1 row affected (0.05 sec) mysql> select * from my_id; +------+ | id | +------+ | 111 | +------+ 1 row in set (0.00 sec) mysql> \q Bye pod "mysql-mcwclient" deleted [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-dbffc69d-jhvvz 1/1 Running 0 27m 10.244.1.4 mcwk8s-node1 <none> <none> [machangwei@mcwk8s-master ~]$
关闭node1模拟故障
[root@mcwk8s-node1 ~]$ shutdown now Connection closed by foreign host. Disconnected from remote host(mcw05) at 20:19:00. Type `help' to learn how to use Xshell prompt. [c:\~]$
验证服务故障转移
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-dbffc69d-jhvvz 1/1 Running 0 38m 10.244.1.4 mcwk8s-node1 <none> <none> [machangwei@mcwk8s-master ~]$ kubectl describe pod mysql-dbffc69d-jhvvz ..... Warning NodeNotReady 111s node-controller Node is not ready Warning NodeNotReady 5m29s node-controller Node is not ready [machangwei@mcwk8s-master ~]$ 当发现节点5分钟左右没准备好的时候，就开始运行新的pod,并部署mysql服务的pod在可用节点上。 [machangwei@mcwk8s-master ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mysql-dbffc69d-ch2vg 1/1 Running 0 2m9s 10.244.2.18 mcwk8s-node2 <none> <none> mysql-dbffc69d-jhvvz 1/1 Terminating 0 43m 10.244.1.4 mcwk8s-node1 <none> <none> [machangwei@mcwk8s-master ~]$ #还是用之前的命令来连接。--后面应该是连接命令吧。-h指定集群服务ip,指定端口和密码 [machangwei@mcwk8s-master ~]$ kubectl get service #--rm是退出后就删的pod. NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d mysql ClusterIP 10.103.171.207 <none> 3306/TCP 79m [machangwei@mcwk8s-master ~]$ kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-mcwclient -- mysql -h 10.103.171.207 -P3306 -ppassword If you don't see a command prompt, try pressing enter. #当我们要进入MySQL命令行，需要按enter。不用一直等待，因为等着没用 mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mcwtest | | mysql | | performance_schema | +--------------------+ 4 rows in set (0.02 sec) mysql> use mcwtest; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +-------------------+ | Tables_in_mcwtest | +-------------------+ | my_id | +-------------------+ 1 row in set (0.01 sec) mysql> select * from my_id; +------+ | id | +------+ | 111 | +------+ 1 row in set (0.02 sec) mysql> \q Bye pod "mysql-mcwclient" deleted 到root下查看数据库的数据存放地址，也就是nfs服务端上。我们设置了即使删除pv,pvc也是不删除数据的。如上可知，当数据库所在主机down之后，那么过一段时间，会重新从其它节点上运行新的pod,旧的pod还存在，只是状态是终止。 [root@mcwk8s-master ~]$ [root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/ auto.cnf ibdata1 ib_logfile0 ib_logfile1 mcwtest mysql performance_schema [root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/mcwtest/ db.opt my_id.frm my_id.ibd [root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/mcwtest/ -lh total 112K -rw-rw----. 1 polkitd ssh_keys 65 Feb 19 20:08 db.opt -rw-rw----. 1 polkitd ssh_keys 8.4K Feb 19 20:09 my_id.frm -rw-rw----. 1 polkitd ssh_keys 96K Feb 19 20:09 my_id.ibd [root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/ -lh total 109M -rw-rw----. 1 polkitd ssh_keys 56 Feb 19 19:23 auto.cnf -rw-rw----. 1 polkitd ssh_keys 12M Feb 19 20:24 ibdata1 -rw-rw----. 1 polkitd ssh_keys 48M Feb 19 20:24 ib_logfile0 -rw-rw----. 1 polkitd ssh_keys 48M Feb 19 19:21 ib_logfile1 drwx------. 2 polkitd ssh_keys 54 Feb 19 20:09 mcwtest drwx------. 2 polkitd ssh_keys 4.0K Feb 19 19:25 mysql drwx------. 2 polkitd ssh_keys 4.0K Feb 19 19:21 performance_schema
删除资源
[machangwei@mcwk8s-master ~]$ kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE mysql 1/1 1 1 19h [machangwei@mcwk8s-master ~]$ ls mysql-pvc.yml mysql-pv.yml mysql.yml [machangwei@mcwk8s-master ~]$ kubectl delete -f mysql.yml service "mysql" deleted deployment.apps "mysql" deleted [machangwei@mcwk8s-master ~]$ kubectl delete -f mysql-pvc.yml persistentvolumeclaim "mysql-pvc" deleted [machangwei@mcwk8s-master ~]$ kubectl delete -f mysql-pv.yml persistentvolume "mysql-pv" deleted [machangwei@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/ auto.cnf ibdata1 ib_logfile0 ib_logfile1 mcwtest mysql performance_schema
参考学习书籍：每天5分钟玩转kubernetes ---cloudman
