存储卷简述
在k8s中部署的应用都是以pod容器的形式运行的，假如部署MySQL、Redis等数据库，需要对这些数据库产生的数据做备份。因为Pod是有生命周期的，如果pod不挂载数据卷，那pod被删除或重启后这些数据会随之消失，如果想要长久的保留这些数据就要用到pod数据持久化存储。
k8s提供的存储卷（Volume）属于pod资源级别，共享于pod内的所有容器，可用于在容器的文件系统之外存储应用程序的相关数据，甚至还可以独立于pod的生命周期之外实现数据持久化。
一、存储卷概述
k8s系统上的存储卷是定义在pod资源之上、可被其内部的所有容器挂载的共享目录，它关联至某外部的存储设备之上的存储空间，从而独立于容器自身的文件系统，而数据是否具有持久能力则取决于存储卷自身是否支持持久化。
1. 存储卷类型
查看k8s支持的存储类型，可以看到k8s支持多种存储卷类型，包括本地存储（节点）和网络存储系统中的诸多存储机制。
[root@k8s-master1 ~]# kubectl explain pod.spec.volumes
KIND: Pod
VERSION: v1
RESOURCE: volumes <[]Object>
DESCRIPTION:
List of volumes that can be mounted by containers belonging to the pod.
More info: https://kubernetes.io/docs/concepts/storage/volumes
Volume represents a named volume in a pod that may be accessed by any
container in the pod.
FIELDS:
awsElasticBlockStore <Object>
AWSElasticBlockStore represents an AWS Disk resource that is attached to a
kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
azureDisk <Object>
AzureDisk represents an Azure Data Disk mount on the host and bind mount to
the pod.
azureFile <Object>
AzureFile represents an Azure File Service mount on the host and bind mount
to the pod.
cephfs <Object>
CephFS represents a Ceph FS mount on the host that shares a pod's lifetime
cinder <Object>
Cinder represents a cinder volume attached and mounted on kubelets host
machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
configMap <Object>
ConfigMap represents a configMap that should populate this volume
csi <Object>
CSI (Container Storage Interface) represents ephemeral storage that is
handled by certain external CSI drivers (Beta feature).
downwardAPI <Object>
DownwardAPI represents downward API about the pod that should populate this
volume
emptyDir <Object>
EmptyDir represents a temporary directory that shares a pod's lifetime.
More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
ephemeral <Object>
Ephemeral represents a volume that is handled by a cluster storage driver
(Alpha feature). The volume's lifecycle is tied to the pod that defines it
- it will be created before the pod starts, and deleted when the pod is
removed.
Use this if: a) the volume is only needed while the pod runs, b) features
of normal volumes like restoring from snapshot or capacity tracking are
needed, c) the storage driver is specified through a storage class, and d)
the storage driver supports dynamic volume provisioning through a
PersistentVolumeClaim (see EphemeralVolumeSource for more information on
the connection between this volume type and PersistentVolumeClaim).
Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes
that persist for longer than the lifecycle of an individual pod.
Use CSI for light-weight local ephemeral volumes if the CSI driver is meant
to be used that way - see the documentation of the driver for more
information.
A pod can use both types of ephemeral volumes and persistent volumes at the
same time.
fc <Object>
FC represents a Fibre Channel resource that is attached to a kubelet's host
machine and then exposed to the pod.
flexVolume <Object>
FlexVolume represents a generic volume resource that is
provisioned/attached using an exec based plugin.
flocker <Object>
Flocker represents a Flocker volume attached to a kubelet's host machine.
This depends on the Flocker control service being running
gcePersistentDisk <Object>
GCEPersistentDisk represents a GCE Disk resource that is attached to a
kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
gitRepo <Object>
GitRepo represents a git repository at a particular revision. DEPRECATED:
GitRepo is deprecated. To provision a container with a git repo, mount an
EmptyDir into an InitContainer that clones the repo using git, then mount
the EmptyDir into the Pod's container.
glusterfs <Object>
Glusterfs represents a Glusterfs mount on the host that shares a pod's
lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath <Object>
HostPath represents a pre-existing file or directory on the host machine
that is directly exposed to the container. This is generally used for
system agents or other privileged things that are allowed to see the host
machine. Most containers will NOT need this. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi <Object>
ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
host machine and then exposed to the pod. More info:
https://examples.k8s.io/volumes/iscsi/README.md
name <string> -required-
Volume's name. Must be a DNS_LABEL and unique within the pod. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
nfs <Object>
NFS represents an NFS mount on the host that shares a pod's lifetime More
info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
persistentVolumeClaim <Object>
PersistentVolumeClaimVolumeSource represents a reference to a
PersistentVolumeClaim in the same namespace. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
photonPersistentDisk <Object>
PhotonPersistentDisk represents a PhotonController persistent disk attached
and mounted on kubelets host machine
portworxVolume <Object>
PortworxVolume represents a portworx volume attached and mounted on
kubelets host machine
projected <Object>
Items for all in one resources secrets, configmaps, and downward API
quobyte <Object>
Quobyte represents a Quobyte mount on the host that shares a pod's lifetime
rbd <Object>
RBD represents a Rados Block Device mount on the host that shares a pod's
lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
scaleIO <Object>
ScaleIO represents a ScaleIO persistent volume attached and mounted on
Kubernetes nodes.
secret <Object>
Secret represents a secret that should populate this volume. More info:
https://kubernetes.io/docs/concepts/storage/volumes#secret
storageos <Object>
StorageOS represents a StorageOS volume attached and mounted on Kubernetes
nodes.
vsphereVolume <Object>
VsphereVolume represents a vSphere volume attached and mounted on kubelets
host machine
常用的存储类型：emptyDir,hostPath,nfs,persistentVolumeClaim,glusterfs,cephfs,configMap,secret等。
其中，Secret用于向pod传递敏感信息，如：密码，私钥、证书文件等，用户可以将这些信息存储于集群中而后由pod进行挂载，从而实现将敏感信息与系统解耦。
ConfigMap资源则用于向pod注入非敏感数据，使用时，用户将数据直接存储于ConfigMap对象中，而后直接在pod中使用ConfigMap卷引用它即可，它可以帮助实现容器配置文件集中化定义和管理。
2. 存储卷的使用方式
在pod中定义使用存储卷的配置由两部分组成：一是通过.spec.volumes字段定义在pod之上的存储卷列表，其支持使用多种不同类型的存储卷且配置参数差别很大；另一个是通过.spec.containers.volumeMounts字段定义在容器上定义的存储卷挂载列表，它只能挂载当前pod资源中定义的具体存储卷，当然，也可以不挂载任何存储卷。
简单来说要使用存储卷，需要经历如下步骤：
1）定义pod的volume，这个volume指明它要关联到哪个存储上的
2）在容器中要使用volumemounts挂载对应的存储
在pod级别定义存储卷时，.spec.volumes字段的值是对象列表格式，每个对象为一个存储卷的定义。下面的资源清单片段中定义了由两个存储卷组成的卷列表，一个是emptyDir类型，一个是gitRepo类型：
spec:
...
volumes:
- name: logdata
emptyDir: {}
- name: example
gitRepo:
repository: https//github.com/iKubernetes/k8s_book.gitRepo
revision: master
directory: .
定义好的存储卷可由当前pod资源内的各容器进行挂载。事实上，也只有多个容器挂载同一个存储卷时，“共享”才有具体意义。当pod中只有一个容器时，使用存储卷的目的通常在于数据持久化。.spec.containers.volumeMounts字段的值也是对象列表格式，由一到多个存储卷挂载定义组成。无论何种类型的存储卷，它们的挂载格式基本上都是相同的。下面的代码段是在容器内定义挂载卷时通用的语法格式：
[root@k8s-master1 ~]# kubectl explain pod.spec.containers.volumeMounts
KIND: Pod
VERSION: v1
RESOURCE: volumeMounts <[]Object>
DESCRIPTION:
Pod volumes to mount into the container's filesystem. Cannot be updated.
VolumeMount describes a mounting of a Volume within a container.
FIELDS:
mountPath <string> -required- #挂载点路径，容器文件系统上的路径，必须字段
Path within the container at which the volume should be mounted. Must not
contain ':'.
mountPropagation <string>
mountPropagation determines how mounts are propagated from the host to
container and the other way around. When not set, MountPropagationNone is
used. This field is beta in 1.10.
name <string> -required- #指定要挂载的存储名称，必选字段
This must match the Name of a Volume.
readOnly <boolean> #是否挂载为只读卷
Mounted read-only if true, read-write otherwise (false or unspecified).
Defaults to false.
subPath <string> #挂载存储卷时使用的子路径，即在mountPath指定的路径下使用一个子路径作为其挂载点
Path within the volume from which the container's volume should be mounted.
Defaults to "" (volume's root).
subPathExpr <string>
Expanded path within the volume from which the container's volume should be
mounted. Behaves similarly to SubPath but environment variable references
$(VAR_NAME) are expanded using the container's environment. Defaults to ""
(volume's root). SubPathExpr and SubPath are mutually exclusive.
下面是一个挂载示例，容器myapp将logdata存储卷挂载于/var/log/myapp目录下，将example挂载到/webdata/example目录下：
spec:
containers:
- name: myapp
image: nginx:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: logdata
mountPath: /var/log/myapp/
- name: example
mountPath: /webdata/example/
二、临时存储卷
k8s支持存储卷类型中，emptyDir存储卷的生命周期与其所属的pod对象相同，它无法脱离pod对象的生命周期提供数据存储功能，因此emptyDir通常仅用于数据缓存或临时存储。不过，基于emptyDir构建的gitRepo存储卷可以在pod对象的生命周期起始时从相应的Git仓库中复制相应的数据文件到底层的emptyDir中，从而使得它具有了一定意义上的持久性。
1. emptyDir存储卷
emptyDir存储卷是pod对象生命周期中的一个临时目录，在pod启动时即被创建，而在pod对象被移除时会被一并删除。不具有持久能力的emptyDir存储卷只能用于某些特殊场景中，例如：同一个pod内的多个容器间文件的共享，或者作为容器数据的临时存储目录用于数据缓存系统等。
emptyDir存储卷则定义于.spec.volumes.emptyDir嵌套的字段中，可用字段主要包括两个，具体如下：
[root@k8s-master1 ~]# kubectl explain pod.spec.volumes.emptyDir
KIND: Pod
VERSION: v1
RESOURCE: emptyDir <Object>
DESCRIPTION:
EmptyDir represents a temporary directory that shares a pod's lifetime.
More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
Represents an empty directory for a pod. Empty directory volumes support
ownership management and SELinux relabeling.
FIELDS:
medium <string> #此目录所在的存储介质的类型，可取值为"default"或"Memory"，默认为default,表示使用节点的默认存储介质；"Memory"表示使用基于RAM的临时文件系统tmpfs,空间受限于内存，但性能非常好，通常用于为容器中的应用提供缓存空间。
What type of storage medium should back this directory. The default is ""
which means to use the node's default medium. Must be an empty string
(default) or Memory. More info:
https://kubernetes.io/docs/concepts/storage/volumes#emptydir
sizeLimit <string> #当前存储卷的空间限额，默认值为nil,表示不受限制；不过在medium字段值为Memory时建议务必定义此限额。
Total amount of local storage required for this EmptyDir volume. The size
limit is also applicable for memory medium. The maximum usage on memory
medium EmptyDir would be the minimum value between the SizeLimit specified
here and the sum of memory limits of all containers in a pod. The default
is nil which means that the limit is undefined. More info:
http://kubernetes.io/docs/user-guide/volumes#emptydir
下面是一个使用了emptyDir存储卷的简单示例：
[root@k8s-master1 pod]# vim emptydir-pod.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 pod]# cat emptydir-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-empty
spec:
containers:
- name: nginx
ports:
- containerPort: 80
image: nginx:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /usr/share/nginx/html
name: html
- name: pagegen
image: alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: html
mountPath: /html
command: ["/bin/sh", "-c"]
args:
- while true; do
echo $(hostname) $(date) >> /html/index.html;
sleep 10;
done
volumes:
- name: html
emptyDir: {}
[root@k8s-master1 pod]# kubectl apply -f emptydir-pod.yaml
pod/pod-empty created
[root@k8s-master1 pod]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-empty 2/2 Running 0 7s 10.244.36.76 k8s-node1 <none> <none>
上面示例中定义的存储卷名称为html,挂载于容器nginx的/usr/share/nginx/html目录，及容器pagen的/html目录。容器pagegen每隔10秒向存储卷上的index.html文件中追加一行信息，而容器nginx中的nginx进程则以其为站点主页。
查看pod的uid，
[root@k8s-master1 pod]# kubectl get pods pod-empty -o yaml | grep uid uid: 70687bdc-d439-4ae5-81b7-a47037e9bff6
登录k8s-node1节点上，查看存储位置，可以看到在/var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/html/ 目录下
[root@k8s-node1 ~]# tree /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6
/var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6
├── containers
│ ├── nginx
│ │ └── 85d0ba7e
│ └── pagegen
│ └── 87b4296f
├── etc-hosts
├── plugins
│ └── kubernetes.io~empty-dir
│ ├── html
│ │ └── ready
│ └── wrapped_default-token-5n29f
│ └── ready
└── volumes
├── kubernetes.io~empty-dir
│ └── html
│ └── index.html
└── kubernetes.io~secret
└── default-token-5n29f
├── ca.crt -> ..data/ca.crt
├── namespace -> ..data/namespace
└── token -> ..data/token
12 directories, 9 files
[root@k8s-node1 ~]# ls -lrt /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/
total 0
drwxrwxrwx 2 root root 24 Sep 18 23:31 html
[root@k8s-node1 ~]# ls -lrt /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/html
total 4
-rw-r--r-- 1 root root 1131 Sep 18 23:36 index.html
[root@k8s-node1 ~]# cat /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/html/index.html
pod-empty Sun Sep 18 15:31:26 UTC 2022
pod-empty Sun Sep 18 15:31:36 UTC 2022
pod-empty Sun Sep 18 15:31:46 UTC 2022
pod-empty Sun Sep 18 15:31:56 UTC 2022
pod-empty Sun Sep 18 15:32:06 UTC 2022
...
pod资源的详细信息中会显示存储卷的相关状态，包括其是否创建成功，相关类型及参数以及容器中的挂载状态等信息。如下面命令结果所示：
[root@k8s-master1 pod]# kubectl describe pods pod-empty
Name: pod-empty
Namespace: default
Priority: 0
Node: k8s-node1/10.0.0.132
Start Time: Sun, 18 Sep 2022 23:31:24 +0800
Labels: <none>
Annotations: cni.projectcalico.org/podIP: 10.244.36.76/32
cni.projectcalico.org/podIPs: 10.244.36.76/32
Status: Running
IP: 10.244.36.76
IPs:
IP: 10.244.36.76
Containers:
nginx:
Container ID: docker://250d903443ede6b8e6594b815cd1b815e079d149d91eefb7cd26cdf5737811ec
Image: nginx:latest
Image ID: docker-pullable://nginx@sha256:b95a99feebf7797479e0c5eb5ec0bdfa5d9f504bc94da550c2f58e839ea6914f
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 18 Sep 2022 23:31:26 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from html (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
pagegen:
Container ID: docker://c5e9b7de94315d7e194110c66e38dc7a15ded921fea7e9c3aeace9e3fb372828
Image: alpine
Image ID: docker-pullable://alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
Args:
while true; do echo $(hostname) $(date) >> /html/index.html; sleep 10; done
State: Running
Started: Sun, 18 Sep 2022 23:31:26 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/html from html (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
html:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m5s default-scheduler Successfully assigned default/pod-empty to k8s-node1
Normal Pulled 7m4s kubelet Container image "nginx:latest" already present on machine
Normal Created 7m4s kubelet Created container nginx
Normal Started 7m4s kubelet Started container nginx
Normal Pulled 7m4s kubelet Container image "alpine" already present on machine
Normal Created 7m4s kubelet Created container pagegen
Normal Started 7m4s kubelet Started container pagegen
而后，可以为其创建Service资源并进行访问测试，或者在集群中直接对pod的地址发起访问请求，以测试pod中的两个容器通过emptyDir卷共享数据的结果状态：
[root@k8s-master1 pod]# curl -kv 10.244.36.76 * About to connect() to 10.244.36.76 port 80 (#0) * Trying 10.244.36.76... * Connected to 10.244.36.76 (10.244.36.76) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 10.244.36.76 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.23.1 < Date: Sun, 18 Sep 2022 15:40:50 GMT < Content-Type: text/html < Content-Length: 2223 < Last-Modified: Sun, 18 Sep 2022 15:40:47 GMT < Connection: keep-alive < ETag: "63273bff-8af" < Accept-Ranges: bytes < pod-empty Sun Sep 18 15:31:26 UTC 2022 pod-empty Sun Sep 18 15:31:36 UTC 2022 pod-empty Sun Sep 18 15:31:46 UTC 2022 pod-empty Sun Sep 18 15:31:56 UTC 2022 pod-empty Sun Sep 18 15:32:06 UTC 2022 pod-empty Sun Sep 18 15:32:16 UTC 2022 pod-empty Sun Sep 18 15:32:26 UTC 2022 ...
作为边车（sidecar）的容器pagegen，其每隔10秒生成一行信息追加到存储卷上的index.html文件中，因此，通过主容器nginx的应用访问到的内容也会处于不停的变动中。
在pod所在的节点上的存储目录下 /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/html/的index.html文件也在不断的变动。若修改 /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/html/的index.html文件，那么测试访问的内容也将发生变化
[root@k8s-node1 ~]# echo "hello world" > /var/lib/kubelet/pods/70687bdc-d439-4ae5-81b7-a47037e9bff6/volumes/kubernetes.io~empty-dir/html/index.html #测试访问 [root@k8s-master1 pod]# curl -kv 10.244.36.76 * About to connect() to 10.244.36.76 port 80 (#0) * Trying 10.244.36.76... * Connected to 10.244.36.76 (10.244.36.76) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 10.244.36.76 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.23.1 < Date: Sun, 18 Sep 2022 15:47:05 GMT < Content-Type: text/html < Content-Length: 12 < Last-Modified: Sun, 18 Sep 2022 15:47:02 GMT < Connection: keep-alive < ETag: "63273d76-c" < Accept-Ranges: bytes < hello world * Connection #0 to host 10.244.36.76 left intact
若将pod-empty对象删除，pod所在节点的存储目录也将被删除
[root@k8s-master1 pod]# kubectl delete pods pod-empty pod "pod-empty" deleted [root@k8s-node1 ~]# ls -lrt /var/lib/kubelet/pods/ total 0 drwxr-x--- 5 root root 71 Aug 1 20:48 2eafe33f-ef87-453b-99f5-6793f27be642 drwxr-x--- 5 root root 71 Aug 1 21:12 7ab91121-203f-41f5-81c4-8801c1383fcf drwxr-x--- 5 root root 71 Sep 18 23:31 70687bdc-d439-4ae5-81b7-a47037e9bff6 [root@k8s-node1 ~]# ls -lrt /var/lib/kubelet/pods/ total 0 drwxr-x--- 5 root root 71 Aug 1 20:48 2eafe33f-ef87-453b-99f5-6793f27be642 drwxr-x--- 5 root root 71 Aug 1 21:12 7ab91121-203f-41f5-81c4-8801c1383fcf [root@k8s-node1 ~]#
emptyDir卷简单易用，但是仅能用于临时存储。
2. gitRepo存储卷
gitRepo存储卷可以看作是emptyDIr存储卷的一种实际应用，使用该存储卷的pod资源可以通过挂载目录访问指定的代码仓库中的数据。使用gitRepo存储卷的pod资源在创建时，会首先创建一个空目录并克隆一份指定的Git仓库中的数据值该目录，而后在创建容器并挂载该存储卷。
定义该gitRepo类型的存储卷时，其可嵌套使用的字段具体包含如下三个：
[root@k8s-master1 ~]# kubectl explain pod.spec.volumes.gitRepo
KIND: Pod
VERSION: v1
RESOURCE: gitRepo <Object>
DESCRIPTION:
GitRepo represents a git repository at a particular revision. DEPRECATED:
GitRepo is deprecated. To provision a container with a git repo, mount an
EmptyDir into an InitContainer that clones the repo using git, then mount
the EmptyDir into the Pod's container.
#使用gitRepo类型的存储卷的pod资源运行的工作节点上必须安装git程序，否则克隆仓库的操作将无从完成。另外，自Kubernetes 1.12起，gitRepo存储卷已经被废弃
Represents a volume that is populated with the contents of a git
repository. Git repo volumes do not support ownership management. Git repo
volumes support SELinux relabeling.
DEPRECATED: GitRepo is deprecated. To provision a container with a git
repo, mount an EmptyDir into an InitContainer that clones the repo using
git, then mount the EmptyDir into the Pod's container.
FIELDS:
directory <string> #目标目录名称，名称中不能包含".."字符；"."表示将仓库中的数据直接复制到卷目录中，否则，即为复制到卷目录中以用户指定的字符串为名称的子目录
Target directory name. Must not contain or start with '..'. If '.' is
supplied, the volume directory will be the git repository. Otherwise, if
specified, the volume will contain the git repository in the subdirectory
with the given name.
repository <string> -required- #Git仓库的URL，必选字段
Repository URL
revision <string> #特定revision的提交哈希码
Commit hash for the specified revision.
下面示例中的pod资源在创建时，首先会创建一个空目录，将指定的Git仓库https://github.com/cnych/presentation-gitlab-k8s.git 中的数据复制一份直接保存于此目录中，而后将此目录创建为存储卷html,最后由容器nginx将此存储卷挂载于/usr/share/nginx/html 目录上。
[root@k8s-master1 pod]# vim vol-gitrepo.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 pod]# cat vol-gitrepo.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-gitrepo
spec:
containers:
- name: nginx
ports:
- containerPort: 80
image: nginx:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /usr/share/nginx/html
name: html
volumes:
- name: html
gitRepo:
repository: https://github.com/cnych/presentation-gitlab-k8s.git
revision: "master"
[root@k8s-master1 pod]# kubectl apply -f vol-gitrepo.yaml
pod/pod-gitrepo created
[root@k8s-master1 pod]# kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-gitrepo 0/1 ContainerCreating 0 5s
[root@k8s-master1 pod]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-gitrepo 1/1 Running 0 4m6s 10.244.36.80 k8s-node1 <none> <none>
访问此pod资源中的nginx服务即可看到其来自于Git仓库中的页面资源。不过，gitRepo存储卷在其创建完成后不会再与指定的仓库指定的仓库执行同步操作，这就意味着在pod资源运行期间，如果仓库中的数据发生变化，那么gitRepo存储卷不会同步到这些内容。
可以登录到pod中查看目录中是否有克隆下仓库的数据
[root@k8s-master1 pod]# kubectl exec -it pod-gitrepo -- /bin/sh # ls /usr/share/nginx/html presentation-gitlab-k8s # ls /usr/share/nginx/html/presentation-gitlab-k8s Dockerfile Gopkg.lock Gopkg.toml LICENSE Makefile README.md VERSION main.go manifests vendor #
在pod所在的k8s-node1节点上查看存储位置中，可以看到克隆到仓库中的数据
[root@k8s-node1 ~]# ll /var/lib/kubelet/pods/58af7421-5454-4d03-9888-0414d2ce1bd1/volumes/kubernetes.io~git-repo/ total 0 drwxrwxrwx 3 root root 37 Sep 21 22:33 html [root@k8s-node1 ~]# ll /var/lib/kubelet/pods/58af7421-5454-4d03-9888-0414d2ce1bd1/volumes/kubernetes.io~git-repo/html/ total 0 drwxr-xr-x 5 root root 221 Sep 21 22:33 presentation-gitlab-k8s [root@k8s-node1 ~]# ll /var/lib/kubelet/pods/58af7421-5454-4d03-9888-0414d2ce1bd1/volumes/kubernetes.io~git-repo/html/presentation-gitlab-k8s/ total 32 -rw-r--r-- 1 root root 86 Sep 21 22:33 Dockerfile -rw-r--r-- 1 root root 2093 Sep 21 22:33 Gopkg.lock -rw-r--r-- 1 root root 843 Sep 21 22:33 Gopkg.toml -rw-r--r-- 1 root root 1072 Sep 21 22:33 LICENSE -rw-r--r-- 1 root root 2314 Sep 21 22:33 main.go -rw-r--r-- 1 root root 724 Sep 21 22:33 Makefile drwxr-xr-x 2 root root 86 Sep 21 22:33 manifests -rw-r--r-- 1 root root 3875 Sep 21 22:33 README.md drwxr-xr-x 4 root root 42 Sep 21 22:33 vendor -rw-r--r-- 1 root root 7 Sep 21 22:33 VERSION
gitRepo存储卷建构于emptyDir存储卷上，它的生命周期与隶属pod对象相同，因此使用时不建议在此类存储卷上保存由容器生成的重要数据。其次，自Kubernetes 1.12起，gitRepo存储卷已经被废弃，所以在之后的版本中如要使用它配置pod对象，建议借助初始化容器，将仓库中的数据复制到emptyDir存储卷上，并在主容器中使用此存储卷。
二、节点存储卷hostPath
hostPath类型的存储卷是指将工作节点上的某个文件系统目录或文件挂载于pod中的一种存储卷，它可独立于pod资源的生命周期，在pod被删除，这个存储卷还是存在的，不会被删除，所以只要同一个pod被调度到同一个节点上来，在pod被删除重新被调度到这个节点之后，对应的数据依然是存在的，因而具有持久性。但它是工作节点本地的存储空间，仅适用于特定情况下的存储卷使用需求，例如，将工作节点上的文件系统关联为pod的存储卷，从而使得容器访问节点文件系统上的数据。这一点在运行由管理任务的系统级pod资源需要访问节点上的文件尤为有用。
配置hostPath存储卷的嵌套字段共有两个：一个是用于指定工作节点上的目录路径的必须字段path，一个是指定存储卷的类型type，
[root@k8s-master1 pod]# kubectl explain pod.spec.volumes.hostPath
KIND: Pod
VERSION: v1
RESOURCE: hostPath <Object>
DESCRIPTION:
HostPath represents a pre-existing file or directory on the host machine
that is directly exposed to the container. This is generally used for
system agents or other privileged things that are allowed to see the host
machine. Most containers will NOT need this. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
Represents a host path mapped into a pod. Host path volumes do not support
ownership management or SELinux relabeling.
FIELDS:
path <string> -required- #工作节点上的目录路径
Path of the directory on the host. If the path is a symlink, it will follow
the link to the real path. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
type <string> #存储卷类型
Type for HostPath Volume Defaults to "" More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
支持使用的卷类型包含以下几种：
DirectoryOrCreate 如果给定路径不存在任何内容，则会根据需要在此处创建一个空目录，权限设置为 0755，与 Kubelet 具有相同的组和所有权 Directory 目录必须存在于给定路径 FileOrCreate 如果给定路径不存在任何内容，则会根据需要在此处创建一个空文件，权限设置为 0644，与 Kubelet 具有相同的组和所有权。 File 文件必须存在于给定路径 Socket 给定路径中必须存在 UNIX 套接字 CharDevice 字符设备必须存在于给定路径中 BlockDevice 块设备必须存在于给定路径
下面为hostPath 配置示例
[root@k8s-master1 pod]# vim hostPath-pod.yaml
[root@k8s-master1 pod]# cat hostPath-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-hostpath
spec:
containers:
- name: nginx
ports:
- containerPort: 80
image: nginx:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /usr/share/nginx/html
name: html
- name: tomcat
image: tomcat:8.5-jre8-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: html
mountPath: /html
volumes:
- name: html
hostPath:
path: /data1
type: DirectoryOrCreate #表示本地有/data1目录，就用本地的，本地没有的，就会在pod调度的节点上自动创建该目录
[root@k8s-master1 pod]# kubectl apply -f hostPath-pod.yaml
pod/pod-hostpath created
[root@k8s-master1 pod]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-hostpath 2/2 Running 0 6s 10.244.36.78 k8s-node1 <none> <none>
由上面可以知道pod调度到了k8s-node1上，登录到k8s-node1机器，查看是否在这台机器创建了存储目录
[root@k8s-node1 ~]# ll /data1 total 0 [root@k8s-node1 ~]#
上面可以看到已经创建了存储目录/data1，这个/data1会作为pod的持久化存储目录
在k8s-node1上的/data1下创建一个文件index.html
[root@k8s-node1 ~]# cd /data1 [root@k8s-node1 data1]# echo "hello world" >> index.html [root@k8s-node1 data1]# cat index.html hello world [root@k8s-node1 data1]# ll total 4 -rw-r--r-- 1 root root 12 Sep 22 21:54 index.html
测试存储卷是否可以正常使用,登录到nginx容器
[root@k8s-master1 pod]# kubectl exec -it pod-hostpath -c nginx -- /bin/sh # ls /usr/share/nginx/html index.html # cat /usr/share/nginx/html/index.html hello world #
测试nginx的访问页面可以正常访问：
[root@k8s-master1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-hostpath 2/2 Running 0 8m35s 10.244.36.78 k8s-node1 <none> <none> [root@k8s-master1 ~]# curl -kv http://10.244.36.78:80 * About to connect() to 10.244.36.78 port 80 (#0) * Trying 10.244.36.78... * Connected to 10.244.36.78 (10.244.36.78) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 10.244.36.78 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.23.1 < Date: Thu, 22 Sep 2022 13:58:57 GMT < Content-Type: text/html < Content-Length: 12 < Last-Modified: Thu, 22 Sep 2022 13:54:43 GMT < Connection: keep-alive < ETag: "632c6923-c" < Accept-Ranges: bytes < hello world * Connection #0 to host 10.244.36.78 left intact
同时我们也登录到tomcat容器，查看存储卷是否可以正常使用
[root@k8s-master1 pod]# kubectl exec -it pod-hostpath -c tomcat -- /bin/sh /usr/local/tomcat # ls /html index.html /usr/local/tomcat # cat /html/index.html hello world
通过上面测试可以看到，同一个pod里的nginx和tomcat这两个容器是共享存储卷的。
hostpath存储卷缺点：单节点。即pod删除之后重新创建必须调度到同一个node节点，数据才不会丢失。可以用分布式存储：nfs，cephfs，glusterfs
另外，在节点中创建的文件或目录默认仅有root可写，若期望容器内的进程拥有写权限，则要么将它运行为特权容器，要么修改该节点上的目录路劲的权限。
三、NFS存储卷
NFS即网络文件系统，它是一种分布式文件系统协议，其功能旨在允许客户端主机可以像访问本地存储一样通过网络访问服务器端文件。
1. 搭建NFS服务
1）安装nfs插件
[root@k8s-master1 ~]# yum install nfs-utils -y
2）在宿主机创建NFS需要的共享目录
[root@k8s-master1 ~]# mkdir -pv /data/volumes
3）配置nfs共享服务器上的/data/volumes目录
[root@k8s-master1 ~]# vim /etc/exports [root@k8s-master1 ~]# cat /etc/exports /data/volumes 10.0.0.131/24(rw,no_root_squash) [root@k8s-master1 ~]#
其中，no_root_squash: 用户具有根目录的完全管理访问权限
4）使NFS配置生效
[root@k8s-master1 ~]# exportfs -arv exporting 10.0.0.131/24:/data/volumes
5）设置成开机自启动
[root@k8s-master1 ~]# systemctl start nfs You have new mail in /var/spool/mail/root [root@k8s-master1 ~]# systemctl enable nfs Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
6）查看nfs是否启动成功
[root@k8s-master1 ~]# systemctl status nfs
● nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: active (exited) since Sat 2022-09-24 10:37:33 CST; 41s ago
Main PID: 27955 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/nfs-server.service
Sep 24 10:37:33 k8s-master1 systemd[1]: Starting NFS server and services...
Sep 24 10:37:33 k8s-master1 systemd[1]: Started NFS server and services.
[root@k8s-master1 ~]# netstat -lntup |grep 111
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 563/rpcbind
tcp6 0 0 :::111 :::* LISTEN 563/rpcbind
udp 0 0 0.0.0.0:111 0.0.0.0:* 563/rpcbind
udp6 0 0 :::111 :::* 563/rpcbind
[root@k8s-master1 ~]#
可以看到nfs是active，说明nfs正常启动了
7）测试手动挂载是否正常，可以看到可以正常挂载
[root@k8s-node1 ~]# mkdir /test [root@k8s-node1 ~]# mount 10.0.0.131:/data/volumes /test/ [root@k8s-node1 ~]# df -hT |grep /test 10.0.0.131:/data/volumes nfs4 40G 5.9G 34G 15% /test [root@k8s-node1 ~]#
手动卸载命令如下：
[root@k8s-node1 ~]# umount /test [root@k8s-node1 ~]# df -hT |grep /test [root@k8s-node1 ~]#
2. NFS存储卷挂载
kubernetes的NFS存储卷用于将某事先存在的NFS服务器上导出的存储空间挂载到pod中以供容器使用。与emptyDir不同的是，NFS存储卷在pod对象终止后仅是被卸载而非删除。另外，NFS是文件系统级共享服务，它支持同时存在的多路挂载请求。定义NFS存储卷时，用到以下字段：
[root@k8s-master1 ~]# kubectl explain pod.spec.volumes.nfs
KIND: Pod
VERSION: v1
RESOURCE: nfs <Object>
DESCRIPTION:
NFS represents an NFS mount on the host that shares a pod's lifetime More
info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
Represents an NFS mount that lasts the lifetime of a pod. NFS volumes do
not support ownership management or SELinux relabeling.
FIELDS:
path <string> -required- #NFS服务器导出（共享）的文件系统路径，必选字段
Path that is exported by the NFS server. More info:
https://kubernetes.io/docs/concepts/storage/volumes#nfs
readOnly <boolean> #是否以只读方式挂载，默认是false
ReadOnly here will force the NFS export to be mounted with read-only
permissions. Defaults to false. More info:
https://kubernetes.io/docs/concepts/storage/volumes#nfs
server <string> -required- #NFS服务器IP地址或主机名，必选字段
Server is the hostname or IP address of the NFS server. More info:
https://kubernetes.io/docs/concepts/storage/volumes#nfs
Redis是一个著名的高性能key-value存储系统，应用非常广泛，将其部署运行于kubernetes系统之上时，需要持久化存储卷的支持。下面是简单使用redis的一个示例：
[root@k8s-master1 pod]# vim nfs-pod.yaml
[root@k8s-master1 pod]# cat nfs-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-nfs
labels:
app: redis
spec:
containers:
- name: redis
ports:
- containerPort: 6379
image: redis:4-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /data
name: redisdata
volumes:
- name: redisdata
nfs:
path: /data/volumes
server: 10.0.0.131
上面示例定义在资源配置清单文件nfs-pod.yaml中，其中pod资源拥有一个关联至NFS服务器：10.0.0.131的存储卷，redis容器将其挂载于/data目录上，它是运行于容器中的redis-server数据的持久保存位置。
使用下列命令创建资源，并查看资源的详细信息
[root@k8s-master1 pod]# kubectl apply -f nfs-pod.yaml
pod/pod-nfs created
You have new mail in /var/spool/mail/root
[root@k8s-master1 pod]# kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-nfs 1/1 Running 0 5s
[root@k8s-master1 pod]# kubectl describe pod pod-nfs
Name: pod-nfs
Namespace: default
Priority: 0
Node: k8s-node1/10.0.0.132
Start Time: Sat, 24 Sep 2022 11:15:28 +0800
Labels: app=redis
Annotations: cni.projectcalico.org/podIP: 10.244.36.74/32
cni.projectcalico.org/podIPs: 10.244.36.74/32
Status: Running
IP: 10.244.36.74
IPs:
IP: 10.244.36.74
Containers:
redis:
Container ID: docker://ce6445f678dee398b2d0cb321ea8a24097cc680b0fad05d290e4fe2e9c4e1ad8
Image: redis:4-alpine
Image ID: docker-pullable://redis@sha256:aaf7c123077a5e45ab2328b5ef7e201b5720616efac498d55e65a7afbb96ae20
Port: 6379/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 24 Sep 2022 11:15:29 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/data from redisdata (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
redisdata:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 10.0.0.131
Path: /data/volumes
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19s default-scheduler Successfully assigned default/pod-nfs to k8s-node1
Normal Pulled 18s kubelet Container image "redis:4-alpine" already present on machine
Normal Created 18s kubelet Created container redis
Normal Started 18s kubelet Started container redis
[root@k8s-master1 pod]#
资源创建完成之后，可通过其命令客户端redis-cli创建测试数据，并手动触发其同步与存储系统中
[root@k8s-master1 pod]# kubectl exec -it pod-nfs -- redis-cli 127.0.0.1:6379> set mykey "hello nfs" OK 127.0.0.1:6379> get mykey "hello nfs" 127.0.0.1:6379> BGSAVE Background saving started 127.0.0.1:6379> exit
可以看到在nfs服务器共享目录中已有相应的数据，说明挂载nfs存储卷成功。
[root@k8s-master1 ~]# ll /data/volumes/ total 4 -rw-r--r-- 1 polkitd 1000 115 Sep 24 11:19 dump.rdb [root@k8s-master1 ~]#
为了测试其数据持久化效果，下面删除pod资源nfs-pod，并于再次重建后检测数据是否依然能够访问
[root@k8s-master1 pod]# kubectl delete pods pod-nfs pod "pod-nfs" deleted [root@k8s-master1 pod]# kubectl apply -f nfs-pod.yaml pod/pod-nfs created [root@k8s-master1 pod]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-nfs 1/1 Running 0 7s 10.244.36.85 k8s-node1 <none> <none>
待其重建完成后，通过再一次创建的pod资源详细信息，可以观察到它挂载使用NFS存储卷的相关信息。接下来再次检查redis-server中是否还保存有此前存储的数据:
[root@k8s-master1 pod]# kubectl describe pods pod-nfs
Name: pod-nfs
Namespace: default
Priority: 0
Node: k8s-node1/10.0.0.132
Start Time: Sat, 24 Sep 2022 11:24:22 +0800
Labels: app=redis
Annotations: cni.projectcalico.org/podIP: 10.244.36.85/32
cni.projectcalico.org/podIPs: 10.244.36.85/32
Status: Running
IP: 10.244.36.85
IPs:
IP: 10.244.36.85
Containers:
redis:
Container ID: docker://6b0b4374818686c7bea5fc64cdcc969ae493928bcea7e18e4a3c9cfe28abca89
Image: redis:4-alpine
Image ID: docker-pullable://redis@sha256:aaf7c123077a5e45ab2328b5ef7e201b5720616efac498d55e65a7afbb96ae20
Port: 6379/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 24 Sep 2022 11:24:24 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/data from redisdata (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
redisdata:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 10.0.0.131
Path: /data/volumes
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m25s default-scheduler Successfully assigned default/pod-nfs to k8s-node1
Normal Pulled 2m24s kubelet Container image "redis:4-alpine" already present on machine
Normal Created 2m24s kubelet Created container redis
Normal Started 2m23s kubelet Started container redis
[root@k8s-master1 pod]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-nfs 1/1 Running 0 2m48s 10.244.36.85 k8s-node1 <none> <none>
[root@k8s-master1 pod]# kubectl exec -it pod-nfs -- redis-cli
127.0.0.1:6379> get mykey
"hello nfs"
127.0.0.1:6379>
从上面的命令结果可以看出，此前创建的键mykey及其数据再pod资源重建后依然存在，这表明在删除pod资源时，其关联的外部存储卷并不会被一同删除。如果需要清除此类的数据，需要用户通过存储系统的管理接口手动进行。如果nfs宕机了，数据也就丢失了，所以需要使用分布式存储，常见的分布式存储有glusterfs和cephfs。
