Kubernetes进阶实战读书笔记:存储卷概述
一、Kubernetes 支持的存储类型
1、从官方手册查看
[root@master ~]# kubectl explain pod.spec.volumes KIND: Pod VERSION: v1 RESOURCE: volumes <[]Object> DESCRIPTION: List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes Volume represents a named volume in a pod that may be accessed by any container in the pod. FIELDS: awsElasticBlockStore <Object> AWSElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore azureDisk <Object> AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod. azureFile <Object> AzureFile represents an Azure File Service mount on the host and bind mount to the pod. cephfs <Object> CephFS represents a Ceph FS mount on the host that shares a pod's lifetime cinder <Object> Cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md configMap <Object> ConfigMap represents a configMap that should populate this volume csi <Object> CSI (Container Storage Interface) represents storage that is handled by an external CSI driver (Alpha feature). downwardAPI <Object> DownwardAPI represents downward API about the pod that should populate this volume emptyDir <Object> EmptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir fc <Object> FC represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod. flexVolume <Object> FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. flocker <Object> Flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running gcePersistentDisk <Object> GCEPersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk gitRepo <Object> GitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. glusterfs <Object> Glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md hostPath <Object> HostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath iscsi <Object> ISCSI represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md name <string> -required- Volume's name. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names nfs <Object> NFS represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs persistentVolumeClaim <Object> PersistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims photonPersistentDisk <Object> PhotonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine portworxVolume <Object> PortworxVolume represents a portworx volume attached and mounted on kubelets host machine projected <Object> Items for all in one resources secrets, configmaps, and downward API quobyte <Object> Quobyte represents a Quobyte mount on the host that shares a pod's lifetime rbd <Object> RBD represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md scaleIO <Object> ScaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes. secret <Object> Secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret storageos <Object> StorageOS represents a StorageOS volume attached and mounted on Kubernetes nodes. vsphereVolume <Object> VsphereVolume represents a vSphere volume attached and mounted on kubelets host machine
2、重点关注
二、存储卷的使用方式
1、存储卷的配置组成
pod.spec.volumes字段定义在pod之上的存储卷列表、其支持使用多种不同类型的存储卷且配置参数差别很大
pod.spec.containers.volumeMounts字段在容器上定义的存储卷挂在列表、它只能挂在当前pod资源中定义的具体存储卷、当然、也可以不挂在任何存储卷
2、示例文档
apiVersion: v1 kind: Pod metadata: name: vol-emptydir-pod spec: volumes: - name: html emptyDir: {} containers: - name: nginx image: nginx:1.12-alpine volumeMounts: - name: html mountPath: /usr/share/nginx/html - name: pagegen image: alpine volumeMounts: - name: html mountPath: /html command: ["/bin/sh", "-c"] args: - while true; do echo $(hostname) $(date) >> /html/index.html; sleep 10; done
事实上、也只有多个容器挂在同一个存储卷时"共享"才有具体的意义
2、重要字段详解
[root@master ~]# kubectl explain pod.spec.containers.volumeMounts KIND: Pod VERSION: v1 RESOURCE: volumeMounts <[]Object> DESCRIPTION: Pod volumes to mount into the container's filesystem. Cannot be updated. VolumeMount describes a mounting of a Volume within a container. FIELDS: mountPath <string> -required- #挂载点路径、容器文件系统上的路径、必选字段 Path within the container at which the volume should be mounted. Must not contain ':'. mountPropagation <string> mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. name <string> -required- #指定要挂在的存储的名称、必选字段 This must match the Name of a Volume. readOnly <boolean> #是否挂在只读卷 Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. subPath <string> #挂在存储卷时使用的子路径、即mountPath指定的路径下使用一个子路径作为挂载点 Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). subPathExpr <string> Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive.
三、临时存储卷
1、应用场景
同一pod内的多个容器间问价你的共享、或者作为容器数据的临时存储目录用户数据缓存系统等
2、字段属性详解
[root@master ~]# kubectl explain pod.spec.volumes.emptyDir KIND: Pod VERSION: v1 RESOURCE: emptyDir <Object> DESCRIPTION: EmptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir Represents an empty directory for a pod. Empty directory volumes support ownership management and SELinux relabeling. FIELDS: medium <string> #medium:此目录所在的存储介质的类型、可取值为"default" 或 "Memory"、默认为default、表示使用节点的默认存储介质 "Memory"表示使用基于RAM的临时文件系统tmpfs、空间受限于内存、但性能非常好、通常用于容器中的应用提供缓存空间 What type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir sizeLimit <string> #当前存储卷的空间限额、默认值为nil、表示不限制;不过、在medium字段值为"Memory"时建议务必定义此限额 Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir
3、资源清单
[root@master chapter7]# cat vol-emptydir.yaml apiVersion: v1 kind: Pod metadata: name: vol-emptydir-pod spec: volumes: - name: html emptyDir: {} containers: - name: nginx image: nginx:1.12-alpine volumeMounts: - name: html mountPath: /usr/share/nginx/html - name: pagegen image: alpine volumeMounts: - name: html mountPath: /html command: ["/bin/sh", "-c"] args: - while true; do echo $(hostname) $(date) >> /html/index.html; sleep 10; done
4、创建运行
[root@master chapter7]# kubectl apply -f vol-emptydir.yaml pod/vol-emptydir-pod created
5、查看验证
[root@master chapter7]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES vol-emptydir-pod 2/2 Running 0 14m 10.244.2.68 node2 <none> <none> [root@master chapter7]# culr 10.244.2.68 -bash: culr: command not found [root@master chapter7]# curl 10.244.2.68 vol-emptydir-pod Mon Aug 10 07:58:35 UTC 2020 vol-emptydir-pod Mon Aug 10 07:58:45 UTC 2020 vol-emptydir-pod Mon Aug 10 07:58:55 UTC 2020 ...... vol-emptydir-pod Mon Aug 10 08:17:56 UTC 2020 vol-emptydir-pod Mon Aug 10 08:18:06 UTC 2020 vol-emptydir-pod Mon Aug 10 08:18:16 UTC 2020
6、查看详细信息
pod 资源的详细信息中会显示存储卷的相关存储的相关状态、包裹其创建成功、相关的类型及参数、以及容器中的挂载状态等信息
[root@master chapter7]# kubectl describe pods vol-emptydir-pod Name: vol-emptydir-pod Namespace: default Priority: 0 Node: node2/192.168.118.20 ...... Containers: nginx: Container ID: docker://b7e816b54cb4145f47d9bb8f4375963d533966df2d3da682b3a58a30f90c19f7 ...... Mounts: /usr/share/nginx/html from html (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-pwl2t (ro) pagegen: ...... Command: /bin/sh -c Args: while true; do echo $(hostname) $(date) >> /html/index.html; sleep 10; done State: Running Started: Mon, 10 Aug 2020 15:58:35 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /html from html (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-pwl2t (ro) Conditions: ...... Volumes: html: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> default-token-pwl2t: Type: Secret (a volume populated by a Secret) SecretName: default-token-pwl2t Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 8m27s kubelet, node2 Pulling image "nginx:1.12-alpine" Normal Scheduled 8m26s default-scheduler Successfully assigned default/vol-emptydir-pod to node2 Normal Pulled 8m10s kubelet, node2 Successfully pulled image "nginx:1.12-alpine" Normal Created 8m10s kubelet, node2 Created container nginx Normal Started 8m9s kubelet, node2 Started container nginx Normal Pulling 8m9s kubelet, node2 Pulling image "alpine" Normal Pulled 7m39s kubelet, node2 Successfully pulled image "alpine" Normal Created 7m39s kubelet, node2 Created container pagegen Normal Started 7m39s kubelet, node2 Started container pagegen
7、emptyDir存储卷也可以基于RAM创建tmpfs文件系统的存储卷、常用于为容器的应用提供高性能缓存
volumes: - name: cache emptyDir: medium: Memory
emptyDir卷简单易用、但技能用于临时存储、另外还存在一些类型的存储卷建构在emptyDir之上、并额外提供了它所没有的功能例如gitRepo
四、临时存储卷(gitRepo)
基于emptyDir构建的gitRepo存储卷可以在pod对象的声明周期其实时从相应的GIT仓库中复制相应的数据文件到底层的 中、从而使得它具有一定意义上的持久性
gitRepo可以看做是emptyDir存储卷的一种实际应用、使用该存储卷的pod资源可以通过挂在目录访问指定的代码仓库中的数据
使用gitRepo存储卷的pod资源在创建时、会首选创建一个空目录并克隆一份指定的git仓库中的数据至该目录、而后再创建容器并挂在存储卷
1、字段详解
[root@master ~]# kubectl explain pod.spec.volumes.gitRepo KIND: Pod VERSION: v1 RESOURCE: gitRepo <Object> DESCRIPTION: GitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. Represents a volume that is populated with the contents of a git repository. Git repo volumes do not support ownership management. Git repo volumes support SELinux relabeling. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. FIELDS: directory <string> #目标目录名称,名称中不包含 ".." 字符 "."表示将仓库中的数据直接复制到卷目录中、否则、即为复制到卷目录中以用户指定的字符喜欢为名称的子目录中 Target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name. repository <string> -required- #git仓库的URL、必选字段 Repository URL revision <string> #特定revision的提交哈希码 Commit hash for the specified revision
2、资源清单
[root@master chapter7]# cat vol-gitrepo.yaml apiVersion: v1 kind: Pod metadata: name: vol-gitrepo spec: containers: - name: nginx image: nginx:1.12-alpine volumeMounts: - name: html mountPath: /usr/share/nginx/html volumes: - name: html gitRepo: repository: https://github.com/iKubernetes/k8s_book.git directory: . revision: "master"
3、创建运行
[root@master chapter7]# kubectl apply -f vol-gitrepo.yaml pod/vol-gitrepo created
使用gitRepo存储卷的pod资源运行的工作节点上必须安装有git程序,否则克隆仓库的操作将无从完成
4、效果验证
[root@master chapter7]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-example 1/1 Running 1 4d23h 10.244.2.9 node2 <none> <none> vol-emptydir-pod 2/2 Running 0 64m 10.244.2.68 node2 <none> <none> vol-gitrepo 0/1 ContainerCreating 0 78s <none> node1 <none> <none> [root@master chapter7]# kubectl describe pod vol-gitrepo Name: vol-gitrepo Namespace: default Priority: 0 Node: node1/192.168.118.19 Start Time: Mon, 10 Aug 2020 17:00:31 +0800 Labels: <none> Annotations: Status: Pending IP: IPs: <none> Containers: nginx: Container ID: Image: nginx:1.12-alpine Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /usr/share/nginx/html from html (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-pwl2t (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: html: Type: GitRepo (a volume that is pulled from git when the pod is created) Repository: https://github.com/iKubernetes/k8s_book.git Revision: master default-token-pwl2t: Type: Secret (a volume populated by a Secret) SecretName: default-token-pwl2t Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 94s default-scheduler Successfully assigned default/vol-gitrepo to node1 Warning FailedMount 25s (x7 over 88s) kubelet, node1 MountVolume.SetUp failed for volume "html" : failed to exec 'git clone -- https://github.com/iKubernetes/k8s_book.git .': Cloning into '.'... fatal: could not read Username for 'https://github.com': No such device or address : exit status 128
kubernetes 1.12版本起、gitRepo存储卷已经被废弃、所以在之后的版本中若要使用它配置pod对象、建议读者借助初始化容器将仓库中的数据复制到emptyDir存储卷上,并在主容器中使用此存储卷
五、临时存储卷(hostPath)
hostPath类型的存储卷时指将工作节点上某个文件系统的目录或文件挂在于pod中的一种存储卷、它可以独立于pod资源的生命周期、因而具有持久性
但它是工作节点本地的存储空间、仅使用于特定情况下的存储卷使用需要
例如:将工作节点陕干的文件系统关联为pod的存储卷、从而使得容器访问节点文件系统上的数据、这一点在运行由管理任务的系统级pod资源需要访问节点上的文件时尤为有用
1、字段详解
DirectoryOrCreate #指定的路径不存在时定将其创建为0755的空目录。属主属组均为Kubelet If nothing exists at the given path, an empty directory will be created there as needed with permission set to 0755, having the same group and ownership with Kubelet. Directory:A directory must exist at the given path #必须存在的目录路径 FileOrCreate: #指定的路径不存在时自动创建其创建权限为0644的空文件、属主和属组同时Kubelet If nothing exists at the given path, an empty file will be created there as needed with permission set to 0644, having the same group and ownership with Kubelet. File:A file must exist at the given path #必须存在的文件路径 Socket:A UNIX socket must exist at the given path #必须存在的socket文件路径 CharDevic:A character device must exist at the given path #必须存在的字符设备文件路径 BlockDevice:A block device must exist at the given path #不需存在的块设备文件路径
2、redis环境
1、资源清单
[root@master chapter5]# cat redis.yaml apiVersion: apps/v1 kind: Deployment metadata: name: redis labels: app: default spec: replicas: 1 selector: matchLabels: app: redis role: logstor template: metadata: labels: app: redis role: logstor spec: containers: - name: redis image: redis:4.0-alpine ports: - name: redis containerPort: 63
2、创建运行
[root@master chapter5]# kubectl apply -f redis.yaml deployment.apps/redis created [root@master chapter7]# kubectl expose deployment redis --port=6379 service/redis exposed
3、验证效果
[root@master chapter7]# kubectl expose deployment redis --port=6379 service/redis exposed [root@master chapter7]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d18h redis ClusterIP 10.106.30.154 <none> 6379/TCP 51s [root@master chapter5]# kubectl get all -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/redis-588694bf8c-s6tls 1/1 Running 0 54s 10.244.1.42 node1 <none> <none> [root@master chapter7]# kubectl exec -it pod/redis-588694bf8c-s6tls -- /bin/sh /data # nslookup redis.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10:53 Name: redis.default.svc.cluster.local Address: 10.106.30.154 /data # ls dump.rdb /data # redis-cli -h redis.default.svc.cluster.local redis.default.svc.cluster.local:6379> keys * (empty list or set)
3、hostPath环境
1、资源清单
[root@master chapter7]# cat vol-hostpath.yaml apiVersion: v1 kind: Pod metadata: name: vol-hostpath-pod spec: containers: - name: filebeat image: ikubernetes/filebeat:5.6.7-alpine env: - name: REDIS_HOST value: redis.default.svc.cluster.local - name: LOG_LEVEL value: info volumeMounts: - name: varlog mountPath: /var/log - name: socket mountPath: /var/run/docker.sock - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: socket hostPath: path: /var/run/docker.sock
2、创建运行
[root@master chapter7]# kubectl apply -f vol-hostpath.yaml pod/vol-hostpath-pod created [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE redis-588694bf8c-s6tls 1/1 Running 0 16h vol-hostpath-pod 1/1 Running 0 15h
3、验证效果
[root@master ~]# kubectl exec -it vol-hostpath-pod -- /bin/sh / # ps aux PID USER TIME COMMAND 1 root 0:19 /usr/local/bin/filebeat -e -c /etc/filebeat/filebeat.yml 14 root 0:00 /bin/sh 20 root 0:00 ps aux / # cat /etc/filebeat/filebeat.yml filebeat.registry_file: /var/log/containers/filebeat_registry filebeat.idle_timeout: 5s filebeat.spool_size: 2048 logging.level: info filebeat.prospectors: - input_type: log paths: - "/var/log/containers/*.log" - "/var/log/docker/containers/*.log" - "/var/log/startupscript.log" - "/var/log/kubelet.log" - "/var/log/kube-proxy.log" - "/var/log/kube-apiserver.log" - "/var/log/kube-controller-manager.log" - "/var/log/kube-scheduler.log" - "/var/log/rescheduler.log" - "/var/log/glbc.log" - "/var/log/cluster-autoscaler.log" symlinks: true json.message_key: log json.keys_under_root: true json.add_error_key: true multiline.pattern: '^\s' multiline.match: after document_type: kube-logs tail_files: true fields_under_root: true output.redis: hosts: ${REDIS_HOST:?No Redis host configured. Use env var REDIS_HOST to set host.} key: "filebeat" / # nslookup redis.default.svc.cluster.local nslookup: can't resolve '(null)': Name does not resolve Name: redis.default.svc.cluster.local Address 1: 10.106.30.154 redis.default.svc.cluster.local