12 容器存储扩容（转载）
目录
容器存储扩容
云原生存储扩容
容器的容量的不足的时候需要对容器的存储空间进行扩容， Rook 默认提供了两种驱动：
RBDCephFS
这两种驱动通过 StorageClass 存储供给者为容器提供存储空间，同时已经自动扩容的能力，其流程为：
客户端通过 PVC 声明的方式向 StorageClass 申请扩容容量空间，通过驱动调整 PV 的容量，调整底层的 RBD 像块容量，最终达到容量的扩展，如下是操作过程：
查看 PVC 大小
[root@m1 monitoring]# cd ../../
[root@m1 kubernetes]# kubectl apply -f mysql.yaml
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 20Gi RWO rook-ceph-block 14s
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
wordpress-mysql-6965fc8cc8-zgvcx 1/1 Running 0 16s
调整 PVC 声明容量空间为 30G
[root@m1 kubernetes]# vim mysql.yaml
......
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi # 调整大小为 30 G
......
更新资源清单
[root@m1 kubernetes]# kubectl apply -f mysql.yaml
service/wordpress-mysql unchanged
persistentvolumeclaim/mysql-pv-claim configured
deployment.apps/wordpress-mysql unchanged
查看 PVC 和 PV 的大小，可以看到 PVC 和 PV 容量已经扩容到 30G
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 30Gi RWO rook-ceph-block 2m53s
[root@m1 kubernetes]# kubectl get pv pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 30Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 4m2s
查看 PV 对应 RBD 块容量
[root@m1 kubernetes]# kubectl get pv pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
......
name: pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 30Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: mysql-pv-claim
namespace: default
resourceVersion: "1283651"
uid: cfe4a32c-c746-46af-84e8-cd981cdf8706
csi:
controllerExpandSecretRef:
name: rook-csi-rbd-provisioner
namespace: rook-ceph
driver: rook-ceph.rbd.csi.ceph.com
fsType: ext4
nodeStageSecretRef:
name: rook-csi-rbd-node
namespace: rook-ceph
volumeAttributes:
clusterID: rook-ceph
imageFeatures: layering
imageFormat: "2"
imageName: csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4 # PV 对应的 RBD 块
journalPool: replicapool # RBD 块所在的 pool
pool: replicapool
radosNamespace: ""
storage.kubernetes.io/csiProvisionerIdentity: 1669523375802-8081-rook-ceph.rbd.csi.ceph.com
volumeHandle: 0001-0009-rook-ceph-0000000000000002-197480ab-6e0f-11ed-b5c4-6efb82c232c4
persistentVolumeReclaimPolicy: Delete
storageClassName: rook-ceph-block
volumeMode: Filesystem
status:
phase: Bound
- 查看
RBD块大小
[root@m1 kubernetes]# rbd -p replicapool info csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4
rbd image 'csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4':
size 30 GiB in 7680 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 265b3d3b67a7a
block_name_prefix: rbd_data.265b3d3b67a7a
format: 2
features: layering
op_features:
flags:
create_timestamp: Sun Nov 27 12:51:04 2022
access_timestamp: Sun Nov 27 12:51:04 2022
modify_timestamp: Sun Nov 27 12:51:04 2022
容器内部磁盘空间查看
[root@m1 kubernetes]# kubectl exec -it wordpress-mysql-6965fc8cc8-zgvcx -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 37G 8.2G 29G 23% /
tmpfs 64M 0 64M 0% /dev
tmpfs 1.4G 0 1.4G 0% /sys/fs/cgroup
/dev/mapper/centos-root 37G 8.2G 29G 23% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/rbd0 30G 160M 30G 1% /var/lib/mysql # 扩容到了 30G
RBD 块扩容原理
PVC—>PV—>RBD块 —> 文件系统扩容
扩容 RBD 镜像块容量大小
[root@m1 kubernetes]# rbd -p replicapool resize csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4 --size 40G # 调整大小
Resizing image: 100% complete...done.
[root@m1 kubernetes]# rbd -p replicapool info csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4
rbd image 'csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4':
size 40 GiB in 10240 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 265b3d3b67a7a
block_name_prefix: rbd_data.265b3d3b67a7a
format: 2
features: layering
op_features:
flags:
create_timestamp: Sun Nov 27 12:51:04 2022
access_timestamp: Sun Nov 27 12:51:04 2022
modify_timestamp: Sun Nov 27 12:51:04 2022
- 查看
PVC/PV大小是否扩容
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 30Gi RWO rook-ceph-block 12m
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 30Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 12m
登陆容器所在宿主机，扩容文件系统的容量
[root@m1 kubernetes]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
prometheus-operator-7ccf6dfc8-sdhdp 1/1 Running 0 75m 172.20.4.37 192.168.100.136 <none> <none>
web-0 1/1 Running 1 2d3h 172.20.0.83 192.168.100.133 <none> <none>
web-1 1/1 Running 1 2d3h 172.20.0.85 192.168.100.133 <none> <none>
web-2 1/1 Running 0 2d3h 172.20.4.29 192.168.100.136 <none> <none>
wordpress-mysql-6965fc8cc8-zgvcx 1/1 Running 0 13m 172.20.4.39 192.168.100.136 <none> <none>
[root@m1 kubernetes]# ssh n3
Last login: Sun Nov 27 12:59:22 2022 from m1
[root@n3 ~]# rbd showmapped
2022-11-27 13:05:57.299341 7fc628c24d80 -1 did not load config file, using default settings.
id pool image snap device
0 replicapool csi-vol-197480ab-6e0f-11ed-b5c4-6efb82c232c4 - /dev/rbd0
2 replicapool csi-vol-a3dbf5a7-6c61-11ed-b5c4-6efb82c232c4 - /dev/rbd2
[root@n3 ~]# df -h | grep pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706
/dev/rbd0 30G 160M 30G 1% /var/lib/kubelet/pods/c001d7c0-0294-4be2-a8c5-10137f870adc/volumes/kubernetes.io~csi/pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706/mount
[root@n3 ~]# df -h | grep pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 | awk '{print $1}'
/dev/rbd0
[root@n3 ~]# resize2fs /dev/rbd0
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/rbd0 is mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706/globalmount/0001-0009-rook-ceph-0000000000000002-197480ab-6e0f-11ed-b5c4-6efb82c232c4; on-line resizing required
old_desc_blocks = 4, new_desc_blocks = 5
The filesystem on /dev/rbd0 is now 10485760 blocks long.
查看校验存储容量扩展情况
[root@n3 ~]# df -h | grep pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706
/dev/rbd0 40G 164M 40G 1% /var/lib/kubelet/pods/c001d7c0-0294-4be2-a8c5-10137f870adc/volumes/kubernetes.io~csi/pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706/mount
[root@n3 ~]# exit
logout
Connection to n3 closed.
[root@m1 kubernetes]# kubectl exec -it wordpress-mysql-6965fc8cc8-zgvcx -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 37G 8.3G 29G 23% /
tmpfs 64M 0 64M 0% /dev
tmpfs 1.4G 0 1.4G 0% /sys/fs/cgroup
/dev/mapper/centos-root 37G 8.3G 29G 23% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/rbd0 40G 164M 40G 1% /var/lib/mysql # 扩容到 40G
- 遗留问题：
PVC/PV信息没有扩容
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 30Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 18m
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-cfe4a32c-c746-46af-84e8-cd981cdf8706 30Gi RWO rook-ceph-block 18m
