06 云原生 RBD 块存储（转载）
目录
云原生 RBD 块存储
RBD 块存储概述
什么是块存储：A block is a sequence of bytes (often 512). Block-based storage interfaces are a mature and common way to store data on media including HDDs, SSDs, CDs, floppy disks, and even tape.
EBSCBSFC块
- Thin-provisioned （受分配，使用多少分配多少，慢慢扩大）
- Images up to 16 exabytes （单个镜像最大16EB）
- Configurable striping（可配置切片）
- In-memory caching （内存缓存）
- Snapshots（支持快照）
- Copy-on-write cloning（快照克隆）
- Kernel driver support（内核支持）
- KVM/libvirt support（kvm/librirt支持）
- Back-end for cloud solutions（后端支持云解决方案）
- Incremental backup（增量备份）
- Disaster recovery (multisite asynchronous replication)（灾难恢复）
RBD 与容器对接
三种方式
volume: 卷的存储方式，支持多种驱动，FC，EBS，Ceph等PV/PVC:Persistent Volume和Persistent Volume ClaimStorageClass: 包含静态+动态两种- 管理员定义好
provioner - 终端用户通过
PVC关联
- 管理员定义好
Ceph 对接回顾
Ceph 和 kubernetes 的对接过程涉及到 pool 的创建， Ceph 认证信息，配置文件， CSI 驱动部署， StorageClass 创建等一系列过程。配置过程有一定的难度，如果对 Ceph 不熟悉的同学，对接可能有一定难度，而 Rook 则将这些配置过程简化，以云原生的方式实现对接，其默认已经继承好相关驱动，直接通过 kubernetes 创建 storageclass 即可。
CSI 驱动信息
- 包含
rbd和cephfs的驱动，csi-cephfsplugin和csi-rbdplugin - 驱动由
provisioner和plugin组成
RBD块存储类
RBD 相关的存储驱动和 provisioner 安装 rook 时候已经创建好，因此接下来只需要直接对接即可， Rook 提供了两种对接的方式：
FlexVolumeCSI
其中 Flex 的方式比较老，默认驱动未安装，需要安装才可以对接，逐步淘汰，不建议使用，推荐使用 CSI 的对接方式
[root@m1 ceph]# cat csi/rbd/storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: rook-ceph # namespace:cluster
pool: replicapool
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/fstype: ext4
allowVolumeExpansion: true
reclaimPolicy: Delete
# 执行资源清单
[root@m1 ceph]# kubectl apply -f csi/rbd/storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created
校验 pool 安装情况
- 配置
ceph客户端认证
[root@m1 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-77bf5b9b7d-9pq6m -- cat /etc/ceph/ceph.conf | tee /etc/ceph/ceph.conf
[global]
mon_host = 10.68.231.222:6789,10.68.163.216:6789,10.68.61.127:6789
[client.admin]
keyring = /etc/ceph/keyring
[root@m1 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-77bf5b9b7d-9pq6m -- cat /etc/ceph/keyring | tee /etc/ceph/keyring
[client.admin]
key = AQDm435jUMuCKhAAXXPqMX08cLyjs/EOvchkzA==
[root@m1 ceph]# ceph -s
cluster:
id: 17a413b5-f140-441a-8b35-feec8ae29521
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 2h)
mgr: a(active, since 6h)
osd: 5 osds: 5 up (since 6h), 5 in (since 6h)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 5.0 GiB used, 245 GiB / 250 GiB avail
pgs: 1 active+clean
- 校验
pool
[root@m1 ceph]# ceph osd lspools
1 device_health_metrics
2 replicapool
[root@m1 ceph]# ceph osd pool get replicapool size
size: 3
校验 StorageClass
[root@m1 ceph]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 2m56s
容器 PVC 调用存储
创建好 storageclass 之后，我们就可以通过 PVC 向 storageclass 申请容量空间了， PVC 会自动和 storageclass 完成存储容量的创建过程，包括自动创建 PV ， PV 与后端存储自动完成 RBD 块存储的创建，整个过程不需要我们关心，均通过 storageclass 和驱动自动完成，我们只需要关注使用即可，如下是一个 wordpress 博客应用连接 MySQL 数据库的一个云原生应用的范例
wordpress 资源清单
[root@m1 ceph]# cd ../
[root@m1 kubernetes]# cat wordpress.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
- port: 80
selector:
app: wordpress
tier: frontend
type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
tier: frontend
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:4.6.1-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
value: changeme
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim
MySQL 资源清单
[root@m1 kubernetes]# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
tier: mysql
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: changeme
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
创建 wordpress 和 mysql
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
wordpress-7b989dbf57-cvqtk 1/1 Running 0 6m25s
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 6m20s
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO rook-ceph-block 29s
wp-pv-claim Bound pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO rook-ceph-block 34s
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 31s
pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 35s
PVC 调用逻辑
PVC 会完成一系列的出对接过程，包含有什么动作。 PVC —> storageclass 申请容量 —> 创建 PV ——> 向 Ceph 申请 RBD 块，完成和 Ceph 对接
查看PVC创建情况
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO rook-ceph-block 8m14s
wp-pv-claim Bound pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO rook-ceph-block 8m19s
- 查看详情
[root@m1 kubernetes]# kubectl get pvc mysql-pv-claim -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
......
name: mysql-pv-claim
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: rook-ceph-block
volumeMode: Filesystem
volumeName: pvc-2bcd6af2-31e3-40b8-add9-476b167399be
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
phase: Bound
PVC 会自动创建 PV
PV信息
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 10m
pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 10m
- 详细的
PV信息
[root@m1 kubernetes]# kubectl get pv pvc-2bcd6af2-31e3-40b8-add9-476b167399be -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
......
name: pvc-2bcd6af2-31e3-40b8-add9-476b167399be
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: mysql-pv-claim
namespace: default
resourceVersion: "502998"
uid: 2bcd6af2-31e3-40b8-add9-476b167399be
csi:
controllerExpandSecretRef:
name: rook-csi-rbd-provisioner
namespace: rook-ceph
driver: rook-ceph.rbd.csi.ceph.com
fsType: ext4
nodeStageSecretRef:
name: rook-csi-rbd-node
namespace: rook-ceph
volumeAttributes:
clusterID: rook-ceph
imageFeatures: layering
imageFormat: "2"
imageName: csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4 # 关联 rbd 信息
journalPool: replicapool
pool: replicapool
radosNamespace: ""
storage.kubernetes.io/csiProvisionerIdentity: 1669260272979-8081-rook-ceph.rbd.csi.ceph.com
volumeHandle: 0001-0009-rook-ceph-0000000000000002-10342079-6beb-11ed-b5c4-6efb82c232c4
persistentVolumeReclaimPolicy: Delete
storageClassName: rook-ceph-block
volumeMode: Filesystem
status:
phase: Bound
PV 会完成和 Ceph 的对接，自动创建 RBD 块存储空间，期间由 plugin 驱动完成创建
rbd信息
[root@m1 kubernetes]# rbd -p replicapool ls
csi-vol-0d43155e-6beb-11ed-b5c4-6efb82c232c4
csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4
mysql使用rbd块信息
[root@m1 kubernetes]# rbd -p replicapool info csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4
rbd image 'csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4':
size 20 GiB in 5120 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 12fae9f8d2613
block_name_prefix: rbd_data.12fae9f8d2613
format: 2
features: layering
op_features:
flags:
create_timestamp: Thu Nov 24 19:28:05 2022
access_timestamp: Thu Nov 24 19:28:05 2022
modify_timestamp: Thu Nov 24 19:28:05 2022
Wordpress 功能验证
wordpress 和 mysql 实际存储空间存放在 Ceph RBD 块中，通过 nodePort 的登录 MySQL ，验证 wordpress 的安装情况，首先需要核对 pods 部署是否正常，确保均 Running
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
wordpress-7b989dbf57-cvqtk 1/1 Running 0 17m
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 17m
wordpress 通过 service 暴露服务，默认类型为 LoadBalancer ， LoadBalancer 依赖于 NodePort ，因此可以直接通过 NodePort 的形式访问，通过任意一个节点的 NodePort 即可访问到 wordpress 的应用。
[root@m1 kubernetes]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 2d3h
wordpress LoadBalancer 10.68.235.228 <pending> 80:36063/TCP 19m
wordpress-mysql ClusterIP None <none> 3306/TCP 19m
- 查看
rbd映射情况
[root@m1 kubernetes]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
wordpress-7b989dbf57-cvqtk 1/1 Running 0 41m 172.20.4.27 192.168.100.136 <none> <none>
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 41m 172.20.4.28 192.168.100.136 <none> <none>
[root@m1 kubernetes]# ssh n3
Last login: Thu Nov 24 20:07:31 2022 from m1
[root@n3 ~]# rbd showmapped
2022-11-24 20:09:07.150816 7f5daf0d6d80 -1 did not load config file, using default settings.
id pool image snap device
0 replicapool csi-vol-0d43155e-6beb-11ed-b5c4-6efb82c232c4 - /dev/rbd0
1 replicapool csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4 - /dev/rbd1
[root@n3 ~]# exit
存储持久化模板
PVC 使用模式适用于单个 pods 容器，如多个 pods 都需要有各自的存储如何实现，需要借助于 StatefulSet 的 volumeClaimTemplates 功能，实现每个 pods 均有各自的存储。
[root@m1 kubernetes]# cat << EOF > sts-test.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "rook-ceph-block"
resources:
requests:
storage: 10Gi
EOF
- 部署服务查看
pv pvc情况
[root@m1 kubernetes]# kubectl apply -f sts-test.yaml
statefulset.apps/web created
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 103s
web-1 1/1 Running 0 65s
web-2 1/1 Running 0 51s
wordpress-7b989dbf57-cvqtk 1/1 Running 0 14h
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 14h
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO rook-ceph-block 14h
wp-pv-claim Bound pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO rook-ceph-block 14h
www-web-0 Bound pvc-9d32f5ff-174a-461f-9f3c-e0f0f5e60cc8 10Gi RWO rook-ceph-block 83s
www-web-1 Bound pvc-af0e2916-ec93-4f55-871f-0d008c0394e0 10Gi RWO rook-ceph-block 45s
www-web-2 Bound pvc-174fb859-91ad-49e2-8e44-d7ee64645e7e 10Gi RWO rook-ceph-block 31s
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-174fb859-91ad-49e2-8e44-d7ee64645e7e 10Gi RWO Delete Bound default/www-web-2 rook-ceph-block 32s
pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 14h
pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 14h
pvc-9d32f5ff-174a-461f-9f3c-e0f0f5e60cc8 10Gi RWO Delete Bound default/www-web-0 rook-ceph-block 84s
pvc-af0e2916-ec93-4f55-871f-0d008c0394e0 10Gi RWO Delete Bound default/www-web-1 rook-ceph-block 46s
