k8s 使用ceph

Ceph 同时提供块块存储、cephfs给k8s使用，在k8s各节点安装ceph客户端工具ceph-common

试验步骤
1.创建存储池

ceph osd pool create kubernetes 32 32

2.为存储池开启rbd功能

ceph osd pool application enable kubernetes  rbd

3.初始化存储

rbd pool init -p kubernetes

由于试验环境一致存在pg同步因此临时把pg副本数量调整为1。生产环境不建议

ceph osd pool set kubernetes min_size 1
ceph osd pool set kubernetes size 1
ceph osd pool get kubernetes size

4.为pod挂载提供用户

ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'

[client.kubernetes]
key = AQAO13di1Rd/AhAAH38UyAR2UhqzdyBP5lxk+g== # 提供给csi-rbd-secret.yml使用

5.获取集群的fsid

ceph mon dump

epoch 1
fsid fbe5d74c-2d76-4103-9768-c9b766b7af1c # 提供给ceph-csi-config.yml使用
last_changed 2022-05-02 13:14:24.128643
created 2022-05-02 13:14:24.128643
min_mon_release 14 (nautilus)
0: [v2:10.4.7.31:3300/0,v1:10.4.7.31:6789/0] mon.ceph-mon2
dumped monmap epoch 1

kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/ceph-config.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/ceph-csi-config.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/ceph-csi-encryption-kms-config.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/csi-rbd-sc.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/csi-rbd-secret.yml

#cehp-csi operator
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
#测试用例
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/test-pod.yml

pod 挂载后查看当前有哪些image

[root@ceph-mon2 ~]# rbd ls -p kubernetes
csi-vol-0bf242b4-d666-11ec-9a2c-eac0121d779d
csi-vol-197f468e-d66c-11ec-9a2c-eac0121d779d
csi-vol-629094d2-d657-11ec-9a2c-eac0121d779d
csi-vol-8044406d-d2c1-11ec-b0b1-0ea9a3ef30b9
csi-vol-994f89b2-d8d9-11ec-9fc1-165dc5a2f727
csi-vol-ad5f03a8-d8d9-11ec-9fc1-165dc5a2f727
csi-vol-dc243bae-d8d8-11ec-9fc1-165dc5a2f727
csi-vol-e43fef15-d667-11ec-9a2c-eac0121d779d
csi-vol-e875cb0a-d667-11ec-9a2c-eac0121d779d

查看指定image的详细属性

[root@ceph-mon2 ~]# rbd --image csi-vol-0bf242b4-d666-11ec-9a2c-eac0121d779d -p kubernetes info
rbd image 'csi-vol-0bf242b4-d666-11ec-9a2c-eac0121d779d':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: acd877947655
        block_name_prefix: rbd_data.acd877947655
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Wed May 18 12:50:31 2022
        access_timestamp: Wed May 18 12:50:31 2022
        modify_timestamp: Wed May 18 12:50:31 2022

备注：
提前准备好镜像，如果你的网络可以这一步可以跳过

docker pull quay.io/k8scsi/csi-provisioner:canary
docker tag quay.io/k8scsi/csi-provisioner:canary gcr.io/k8s-staging-sig-storage/csi-provisioner:canary
 
docker pull antidebug/csi-snapshotter:v5.0.0
docker tag antidebug/csi-snapshotter:v5.0.0 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1
 
docker pull dyrnq/csi-attacher:v3.4.0
docker tag dyrnq/csi-attacher:v3.4.0 registry.k8s.io/sig-storage/csi-attacher:v3.4.0
 
docker pull objectscale/csi-resizer:v1.4.0
docker tag objectscale/csi-resizer:v1.4.0 registry.k8s.io/sig-storage/csi-resizer:v1.4.0
 
docker pull dyrnq/csi-node-driver-registrar:v2.4.0
docker tag dyrnq/csi-node-driver-registrar:v2.4.0 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.4.0
 
docker pull quay.io/cephcsi/cephcsi:canary
docker pull quay.io/cephcsi/cephcsi:canary