k8s 使用ceph
Ceph 同时提供块块存储、cephfs给k8s使用，在k8s各节点安装ceph客户端工具ceph-common
试验步骤
1.创建存储池
ceph osd pool create kubernetes 32 32
2.为存储池开启rbd功能
ceph osd pool application enable kubernetes rbd
3.初始化存储
rbd pool init -p kubernetes
由于试验环境一致存在pg同步因此临时把pg副本数量调整为1。生产环境不建议
ceph osd pool set kubernetes min_size 1
ceph osd pool set kubernetes size 1
ceph osd pool get kubernetes size
4.为pod挂载提供用户
ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
key = AQAO13di1Rd/AhAAH38UyAR2UhqzdyBP5lxk+g== # 提供给csi-rbd-secret.yml使用
5.获取集群的fsid
ceph mon dump
epoch 1
fsid fbe5d74c-2d76-4103-9768-c9b766b7af1c # 提供给ceph-csi-config.yml使用
last_changed 2022-05-02 13:14:24.128643
created 2022-05-02 13:14:24.128643
min_mon_release 14 (nautilus)
0: [v2:10.4.7.31:3300/0,v1:10.4.7.31:6789/0] mon.ceph-mon2
dumped monmap epoch 1
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/ceph-config.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/ceph-csi-config.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/ceph-csi-encryption-kms-config.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/csi-rbd-sc.yml
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/csi-rbd-secret.yml
#cehp-csi operator
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
#测试用例
kubectl apply -f https://gitee.com/mingtian66/k8s/raw/master/存储/test-pod.yml
pod 挂载后查看当前有哪些image
[root@ceph-mon2 ~]# rbd ls -p kubernetes
csi-vol-0bf242b4-d666-11ec-9a2c-eac0121d779d
csi-vol-197f468e-d66c-11ec-9a2c-eac0121d779d
csi-vol-629094d2-d657-11ec-9a2c-eac0121d779d
csi-vol-8044406d-d2c1-11ec-b0b1-0ea9a3ef30b9
csi-vol-994f89b2-d8d9-11ec-9fc1-165dc5a2f727
csi-vol-ad5f03a8-d8d9-11ec-9fc1-165dc5a2f727
csi-vol-dc243bae-d8d8-11ec-9fc1-165dc5a2f727
csi-vol-e43fef15-d667-11ec-9a2c-eac0121d779d
csi-vol-e875cb0a-d667-11ec-9a2c-eac0121d779d
查看指定image的详细属性
[root@ceph-mon2 ~]# rbd --image csi-vol-0bf242b4-d666-11ec-9a2c-eac0121d779d -p kubernetes info
rbd image 'csi-vol-0bf242b4-d666-11ec-9a2c-eac0121d779d':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: acd877947655
block_name_prefix: rbd_data.acd877947655
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed May 18 12:50:31 2022
access_timestamp: Wed May 18 12:50:31 2022
modify_timestamp: Wed May 18 12:50:31 2022
备注：
提前准备好镜像，如果你的网络可以这一步可以跳过
docker pull quay.io/k8scsi/csi-provisioner:canary
docker tag quay.io/k8scsi/csi-provisioner:canary gcr.io/k8s-staging-sig-storage/csi-provisioner:canary
docker pull antidebug/csi-snapshotter:v5.0.0
docker tag antidebug/csi-snapshotter:v5.0.0 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1
docker pull dyrnq/csi-attacher:v3.4.0
docker tag dyrnq/csi-attacher:v3.4.0 registry.k8s.io/sig-storage/csi-attacher:v3.4.0
docker pull objectscale/csi-resizer:v1.4.0
docker tag objectscale/csi-resizer:v1.4.0 registry.k8s.io/sig-storage/csi-resizer:v1.4.0
docker pull dyrnq/csi-node-driver-registrar:v2.4.0
docker tag dyrnq/csi-node-driver-registrar:v2.4.0 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.4.0
docker pull quay.io/cephcsi/cephcsi:canary
docker pull quay.io/cephcsi/cephcsi:canary
