k8s集成ceph rbd(StorageClass方式)
-
ceph 集群创建存储池
ceph osd pool create k8s 128 128
-
获取 key
$ ceph auth get-key client.admin | base64 QVFEMjVxVmhiVUNJRHhBQUxwdmVHbUdNTWtXZjB6VXovbWlBY3c9PQ==
-
k8s 集群节点安装 ceph-common,版本需和 ceph 集群一致
rpm -ivh http://download.ceph.com/rpm-luminous/el7/noarch/ceph-release-1-1.el7.noarch.rpm sed -i 's#download.ceph.com#mirrors.aliyun.com/ceph#g' /etc/yum.repos.d/ceph.repo yum install epel-release -y yum install -y ceph-common
-
编辑 yaml 文件
$ vi ceph-sc.yaml apiVersion: v1 kind: Namespace metadata: name: ceph --- apiVersion: v1 kind: Secret metadata: name: ceph-storageclass-secret namespace: ceph data: key: QVFEMjVxVmhiVUNJRHhBQUxwdmVHbUdNTWtXZjB6VXovbWlBY3c9PQ== type: kubernetes.io/rbd --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-storageclass annotations: storageclass.kubernetes.io/is-default-class: "false" provisioner: kubernetes.io/rbd parameters: #monitors: 10.10.10.51:6789,10.10.10.53:6789,10.10.10.53:6789 monitors: ceph01:6789,ceph02:6789,ceph03:6789 adminId: admin adminSecretName: ceph-storageclass-secret adminSecretNamespace: ceph pool: k8s userId: admin userSecretName: ceph-storageclass-secret userSecretNamespace: ceph imageFormat: "2" imageFeatures: "layering"
- adminSecretNamespace 和 userSecretNamespace 非常重要,必须设置,否则后面 pod 调用 pvc 时会默认从 pod 的 namespace 查找 ceph-storageclass-secret,此时 pod 可能会报类似错误:couldn't get secret gitlab/ceph-storageclass-secret err: secrets "ceph-storageclass-secret" not found
测试 yaml:
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: ceph-pvc-test1 namespace: default annotations: volume.beta.kubernetes.io/storage-class: ceph-storageclass spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi # 或者 kind: PersistentVolumeClaim apiVersion: v1 metadata: name: ceph-pvc-test2 namespace: default spec: storageClassName: ceph-storageclass accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
- ceph rbd 只支持 ReadWriteOnce(RWO), ReadOnlyMany(ROX),不支持 ReadWriteMany(RWX),详见:Persistent Volumes | Kubernetes
-
执行
kubectl apply -f .
-
验证
$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE ceph-storageclass kubernetes.io/rbd Delete Immediate false 28s $ kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE default ceph-pvc-test1 Bound pvc-069bd7d7-cb5c-4f70-a760-691c64330dda 1Gi RWO ceph-storageclass 34s default ceph-pvc-test2 Bound pvc-9adb2d07-e72c-4bda-9012-1fc8e5389d1c 1Gi RWO ceph-storageclass 34s
注意:以上方法只适用于二进制方式安装的 k8s 集群,如果是使用的 pod 方式运行 kube-controller-manager,则会遇到以下错误:
rbd: create volume failed, err: failed to create rbd image: executable file not found in $PATH:
出现这个报错问题的原因其实很简单:gcr.io
中自带的kube-controller-manager镜像没有自带rbd
子命令。
解决方法是定义外部 provisioner:
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: rbd-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
image: "quay.io/external_storage/rbd-provisioner:latest"
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccountName: persistent-volume-binder
然后定义 storageClass 时:provisioner 指定为 provisioner: ceph.com/rbd 即可
···
provisioner: ceph.com/rbd
···
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 一个奇形怪状的面试题:Bean中的CHM要不要加volatile?
· [.NET]调用本地 Deepseek 模型
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· .NET Core 托管堆内存泄露/CPU异常的常见思路
· PostgreSQL 和 SQL Server 在统计信息维护中的关键差异
· DeepSeek “源神”启动!「GitHub 热点速览」
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· 我与微信审核的“相爱相杀”看个人小程序副业
· C# 集成 DeepSeek 模型实现 AI 私有化(本地部署与 API 调用教程)
· spring官宣接入deepseek,真的太香了~