K8S使用ceph-csi持久化存储之RBD
Kubernetes集成Ceph一般有两种方案：Ceph组件运行在Kubernetes中，可以使用Rook；还有就是Kubernetes集成外部的Ceph集群。
Ceph版本
[root@master ~]# ceph -v
ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)
Kubernetes版本
[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Ceph集群准备
在Ceph集群中单独创建一个存储池并初始化，创建存储池可以指定pg数等
[root@master ~]# ceph osd pool create kubernetes 128
pool 'kubernetes' created
[root@master ~]# rbd pool init kubernetes
[root@master ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
key = AQDX7tBkCRkAOxAA+q1qggqp/bULIyeZoEU9OA==
# 这个ID下面04-csi-rbd-secret.yaml文件会用到
生成ceph-csi的kubernetes configmap
[root@master ~]# ceph mon dump
epoch 1
fsid 83738b81-56e4-4d34-bdc2-3a60d789d224
last_changed 2023-08-07 19:35:54.436232
created 2023-08-07 19:35:54.436232
min_mon_release 14 (nautilus)
0: [v2:192.168.1.60:3300/0,v1:192.168.1.60:6789/0] mon.master
1: [v2:192.168.1.70:3300/0,v1:192.168.1.70:6789/0] mon.node01
2: [v2:192.168.1.80:3300/0,v1:192.168.1.80:6789/0] mon.node02
dumped monmap epoch 1
------ # fsid = cluster id
[root@master ~]# ceph -s
cluster:
id: 83738b81-56e4-4d34-bdc2-3a60d789d224
health: HEALTH_OK
创建ConfigMap，包含Ceph的连接信息
[root@master ~]# mkdir csi_rbd_sc && cd csi_rbd_sc
[root@master csi_rbd_sc]# cat 01-csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
[
{
"clusterID": "83738b81-56e4-4d34-bdc2-3a60d789d224",
"monitors": [
"192.168.1.60:6789",
"192.168.1.70:6789",
"192.168.1.80:6789"
]
}
]
metadata:
name: ceph-csi-config
创建名为 ceph-csi-encryption-kms-config 的ConfigMap，因没有使用KMS，配置内容为空（不能省略，否则后面容器启动会报错）
[root@master csi_rbd_sc]# cat 02-csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
{}
metadata:
name: ceph-csi-encryption-kms-config
创建名为 ceph-config 的ConfigMap，作用是定义使用cephx认证方式访问Ceph
[root@master csi_rbd_sc]# cat 03-ceph-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
ceph.conf: |
[global]
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
# keyring is a required key and its value should be empty
keyring: |
metadata:
name: ceph-config
查看 client.kubernetes 用户信息
[root@master csi_rbd_sc]# ceph auth get client.kubernetes
[client.kubernetes]
key = AQDX7tBkCRkAOxAA+q1qggqp/bULIyeZoEU9OA==
caps mgr = "profile rbd pool=kubernetes"
caps mon = "profile rbd"
caps osd = "profile rbd pool=kubernetes"
exported keyring for client.kubernetes
# 查看全部用户：ceph auth ls
创建Secret 对象，存储访问Ceph的key
[root@master csi_rbd_sc]# cat 04-csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: default
stringData:
userID: kubernetes
userKey: AQDX7tBkCRkAOxAA+q1qggqp/bULIyeZoEU9OA==
配置访问Kubernetes API对象的ServiceAccount和RBAC
ceph-csi插件(kubernetes上的rbac和提供存储功能的容器)
ClusterRole/ClusterRoleBinding
可以通信github直接部署
# kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
# kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
离线文件部署：
csi-provisioner-rbac.yaml
[root@master csi_rbd_sc]# cat 05-csi-provisioner-rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-csi-provisioner
# replace with non-default namespace name
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-external-provisioner-runner
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["persistentvolumeclaims/status"]
verbs: ["update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots/status"]
verbs: ["get", "list", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents/status"]
verbs: ["update", "patch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["get"]
- apiGroups: [""]
resources: ["serviceaccounts/token"]
verbs: ["create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-provisioner-role
subjects:
- kind: ServiceAccount
name: rbd-csi-provisioner
# replace with non-default namespace name
namespace: default
roleRef:
kind: ClusterRole
name: rbd-external-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
# replace with non-default namespace name
namespace: default
name: rbd-external-provisioner-cfg
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-provisioner-role-cfg
# replace with non-default namespace name
namespace: default
subjects:
- kind: ServiceAccount
name: rbd-csi-provisioner
# replace with non-default namespace name
namespace: default
roleRef:
kind: Role
name: rbd-external-provisioner-cfg
apiGroup: rbac.authorization.k8s.io
csi-nodeplugin-rbac.yaml
[root@master csi_rbd_sc]# cat 06-csi-nodeplugin-rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-csi-nodeplugin
# replace with non-default namespace name
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-nodeplugin
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
# allow to read Vault Token and connection options from the Tenants namespace
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["get"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["serviceaccounts/token"]
verbs: ["create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-nodeplugin
subjects:
- kind: ServiceAccount
name: rbd-csi-nodeplugin
# replace with non-default namespace name
namespace: default
roleRef:
kind: ClusterRole
name: rbd-csi-nodeplugin
apiGroup: rbac.authorization.k8s.io
运行ceph-csi provisoner和节点插件
包含镜像版本，要是用其他版本，请自行修改yaml文件，以下为已经修改
官方文件
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
离线文件
csi-rbdplugin-provisioner.yaml
[root@master csi_rbd_sc]# cat 07-csi-rbdplugin-provisioner.yaml
---
kind: Service
apiVersion: v1
metadata:
name: csi-rbdplugin-provisioner
# replace with non-default namespace name
namespace: default
labels:
app: csi-metrics
spec:
selector:
app: csi-rbdplugin-provisioner
ports:
- name: http-metrics
port: 8080
protocol: TCP
targetPort: 8680
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: csi-rbdplugin-provisioner
# replace with non-default namespace name
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: csi-rbdplugin-provisioner
template:
metadata:
labels:
app: csi-rbdplugin-provisioner
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- csi-rbdplugin-provisioner
topologyKey: "kubernetes.io/hostname"
serviceAccountName: rbd-csi-provisioner
priorityClassName: system-cluster-critical
containers:
- name: csi-provisioner
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-provisioner_v3.5.0
args:
- "--csi-address=$(ADDRESS)"
- "--v=1"
- "--timeout=150s"
- "--retry-interval-start=500ms"
- "--leader-election=true"
# set it to true to use topology based provisioning
- "--feature-gates=Topology=false"
- "--feature-gates=HonorPVReclaimPolicy=true"
- "--prevent-volume-mode-conversion=true"
# if fstype is not specified in storageclass, ext4 is default
- "--default-fstype=ext4"
- "--extra-create-metadata=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-snapshotter
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-snapshotter_v6.2.2
args:
- "--csi-address=$(ADDRESS)"
- "--v=1"
- "--timeout=150s"
- "--leader-election=true"
- "--extra-create-metadata=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-attacher
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-attacher_v4.3.0
args:
- "--v=1"
- "--csi-address=$(ADDRESS)"
- "--leader-election=true"
- "--retry-interval-start=500ms"
- "--default-fstype=ext4"
env:
- name: ADDRESS
value: /csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-resizer
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-resizer_v1.8.0
args:
- "--csi-address=$(ADDRESS)"
- "--v=1"
- "--timeout=150s"
- "--leader-election"
- "--retry-interval-start=500ms"
- "--handle-volume-inuse-error=false"
- "--feature-gates=RecoverVolumeExpansionFailure=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-rbdplugin
# for stable functionality replace canary with latest release version
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
args:
- "--nodeid=$(NODE_ID)"
- "--type=rbd"
- "--controllerserver=true"
- "--endpoint=$(CSI_ENDPOINT)"
- "--csi-addons-endpoint=$(CSI_ADDONS_ENDPOINT)"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--pidlimit=-1"
- "--rbdhardmaxclonedepth=8"
- "--rbdsoftmaxclonedepth=4"
- "--enableprofiling=false"
- "--setmetadata=true"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# - name: KMS_CONFIGMAP_NAME
# value: encryptionConfig
- name: CSI_ENDPOINT
value: unix:///csi/csi-provisioner.sock
- name: CSI_ADDONS_ENDPOINT
value: unix:///csi/csi-addons.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- mountPath: /dev
name: host-dev
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
#- name: ceph-csi-encryption-kms-config
# mountPath: /etc/ceph-csi-encryption-kms-config/
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
#- name: ceph-config
# mountPath: /etc/ceph/
- name: oidc-token
mountPath: /run/secrets/tokens
readOnly: true
- name: csi-rbdplugin-controller
# for stable functionality replace canary with latest release version
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
args:
- "--type=controller"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--drivernamespace=$(DRIVER_NAMESPACE)"
- "--setmetadata=true"
env:
- name: DRIVER_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
#- name: ceph-config
# mountPath: /etc/ceph/
- name: liveness-prometheus
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
args:
- "--type=liveness"
- "--endpoint=$(CSI_ENDPOINT)"
- "--metricsport=8680"
- "--metricspath=/metrics"
- "--polltime=60s"
- "--timeout=3s"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi-provisioner.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: socket-dir
mountPath: /csi
imagePullPolicy: "IfNotPresent"
volumes:
- name: host-dev
hostPath:
path: /dev
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: socket-dir
emptyDir: {
medium: "Memory"
}
#- name: ceph-config
# configMap:
# name: ceph-config
- name: ceph-csi-config
configMap:
name: ceph-csi-config
#- name: ceph-csi-encryption-kms-config
# configMap:
# name: ceph-csi-encryption-kms-config
- name: keys-tmp-dir
emptyDir: {
medium: "Memory"
}
- name: oidc-token
projected:
sources:
- serviceAccountToken:
path: oidc-token
expirationSeconds: 3600
audience: ceph-csi-kms
csi-rbdplugin.yaml
[root@master csi_rbd_sc]# cat 08-csi-rbdplugin.yaml
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-rbdplugin
# replace with non-default namespace name
namespace: default
spec:
selector:
matchLabels:
app: csi-rbdplugin
template:
metadata:
labels:
app: csi-rbdplugin
spec:
serviceAccountName: rbd-csi-nodeplugin
hostNetwork: true
hostPID: true
priorityClassName: system-node-critical
# to use e.g. Rook orchestrated cluster, and mons' FQDN is
# resolved through k8s service, set dns policy to cluster first
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: driver-registrar
# This is necessary only for systems with SELinux, where
# non-privileged sidecar containers cannot access unix domain socket
# created by privileged CSI driver container.
securityContext:
privileged: true
allowPrivilegeEscalation: true
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-node-driver-registrar_v2.8.0
args:
- "--v=1"
- "--csi-address=/csi/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: csi-rbdplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
# for stable functionality replace canary with latest release version
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
args:
- "--nodeid=$(NODE_ID)"
- "--pluginpath=/var/lib/kubelet/plugins"
- "--stagingpath=/var/lib/kubelet/plugins/kubernetes.io/csi/"
- "--type=rbd"
- "--nodeserver=true"
- "--endpoint=$(CSI_ENDPOINT)"
- "--csi-addons-endpoint=$(CSI_ADDONS_ENDPOINT)"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--enableprofiling=false"
# If topology based provisioning is desired, configure required
# node labels representing the nodes topology domain
# and pass the label names below, for CSI to consume and advertise
# its equivalent topology domain
# - "--domainlabels=failure-domain/region,failure-domain/zone"
#
# Options to enable read affinity.
# If enabled Ceph CSI will fetch labels from kubernetes node and
# pass `read_from_replica=localize,crush_location=type:value` during
# rbd map command. refer:
# https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
# for more details.
# - "--enable-read-affinity=true"
# - "--crush-location-labels=topology.io/zone,topology.io/rack"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# - name: KMS_CONFIGMAP_NAME
# value: encryptionConfig
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: CSI_ADDONS_ENDPOINT
value: unix:///csi/csi-addons.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- mountPath: /dev
name: host-dev
- mountPath: /sys
name: host-sys
- mountPath: /run/mount
name: host-mount
- mountPath: /etc/selinux
name: etc-selinux
readOnly: true
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: ceph-csi-encryption-kms-config
mountPath: /etc/ceph-csi-encryption-kms-config/
- name: plugin-dir
mountPath: /var/lib/kubelet/plugins
mountPropagation: "Bidirectional"
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: ceph-logdir
mountPath: /var/log/ceph
- name: ceph-config
mountPath: /etc/ceph/
- name: oidc-token
mountPath: /run/secrets/tokens
readOnly: true
- name: liveness-prometheus
securityContext:
privileged: true
allowPrivilegeEscalation: true
image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
args:
- "--type=liveness"
- "--endpoint=$(CSI_ENDPOINT)"
- "--metricsport=8680"
- "--metricspath=/metrics"
- "--polltime=60s"
- "--timeout=3s"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: socket-dir
mountPath: /csi
imagePullPolicy: "IfNotPresent"
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
type: DirectoryOrCreate
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins
type: Directory
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: DirectoryOrCreate
- name: ceph-logdir
hostPath:
path: /var/log/ceph
type: DirectoryOrCreate
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/
type: Directory
- name: host-dev
hostPath:
path: /dev
- name: host-sys
hostPath:
path: /sys
- name: etc-selinux
hostPath:
path: /etc/selinux
- name: host-mount
hostPath:
path: /run/mount
- name: lib-modules
hostPath:
path: /lib/modules
- name: ceph-config
configMap:
name: ceph-config
- name: ceph-csi-config
configMap:
name: ceph-csi-config
- name: ceph-csi-encryption-kms-config
configMap:
name: ceph-csi-encryption-kms-config
- name: keys-tmp-dir
emptyDir: {
medium: "Memory"
}
- name: oidc-token
projected:
sources:
- serviceAccountToken:
path: oidc-token
expirationSeconds: 3600
audience: ceph-csi-kms
---
# This is a service to expose the liveness metrics
apiVersion: v1
kind: Service
metadata:
name: csi-metrics-rbdplugin
# replace with non-default namespace name
namespace: default
labels:
app: csi-metrics
spec:
ports:
- name: http-metrics
port: 8080
protocol: TCP
targetPort: 8680
selector:
app: csi-rbdplugin
上面两个yaml文件是核心，如果对其中运行Pod的作用不清楚，可以了解下容器存储接口-CSI
注意：所使用的镜像以及修改为本地仓库镜像，请根据自己网络环境调整
[root@master csi_rbd_sc]# grep "image" 07-csi-rbdplugin-provisioner.yaml 08-csi-rbdplugin.yaml
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-provisioner_v3.5.0
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-snapshotter_v6.2.2
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-attacher_v4.3.0
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-resizer_v1.8.0
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
07-csi-rbdplugin-provisioner.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
07-csi-rbdplugin-provisioner.yaml: imagePullPolicy: "IfNotPresent"
08-csi-rbdplugin.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:csi-node-driver-registrar_v2.8.0
08-csi-rbdplugin.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
08-csi-rbdplugin.yaml: imagePullPolicy: "IfNotPresent"
08-csi-rbdplugin.yaml: image: registry.cn-hangzhou.aliyuncs.com/chuyio/k8s_images:cephcsi_canary
08-csi-rbdplugin.yaml: imagePullPolicy: "IfNotPresent"
创建 StorageClass 使用ceph块儿设备
Kubernetes实现按需创建PV，需要创建StorageClass API对象，如下
[root@master csi_rbd_sc]# cat 09-csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
clusterID: 83738b81-56e4-4d34-bdc2-3a60d789d224
pool: kubernetes
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
创建相关文件
[root@master csi_rbd_sc]# ls
01-csi-config-map.yaml 03-ceph-config-map.yaml 05-csi-provisioner-rbac.yaml 07-csi-rbdplugin-provisioner.yaml 09-csi-rbd-sc.yaml
02-csi-kms-config-map.yaml 04-csi-rbd-secret.yaml 06-csi-nodeplugin-rbac.yaml 08-csi-rbdplugin.yaml raw-block-pvc.yaml
[root@master csi_rbd_sc]# kubectl apply -f 01-csi-config-map.yaml
configmap/ceph-csi-config created
[root@master csi_rbd_sc]# kubectl apply -f 02-csi-kms-config-map.yaml
configmap/ceph-csi-encryption-kms-config created
[root@master csi_rbd_sc]# kubectl apply -f 03-ceph-config-map.yaml
configmap/ceph-config created
[root@master csi_rbd_sc]# kubectl apply -f 04-csi-rbd-secret.yaml
secret/csi-rbd-secret created
[root@master csi_rbd_sc]# kubectl apply -f 05-csi-provisioner-rbac.yaml
serviceaccount/rbd-csi-provisioner created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created
[root@master csi_rbd_sc]# kubectl apply -f 06-csi-nodeplugin-rbac.yaml
serviceaccount/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
[root@master csi_rbd_sc]# kubectl apply -f 07-csi-rbdplugin-provisioner.yaml
service/csi-rbdplugin-provisioner created
deployment.apps/csi-rbdplugin-provisioner created
[root@master csi_rbd_sc]# kubectl apply -f 08-csi-rbdplugin.yaml
daemonset.apps/csi-rbdplugin created
service/csi-metrics-rbdplugin created
[root@master csi_rbd_sc]# kubectl apply -f 09-csi-rbd-sc.yaml
storageclass.storage.k8s.io/csi-rbd-sc created
查看pod状态
[root@master csi_rbd_sc]# kubectl get pod
NAME READY STATUS RESTARTS AGE
csi-rbdplugin-httm8 0/3 ContainerCreating 0 77s
csi-rbdplugin-jckw7 0/3 ContainerCreating 0 77s
csi-rbdplugin-provisioner-66df86d8f9-6tlr7 0/7 ContainerCreating 0 83s
csi-rbdplugin-provisioner-66df86d8f9-psn6m 0/7 ContainerCreating 0 83s
csi-rbdplugin-provisioner-66df86d8f9-tsk9g 0/7 ContainerCreating 0 83s
csi-rbdplugin-tshw8 0/3 ContainerCreating 0 77s
等待pod启动
[root@master csi_rbd_sc]# kubectl get pod
NAME READY STATUS RESTARTS AGE
csi-rbdplugin-httm8 3/3 Running 0 114s
csi-rbdplugin-jckw7 3/3 Running 0 114s
csi-rbdplugin-provisioner-66df86d8f9-6tlr7 7/7 Running 0 2m
csi-rbdplugin-provisioner-66df86d8f9-psn6m 7/7 Running 0 2m
csi-rbdplugin-provisioner-66df86d8f9-tsk9g 7/7 Running 0 2m
csi-rbdplugin-tshw8 3/3 Running 0 114s
查看 storageclass
[root@master csi_rbd_sc]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 2m11s
volumeBindingMode：卷绑定模式
该volumeBindingMode字段控制何时应该发生卷绑定和动态配置。未设置时，默认使用立即模式。
该Immediate模式指示一旦创建PersistentVolumeClaim，就会发生卷绑定和动态供应。对于拓扑受限且无法从集群中的所有节点全局访问的存储后端，将在不知道 Pod 调度要求的情况下绑定或配置PersistentVolume。这可能会导致不可调度的 Pod。
集群管理员可以通过指定模式来解决此问题，该WaitForFirstConsumer模式将延迟PersistentVolume的绑定和配置，直到创建使用PersistentVolumeClaim的 Pod。PersistentVolume将根据 Pod 的调度约束指定的拓扑进行选择或配置。这些包括但不限于资源需求、 节点选择器、 pod 亲和性和反亲和性以及污点和容忍度。
