K8s & K3s 集群基于 rook & ceph 操作指南

动态存储简介

虽然K8S提供了PVC的方式进行存储的便利性，但是PV的创建还是要手工的，使用起来不是很方便，在k8s 1.4以后，kubernetes提供了一种更加方便的动态创建PV的方式，即StorageClass。使用StorageClass时无需预先创建固定大小的PV来等待使用者创建PVC使用，而是直接创建PVC即可使用。

本文主要使用Rook搭建一个ceph集群，然后使用StorageClass来直接创建PVC。

1 PV

PV全称叫做Persistent Volume，持久化存储卷。它是用来描述或者说用来定义一个存储卷的，这个通常都是有运维或者数据存储工程师来定义。比如下面我们定义一个NFS类型的PV：
accessModes：支持三种类型

• ReadWriteMany 多路读写，卷能被集群多个节点挂载并读写
• ReadWriteOnce 单路读写，卷只能被单一集群节点挂载读写
• ReadOnlyMany 多路只读，卷能被多个集群节点挂载且只能读

2 PVC

PVC是用来描述希望使用什么样的或者说是满足什么条件的存储，它的全称是Persistent Volume Claim，也就是持久化存储声明。开发人员使用这个来描述该容器需要一个什么存储。比如下面使用NFS的PVC：
这个PVC就会和上面的PV进行绑定，为什么呢？它有一些原则：

• PV和PVC中的spec关键字段要匹配，比如存储（storage）大小。
• PV和PVC中的storageClassName字段必须一致，这个后面再说。

3 StorageClass

StorageClassPV就是动态供给创建PV模板,是运维人员来创建的，开发操作PVC，可是大规模集群中可能会有很多PV，如果这些PV都需要运维手动来处理这也是一件很繁琐的事情，所以就有了动态供给概念，也就是Dynamic Provisioning。而我们上面的创建的PV都是静态供给方式，也就是Static Provisioning。
创建StorageClass里面需要定义PV属性比如存储类型、大小等；另外创建这种PV需要用到存储插件。最终效果是，用户提交PVC，里面指定存储类型，如果符合我们定义的StorageClass，则会为其自动创建PV并进行绑定。

Ceph 动态存储

Rook Ceph 环境搭建： https://www.cnblogs.com/vpc123/articles/14397126.html

# 进入客户端控制台
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
# 工具箱中的所有可用工具命令均已准备就绪，可满足您的故障排除需求
ceph status
ceph osd status
ceph df
rados df

1 Ceph 动态 StorageClass

storageclass.yaml 内容如下:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
    # clusterID is the namespace where the rook cluster is running
    clusterID: rook-ceph
    # Ceph pool into which the RBD image shall be created
    pool: replicapool

    # RBD image format. Defaults to "2".
    imageFormat: "2"

    # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
    imageFeatures: layering

    # The secrets contain Ceph admin credentials.
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

    # Specify the filesystem type of the volume. If not specified, csi-provisioner
    # will set default as `ext4`.
    csi.storage.k8s.io/fstype: xfs

# Delete the rbd volume when a PVC is deleted
reclaimPolicy: Delete

资源检查:

# 创建动态资源池
kubectl create -f storageclass.yaml
# 存储资源检查
[root@iZbp1dnzy9ygt2oosvzzzcZ ceph-demo]# kubectl -n rook-ceph get cephblockpools.ceph.rook.io
NAME          AGE
replicapool   38m
[root@iZbp1dnzy9ygt2oosvzzzcZ ceph-demo]# kubectl -n rook-ceph get storageclasses.storage.k8s.io
NAME                      PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)      rancher.io/local-path        Delete          WaitForFirstConsumer   false                  21h
rook-ceph-retain-bucket   ceph.rook.io/bucket          Retain          Immediate              false                  160m
rook-ceph-block           rook-ceph.rbd.csi.ceph.com   Delete          Immediate              false                  38m

2 Pod 实例

# 案例资源获取
wget https://raw.githubusercontent.com/rook/rook/release-1.2/cluster/examples/kubernetes/mysql.yaml
wget https://raw.githubusercontent.com/rook/rook/release-1.2/cluster/examples/kubernetes/wordpress.yam

# 资源部署
kubectl apply -f mysql.yaml
kubectl apply -f wordpress.yaml

# 查看存储资源
[root@iZbp1dnzy9ygt2oosvzzzcZ ceph-demo]# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
myclaim          Bound    pvc-d63e22c9-c47c-4659-b467-d690431fa76f   2Gi        RWO            rook-ceph-block   78m
mysql-pv-claim   Bound    pvc-6b792873-c281-4541-ba53-e7eb0e2b36f5   20Gi       RWO            rook-ceph-block   41m
wp-pv-claim      Bound    pvc-9de3bbb1-9567-4508-86c8-744eeb660c54   20Gi       RWO            rook-ceph-block   39m

3 资源清理

kubectl delete -f wordpress.yaml
kubectl delete -f mysql.yaml
kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapool
kubectl delete storageclass rook-ceph-block

存储解析

mysql.yaml 解析:

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  storageClassName: rook-ceph-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
    tier: mysql
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: changeme
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

1 查看动态 sc

[root@iZbp1dnzy9ygt2oosvzzzcZ ceph-demo]# kubectl get sc
NAME                      PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)      rancher.io/local-path        Delete          WaitForFirstConsumer   false                  21h
rook-ceph-retain-bucket   ceph.rook.io/bucket          Retain          Immediate              false                  171m
rook-ceph-block           rook-ceph.rbd.csi.ceph.com   Delete          Immediate              false                  49m

2 申请 pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  storageClassName: rook-ceph-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

2 绑定 pvc

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
    tier: mysql
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: changeme
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

拓展阅读

后续的文章将会从云原生数据持久化的各个方面进行展开，存储的落地探讨：