Kubernetes 部署 NFS server

环境要求

kubernetes：1.21+

部署 NFS server

下载 nfs server 部署清单

# wget https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/deploy/example/nfs-provisioner/nfs-server.yaml

添加 ns nfs

apiVersion: v1
kind: Namespace
metadata:
  name: nfs
---
kind: Service
apiVersion: v1
metadata:
  name: nfs-server
  namespace: nfs
  labels:
    app: nfs-server
spec:
  type: ClusterIP  # use "LoadBalancer" to get a public ip
  selector:
    app: nfs-server
  ports:
    - name: tcp-2049
      port: 2049
      protocol: TCP
    - name: udp-111
      port: 111
      protocol: UDP
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-server
  namespace: nfs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-server
  template:
    metadata:
      name: nfs-server
      labels:
        app: nfs-server
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: nfs-server
          image: itsthenetwork/nfs-server-alpine:latest
          env:
            - name: SHARED_DIRECTORY
              value: "/exports"
          volumeMounts:
            - mountPath: /exports
              name: nfs-vol
          securityContext:
            privileged: true
          ports:
            - name: tcp-2049
              containerPort: 2049
              protocol: TCP
            - name: udp-111
              containerPort: 111
              protocol: UDP
      volumes:
        - name: nfs-vol
          hostPath:
            path: /nfs-vol  # modify this to specify another path to store nfs share data
            type: DirectoryOrCreate

部署 nfs server

# kubectl apply -f nfs-server.yaml

namespace/nfs created
service/nfs-server created
deployment.apps/nfs-server created

查看 pod

# kubectl get pods -n nfs

NAME                          READY   STATUS    RESTARTS   AGE
nfs-server-7cc5bcdcd5-m7m7m   1/1     Running   0          35s

查看 svc

# kubectl get svc -n nfs

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
nfs-server   ClusterIP   10.100.157.133   <none>        2049/TCP,111/UDP   90s

部署 NFS CSI driver

下载部署脚本

# git clone https://github.com/kubernetes-csi/csi-driver-nfs.git
# cd csi-driver-nfs

更改 registry.k8s.io

# sed -i 's@registry.k8s.io/sig-storage@registry.aliyuncs.com/google_containers@g' deploy/v4.5.0/*.yaml

部署 nfs csi

# ./deploy/install-driver.sh v4.5.0 local

use local deploy
Installing NFS CSI driver, version: v4.5.0 ...
serviceaccount/csi-nfs-controller-sa created
serviceaccount/csi-nfs-node-sa created
clusterrole.rbac.authorization.k8s.io/nfs-external-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/nfs-csi-provisioner-binding created
csidriver.storage.k8s.io/nfs.csi.k8s.io created
deployment.apps/csi-nfs-controller created
daemonset.apps/csi-nfs-node created
NFS CSI driver installed successfully.

查看 pod 状态

csi-nfs-controller pod

# kubectl -n kube-system get pod -o wide -l app=csi-nfs-controller

NAME                                READY   STATUS    RESTARTS   AGE   IP                NODE          NOMINATED NODE   READINESS GATES
csi-nfs-controller-f4bd4f45-v89wl   4/4     Running   0          7s    192.168.174.106   k8s-node-01   <none>           <none>

csi-nfs-node pod

# kubectl -n kube-system get pod -o wide -l app=csi-nfs-node

NAME                 READY   STATUS    RESTARTS   AGE   IP                NODE            NOMINATED NODE   READINESS GATES
csi-nfs-node-b2dxs   3/3     Running   0          7s    192.168.174.100   k8s-master-01   <none>           <none>
csi-nfs-node-f2dct   3/3     Running   0          7s    192.168.174.107   k8s-node-02     <none>           <none>
csi-nfs-node-f8dbq   3/3     Running   0          7s    192.168.174.106   k8s-node-01     <none>           <none>
csi-nfs-node-pj5lb   3/3     Running   0          7s    192.168.174.108   k8s-node-03     <none>           <none>

卸载 NFS CSI driver

# ./deploy/uninstall-driver.sh v4.5.0 local

use local deploy
Uninstalling NFS driver, version: v4.5.0 ...
deployment.apps "csi-nfs-controller" deleted
daemonset.apps "csi-nfs-node" deleted
csidriver.storage.k8s.io "nfs.csi.k8s.io" deleted
serviceaccount "csi-nfs-controller-sa" deleted
serviceaccount "csi-nfs-node-sa" deleted
clusterrole.rbac.authorization.k8s.io "nfs-external-provisioner-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "nfs-csi-provisioner-binding" deleted
Uninstalled NFS driver successfully.

测试 NFS server

nginx-pod.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: nfs.csi.k8s.io
  name: pv-nginx
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  mountOptions:
    - nfsvers=4.1
  csi:
    driver: nfs.csi.k8s.io
    # volumeHandle format: {nfs-server-address}#{sub-dir-name}#{share-name}
    # make sure this value is unique for every share in the cluster
    volumeHandle: nfs-server.nfs.svc.wgs.local/share##
    volumeAttributes:
      server: nfs-server.nfs.svc.wgs.local
      share: /
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nginx
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-nginx
  storageClassName: ""
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-nfs-example
spec:
  containers:
    - image: nginx
      name: nginx
      ports:
        - containerPort: 80
          protocol: TCP
      volumeMounts:
        - mountPath: /var/www
          name: pvc-nginx
          readOnly: false
  volumes:
    - name: pvc-nginx
      persistentVolumeClaim:
        claimName: pvc-nginx

创建测试pod

# kubectl apply -f nginx-pod.yaml

persistentvolume/pv-nginx created
persistentvolumeclaim/pvc-nginx created
pod/nginx-nfs-example created

验证 NFS  server

# kubectl exec nginx-nfs-example -- bash -c "findmnt /var/www -o TARGET,SOURCE,FSTYPE"

TARGET   SOURCE                         FSTYPE
/var/www nfs-server.nfs.svc.wgs.local:/ nfs4

创建 storage class

storageclass-nfs.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-server.nfs.svc.wgs.local
  share: /
  # csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
  # csi.storage.k8s.io/provisioner-secret-name: "mount-options"
  # csi.storage.k8s.io/provisioner-secret-namespace: "default"
reclaimPolicy: Delete     # Retain
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1

创建 storageclass 

# kubectl apply -f storageclass-nfs.yaml

storageclass.storage.k8s.io/nfs-csi created

查看 storageclass 

# kubectl get StorageClass

NAME      PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi   nfs.csi.k8s.io   Retain          Immediate           false                  71s

创建 pvc

pvc-nfs-csi-dynamic.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-dynamic
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-csi

创建 pvc

# kubectl apply -f pvc-nfs-csi-dynamic.yaml

persistentvolumeclaim/pvc-nfs-dynamic created

查看 pvc

# kubectl get pvc

NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-nfs-dynamic   Bound    pvc-a42b5bd5-3dba-4cf2-b09c-de5d9952b699   1Gi        RWX            nfs-csi        59s

查看 pv

# kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-a42b5bd5-3dba-4cf2-b09c-de5d9952b699   1Gi        RWX            Retain           Bound    default/pvc-nfs-dynamic   nfs-csi                 117s

参考文档

https://github.com/kubernetes-csi/csi-driver-nfs