kubernetes（四）

一、基于StatefulSet部署有状态访问、基于DaemonSet在每一个node节点部署一个prometheus node-exporter

1.1 StatefulSet

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/statefulset/

StatefulSet为了解决有状态服务的集群部署、集群之间的数据同步问题（MySQL主从等）
StatefulSet所管理的pod拥有唯一且固定的pod名称
StatefulSet按照顺序对pod进行启停、伸缩和回收
Headless Service（无头服务，请求的解析直接解析到Pod IP）

1.1.1 编写StatefulSet

 apiVersion: apps/v1
kind: StatefulSet 
metadata:
  name: myserver-myapp
  namespace: myserver
spec:
  replicas: 3				# 默认值是 1
  serviceName: "myserver-myapp-service"	# service名称
  selector:
    matchLabels:
      app: myserver-myapp-frontend	# 必须匹配 .spec.template.metadata.labels
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend	# 必须匹配 .spec.selector.matchLabels
    spec:
      containers:
      - name: myserver-myapp-frontend
        image: harbor.chu.net/baseimages/nginx:1.20.0
        ports:
          - containerPort: 80
 
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-service
  namespace: myserver
spec:
  clusterIP: None		# 名为 myserver-myapp-service 的 Headless Service 用来控制网络域名。
  ports:
  - name: http
    port: 80
  selector:
    app: myserver-myapp-frontend

‍

查看

 kubectl apply -f statefulset.yaml
# statefulset信息
[root@k8s-deploy case12-Statefulset]#kubectl get sts -n myserver -owide
NAME             READY   AGE     CONTAINERS                IMAGES
myserver-myapp   3/3     2m54s   myserver-myapp-frontend   harbor.chu.net/baseimages/nginx:1.20.0
 
# service信息
[root@k8s-deploy case12-Statefulset]#kubectl get svc -n myserver -owide
NAME                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
myserver-myapp-service   ClusterIP   None         <none>        80/TCP    2m47s   app=myserver-myapp-frontend
 
# service对应后端pod ip：端口
[root@k8s-deploy case12-Statefulset]#kubectl get ep -n myserver
NAME                     ENDPOINTS                                             AGE
myserver-myapp-service   10.200.107.222:80,10.200.169.161:80,10.200.36.80:80   7m45s
 
# pod名称唯一，且按顺序命名
[root@k8s-deploy case12-Statefulset]#kubectl get pod -n myserver -owide
NAME               READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
myserver-myapp-0   1/1     Running   0          2m58s   10.200.107.222   10.0.0.43   <none>           <none>
myserver-myapp-1   1/1     Running   0          2m56s   10.200.169.161   10.0.0.42   <none>           <none>
myserver-myapp-2   1/1     Running   0          2m54s   10.200.36.80     10.0.0.41   <none>           <none>
 
# 按顺序创建pod
[root@k8s-deploy yaml]#kubectl describe sts myserver-myapp -n myserver
......
Events:
  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  2m    statefulset-controller  create Pod myserver-myapp-0 in StatefulSet myserver-myapp successful
  Normal  SuccessfulCreate  2m    statefulset-controller  create Pod myserver-myapp-1 in StatefulSet myserver-myapp successful
  Normal  SuccessfulCreate  2m    statefulset-controller  create Pod myserver-myapp-2 in StatefulSet myserver-myapp successful

1.1.2 验证测试

 # 创建测试pod
kubectl run net-test1 --image=alpine sleep 10000 -n myserver
 
# 进入pod测试网络，ping无头服务myserver-myapp-service，直接解析后端pod IP
[root@k8s-deploy yaml]#kubectl exec -it net-test1 -n myserver sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # ping myserver-myapp-service
PING myserver-myapp-service (10.200.169.161): 56 data bytes
64 bytes from 10.200.169.161: seq=0 ttl=62 time=0.401 ms
64 bytes from 10.200.169.161: seq=1 ttl=62 time=0.389 ms
 
/ # ping myserver-myapp-service
PING myserver-myapp-service (10.200.36.80): 56 data bytes
64 bytes from 10.200.36.80: seq=0 ttl=62 time=0.287 ms
64 bytes from 10.200.36.80: seq=1 ttl=62 time=0.290 ms
 
/ # ping myserver-myapp-service
PING myserver-myapp-service (10.200.107.222): 56 data bytes
64 bytes from 10.200.107.222: seq=0 ttl=63 time=0.057 ms
64 bytes from 10.200.107.222: seq=1 ttl=63 time=0.101 ms

伸缩测试

 # 缩减pod数量时，从最后往前缩减
[root@k8s-deploy yaml]#kubectl scale sts myserver-myapp -n myserver --replicas=2
statefulset.apps/myserver-myapp scaled
[root@k8s-deploy yaml]#kubectl get pod -n myserver
NAME               READY   STATUS    RESTARTS   AGE
myserver-myapp-0   1/1     Running   0          32m
myserver-myapp-1   1/1     Running   0          32m
 
 
# 增加pod数量时，依次往后增加
[root@k8s-deploy yaml]#kubectl scale sts myserver-myapp -n myserver --replicas=4
statefulset.apps/myserver-myapp scaled
[root@k8s-deploy yaml]#kubectl get pod -n myserver
NAME               READY   STATUS              RESTARTS   AGE
myserver-myapp-0   1/1     Running             0          32m
myserver-myapp-1   1/1     Running             0          32m
myserver-myapp-2   1/1     Running             0          2s
myserver-myapp-3   0/1     ContainerCreating   0          1s

‍

1.2 DaemonSet

https://kubernetes.io/zh-cn/docs/concepts/workloads/controllers/daemonset/

DaemonSet在当前集群中每个节点运行同一个pod，当有新的节点加入集群时也会为新的节点配置相同的pod，当节点从集群移除时其pod也会被kubernetes回收，删除DaemonSet控制器时将删除其创建的所有pod

DaemonSet 的一些典型用法：

在每个节点上运行集群守护进程
在每个节点上运行日志收集守护进程
在每个节点上运行监控守护进程

1.2.1 编写DaemonSet

 apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring 
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
        k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      tolerations:
        # 这些容忍度设置是为了让该守护进程集在控制平面节点上运行
        # 如果不希望自己的控制平面节点运行 Pod，可以删除它们
        - effect: NoSchedule
          key: node-role.kubernetes.io/master		#执行kubectl get node --show-labels命令，可查看节点label
      containers:
      - image: harbor.chu.net/baseimages/prom-node-exporter:v1.3.1 
        imagePullPolicy: IfNotPresent			# 只有当镜像在本地不存在时才会拉取。
        name: prometheus-node-exporter
        ports:
        - containerPort: 9100
          hostPort: 9100
          protocol: TCP
          name: metrics
        volumeMounts:
        - mountPath: /host/proc
          name: proc
        - mountPath: /host/sys
          name: sys
        - mountPath: /host
          name: rootfs
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host
      volumes:
        # 将宿主机目录挂载至pod中
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /
      hostNetwork: true		# 使用宿主机网络
      hostPID: true
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring 
spec:
  type: NodePort
  ports:
  - name: http
    port: 9100
    nodePort: 30022
    protocol: TCP
  selector:
    k8s-app: node-exporter

执行创建命令

 # 先创建namespace
kubectl create ns monitoring
kubectl apply -f daemonset.yaml

‍

1.2.2 验证测试

 [root@k8s-deploy yaml]#kubectl get daemonset -n monitoring
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   6         6         6       6            6           <none>          106s
 
[root@k8s-deploy yaml]#kubectl get pod -n monitoring -owide
NAME                  READY   STATUS    RESTARTS   AGE    IP          NODE        NOMINATED NODE   READINESS GATES
node-exporter-77lmq   1/1     Running   0          2m2s   10.0.0.13   10.0.0.13   <none>           <none>
node-exporter-jf6xz   1/1     Running   0          2m2s   10.0.0.11   10.0.0.11   <none>           <none>
node-exporter-k6z4z   1/1     Running   0          2m2s   10.0.0.41   10.0.0.41   <none>           <none>
node-exporter-mpfwv   1/1     Running   0          2m2s   10.0.0.43   10.0.0.43   <none>           <none>
node-exporter-xtkfs   1/1     Running   0          2m2s   10.0.0.12   10.0.0.12   <none>           <none>
node-exporter-zzszd   1/1     Running   0          2m2s   10.0.0.42   10.0.0.42   <none>           <none>

每个节点都部署了一个node-exporter pod

浏览器访问

 [root@k8s-deploy yaml]#kubectl get svc -n monitoring -owide
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE     SELECTOR
node-exporter   NodePort   10.100.70.108   <none>        9100:30022/TCP   4m19s   k8s-app=node-exporter

‍

二、熟悉pod的常见状态及故障原因

https://kubernetes.io/zh-cn/docs/concepts/workloads/pods/pod-lifecycle/

pod调度流程

‍

pod常见状态及原因如下：

Unschedulabel

pod不能被调度，kube-scheduler没有匹配到合适的node节点
PodScheduler

pod正处于调度中，在kube-scheduler刚开始调度的时候，还没有将pod分配到指定的node，在筛选出合适的节点后就会更新etcd数据，将pod分配到指定的pod
Pending

正在创建Pod但是pod中的容器还没有全部被创建完成，处于此状态的Pod应该检查Pod依赖的存储是否有权限挂载等。
Failed

Pod中有容器启动失败而导致pod工作异常
Unknown

由于某种原因而无法获得pod的当前状态，通常是由于与pod所在的node节点通信错误
Initialized

所有pod中的初始化容器已经完成了
ImagePullBackOff

Pod所在的node节点下载镜像失败
Running

Pod内部的容器已经被创建并且启动
Ready

表示pod中的容器已经可以提供访问服务
Error

pod启动过程发生错误
NodeLost

pod所在节点失联
Waiting

pod等待启动
Terminating

pod正在被销毁
CrashLoopBackOff

pod之前已启动，因异常退出，但是kubelet正在将它重启
InvaliImageName

node节点无法解析镜像名称，导致镜像无法下载
ImageInspectError

无法校验镜像，镜像不完整导致
ErrImageNeverPull

策略紧张拉取镜像，镜像中心权限是私有等
RegistryUnavailable

镜像服务器不可用，网络原因或harbor宕机
ErrImagePull

镜像拉取出错，超时或下载被强制终止
CreateContainerConfigError

不能创建kebelet使用的容器配置
CreateContainerError

创建容器失败
RunContainerError

pod运行失败，容器中没有初始化pid为1的守护进程等
ContainersNotInitialized

pod没有初始化完成
ContainersNotReady

pod没有准备完毕
ContainerCreating

pod正在创建中
PodInitializing

pod正在初始化中
DockeDaemonNotReady

node节点docker服务没有启动
NetworkPluginNotReady

网络插件没有启动

‍

三、熟练使用startupProbe、livenessProbe、readinessProbe探针对pod进行状态监测

pod的生命周期

3.1 探针简介

https://kubernetes.io/zh-cn/docs/concepts/workloads/pods/pod-lifecycle/#container-probes

https://kubernetes.io/zh-cn/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

probe（探针）是由 kubelet 对容器执行的定期诊断，以保证pod的状态始终处于运行状态，要执行诊断，kubelet 既可以在容器内执行代码，也可以发出一个网络请求。

3.1.1 检查机制

使用探针来检查容器有四种不同的方法。

exec

在容器内执行指定命令。如果命令退出时返回码为 0 则认为诊断成功。

grpc

使用 gRPC 执行一个远程过程调用。目标应该实现 gRPC健康检查。如果响应的状态是 "SERVING"，则认为诊断成功。 gRPC 探针是一个 Alpha 特性，只有在启用了 "GRPCContainerProbe" 特性门控时才能使用。

httpGet

对容器的 IP 地址上指定端口和路径执行 HTTP GET 请求。如果响应的状态码大于等于 200 且小于 400，则诊断被认为是成功的。

tcpSocket

对容器的 IP 地址上的指定端口执行 TCP 检查。如果端口打开，则诊断被认为是成功的。

‍

3.1.2 探测结果

每次探测都将获得以下三种结果之一：

Success（成功）

容器通过了诊断。
Failure（失败）

容器未通过诊断。
Unknown（未知）

诊断失败，因此不会采取任何行动。

‍

3.1.3 探测类型

针对运行中的容器，kubelet 可以选择是否执行以下三种探针，以及如何针对探测结果作出反应：

livenessProbe（存活探针）

指示容器是否正在运行。如果存活态探测失败，则 kubelet 会杀死容器，并且容器将根据其重启策略决定未来。如果容器不提供存活探针，则默认状态为 Success。

readinessProbe（就绪探针）

指示容器是否准备好为请求提供服务。如果就绪态探测失败，端点控制器将从与 Pod 匹配的所有服务的端点列表中删除该 Pod 的 IP 地址。初始延迟之前的就绪态的状态值默认为 Failure。如果容器不提供就绪态探针，则默认状态为 Success。

startupProbe（启动探针）

指示容器中的应用是否已经启动。如果提供了启动探针，则所有其他探针都会被禁用，直到此探针成功为止。如果启动探测失败，kubelet 将杀死容器，而容器依其重启策略进行重启。如果容器没有提供启动探测，则默认状态为 Success。

‍

3.2 配置探针

https://kubernetes.io/zh-cn/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

探针配置字段说明：

initialDelaySeconds：容器启动后要等待多少秒后才启动启动、存活和就绪探针，默认是 0 秒，最小值是 0。
periodSeconds：执行探测的时间间隔（单位是秒）。默认是 10 秒。最小值是 1。
timeoutSeconds：探测的超时后等待多少秒。默认值是 1 秒。最小值是 1。
successThreshold：探针在失败后，被视为成功的最小连续成功数。默认值是 1。存活和启动探测的这个值必须是 1。最小值是 1。
failureThreshold：探针连续失败了 failureThreshold 次之后， Kubernetes 认为总体上检查已失败：容器状态未就绪、不健康、不活跃。对于启动探针或存活探针而言，如果至少有 failureThreshold 个探针已失败， Kubernetes 会将容器视为不健康并为这个特定的容器触发重启操作。 kubelet 会考虑该容器的 terminationGracePeriodSeconds 设置。对于失败的就绪探针，kubelet 继续运行检查失败的容器，并继续运行更多探针；因为检查失败，kubelet 将 Pod 的 Ready 状况设置为 false。
terminationGracePeriodSeconds：为 kubelet 配置从为失败的容器触发终止操作到强制容器运行时停止该容器之前等待的宽限时长。默认值是继承 Pod 级别的 terminationGracePeriodSeconds 值（如果不设置则为 30 秒），最小值为 1。

‍

3.2.1 livenessProbe

编写livenessProbe

 apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp-frontend-deployment
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: myserver-myapp-frontend-label
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend-label
    spec:
      containers:
      - name: myserver-myapp-frontend-label
        image: harbor.chu.net/baseimages/nginx:1.20.0
        ports:
        - containerPort: 80
        #readinessProbe:
        livenessProbe:			# 存活探针
          httpGet:			# 检测方式：http
            path: /index.html		# 访问http服务的路径
            port: 80			# 访问容器的端口号或端口名，gRPC 探测不能使用命名端口或定制主机。
          initialDelaySeconds: 5
          periodSeconds: 3
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
 
 
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-frontend-service
  namespace: myserver
spec:
  ports:
  - name: http
    port: 81
    targetPort: 80
    nodePort: 30023
    protocol: TCP
  type: NodePort
  selector:
    app: myserver-myapp-frontend-label

验证

查看状态

 [root@k8s-deploy case3-Probe]#kubectl get svc -n myserver
NAME                              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
myserver-myapp-frontend-service   NodePort   10.100.120.93   <none>        81:30023/TCP   95s
 
 
# pod状态running
[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
NAME                                                  READY   STATUS    RESTARTS        AGE
myserver-myapp-frontend-deployment-777fbb9c56-wnf49   1/1     Running   0               8s

进入容器，删除index.html文件

 # 删除index.html文件，
[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-777fbb9c56-wnf49 -n myserver bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@myserver-myapp-frontend-deployment-777fbb9c56-wnf49:/# rm /usr/share/nginx/html/index.html 
# 存活探测失败，kubelet 会杀死容器，退出当前连接
root@myserver-myapp-frontend-deployment-777fbb9c56-wnf49:/# command terminated with exit code 137
 
# pod自动重启1次，状态恢复正常
[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
NAME                                                  READY   STATUS    RESTARTS        AGE
myserver-myapp-frontend-deployment-777fbb9c56-wnf49   1/1     Running   1 (4s ago)      2m13s
 
# pod重启后，容器恢复初始化状态，丢失的index.html文件恢复正常。
[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-777fbb9c56-wnf49 -n myserver bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@myserver-myapp-frontend-deployment-777fbb9c56-wnf49:/# ls /usr/share/nginx/html/index.html 
/usr/share/nginx/html/index.html

存活探针检测失败，自动重启pod，将pod恢复初始状态

‍

3.2.2 readlinessProbe

编写readlinessProbe

 apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp-frontend-deployment
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: myserver-myapp-frontend-label
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend-label
    spec:
      containers:
      - name: myserver-myapp-frontend-label
        image: harbor.chu.net/baseimages/nginx:1.20.0
        ports:
        - containerPort: 80
        readinessProbe:			# 就绪探针
        #livenessProbe:
          httpGet:
            path: /index.html
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 3
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
 
 
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-frontend-service
  namespace: myserver
spec:
  ports:
  - name: http
    port: 81
    targetPort: 80
    nodePort: 30023
    protocol: TCP
  type: NodePort
  selector:
    app: myserver-myapp-frontend-label

验证

查看状态

service后端pod的ip：port为10.200.169.165:80

 [root@k8s-deploy case3-Probe]#kubectl get svc -n myserver
NAME                              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
myserver-myapp-frontend-service   NodePort   10.100.198.37   <none>        81:30023/TCP   7s
 
# endpoint地址
[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
NAME                              ENDPOINTS           AGE
myserver-myapp-frontend-service   10.200.169.165:80   14s

进入容器，删除index.html文件

 # 进入容器，删除http检测的index.html文件
[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-f68bdc86d-rcgbp  -n myserver bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@myserver-myapp-frontend-deployment-f68bdc86d-rcgbp:/# mv /usr/share/nginx/html/index.html /tmp
 
# 查看endpoint地址为空
[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
NAME                              ENDPOINTS   AGE
myserver-myapp-frontend-service               59s
 
# pod 中容器就绪状态数量为0
[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
NAME                                                 READY   STATUS    RESTARTS      AGE
myserver-myapp-frontend-deployment-f68bdc86d-rcgbp   0/1     Running   0             65s
 
# 网页无法正常访问
[root@k8s-deploy case3-Probe]#curl 10.0.0.42:30023
curl: (7) Failed to connect to 10.0.0.42 port 30023: Connection refused

就绪探针检测失败，自动将service后端的endpoint地址移除，web服务不可用

恢复容器index.html文件

 root@myserver-myapp-frontend-deployment-f68bdc86d-rcgbp:/# echo 'readinessProbe test' >> /usr/share/nginx/html/index.html
 
# pod状态恢复就绪
[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
NAME                                                 READY   STATUS    RESTARTS      AGE
myserver-myapp-frontend-deployment-f68bdc86d-rcgbp   1/1     Running   0             111s
 
# endpoint地址恢复正常
[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
NAME                              ENDPOINTS           AGE
myserver-myapp-frontend-service   10.200.169.165:80   113s
 
# 网页访问恢复正常
[root@k8s-deploy case3-Probe]#curl 10.0.0.42:30023
readinessProbe test

就绪探针检测成功，自动将service后端的endpoint地址添加回来，网页访问恢复正常

‍

3.2.3 startupProbe

创建yaml文件

 apiVersion: apps/v1
kind: Deployment
metadata:
  name: myserver-myapp-frontend-deployment
  namespace: myserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myserver-myapp-frontend-label
  template:
    metadata:
      labels:
        app: myserver-myapp-frontend-label
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: myserver-myapp-frontend-label
        image: nginx:1.20.2
        ports:
        - containerPort: 80
        startupProbe:
          httpGet:
            path: /my-index.html	# 启动检测my-index.html
            port: 80
          initialDelaySeconds: 5	#首次检测延迟5s
          failureThreshold: 3		#从成功转为失败的次数
          periodSeconds: 3		#探测间隔周期
        readinessProbe:
          httpGet:
            path: /index.html
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 3
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /index.html
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 3
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
 
 
---
apiVersion: v1
kind: Service
metadata:
  name: myserver-myapp-frontend-service
  namespace: myserver
spec:
  ports:
  - name: http
    port: 81
    targetPort: 80
    nodePort: 30023
    protocol: TCP
  type: NodePort
  selector:
    app: myserver-myapp-frontend-label

添加启动、存活、就绪探针，启动探针检测成功后存活探针和就绪探针才会正常检测（存活探针、就绪探针检测无先后顺序）。

‍

验证

查看状态

 [root@k8s-deploy case3-Probe]#kubectl get svc -n myserver
NAME                              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
myserver-myapp-frontend-service   NodePort   10.100.115.71   <none>        81:30023/TCP   52s
 
# pod服务就绪数量0
[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
NAME                                                 READY   STATUS    RESTARTS      AGE
myserver-myapp-frontend-deployment-f9774f8b8-m85br   0/1     Running   2 (3s ago)    54s
 
# service无后端地址
[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
NAME                              ENDPOINTS   AGE
myserver-myapp-frontend-service   <none>      59s
 
# 启动探针检测失败
[root@k8s-deploy case3-Probe]#kubectl describe pod myserver-myapp-frontend-deployment-f9774f8b8-m85br -n myserver
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  23s                default-scheduler  Successfully assigned myserver/myserver-myapp-frontend-deployment-7cfc55c668-jskcs to 10.0.0.42
  Normal   Pulled     11s (x2 over 22s)  kubelet            Container image "nginx:1.20.2" already present on machine
  Normal   Created    11s (x2 over 22s)  kubelet            Created container myserver-myapp-frontend-label
  Normal   Started    11s (x2 over 22s)  kubelet            Started container myserver-myapp-frontend-label
  Normal   Killing    11s                kubelet            Container myserver-myapp-frontend-label failed startup probe, will be restarted
  Warning  Unhealthy  2s (x5 over 17s)   kubelet            Startup probe failed: HTTP probe failed with statuscode: 404

启动探针检测my-index.html文件失败，pod服务未就绪

‍

添加文件

 # 添加my-index.html文件
[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-f9774f8b8-lb2jb -n myserver bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@myserver-myapp-frontend-deployment-f9774f8b8-lb2jb:/# cp /usr/share/nginx/html/index.html /usr/share/nginx/html/my-index.html
 
# ready数量为1
[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
NAME                                                 READY   STATUS    RESTARTS      AGE
myserver-myapp-frontend-deployment-f9774f8b8-lb2jb   1/1     Running   0             36s
net-test1                                            1/1     Running   2 (30m ago)   6h4m
 
[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
NAME                              ENDPOINTS           AGE
myserver-myapp-frontend-service   10.200.169.166:80   40s
 
# 获取http响应码
[root@k8s-deploy yaml]#curl -Is 10.0.0.42:30023/my-index.html|grep HTTP
HTTP/1.1 200 OK

立即进入容器添加my-index.html文件，启动探针检测文件成功，pod服务就绪，存活探针、就绪探针正常进行检测。

‍

四、掌握基于nerdctl + buildkitd构建容器镜像

4.1 部署buildkitd

https://github.com/moby/buildkit

buildkitd组成部分

 buildkitd(服务端)，支持runc和containerd作为镜像构建环境，默认是runc，可以更换为containerd
buildctl(客户端)，负责解析Dockerfile文件，并向服务端buildkitd发送构建请求

主机

 k8s-master2 10.0.0.12 buildkitd服务端（镜像构建服务器）

安装buildkitd

  cd /usr/local/src
# 下载并解tar
wget https://github.com/moby/buildkit/releases/download/v0.11.0/buildkit-v0.11.0.linux-amd64.tar.gz
tar xvf buildkit-v0.11.0.linux-amd64.tar.gz
mv bin/buildctl bin/buildkitd /usr/local/bin/
 
# 准备buildkit socket
cat >> /lib/systemd/system/buildkit.socket <<EOF
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
 
[Socket]
ListenStream=%t/buildkit/buildkitd.sock
 
[Install]
WantedBy=sockets.target
EOF
 
# 准备buildkit service
cat >> /lib/systemd/system/buildkit.service <<EOF
[Unit]
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socket
Documentation=https://github.com/moby/buildkit
 
[Service]
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true
 
[Install]
WantedBy=multi-user.target
EOF
 
# 启动服务
systemctl daemon-reload
systemctl enable buildkit
systemctl start buildkit

查看buildkit服务状态

 [root@k8s-master2 src]#systemctl is-active buildkit.service 
active

4.2 部署nerdctl

 # 安装nerdctl
wget https://github.com/containerd/nerdctl/releases/download/v1.0.0/nerdctl-1.0.0-linux-amd64.tar.gz
tar -xvf nerdctl-1.0.0-linux-amd64.tar.gz
mv nerdctl /usr/bin/
 
# 安装CNI插件
wget https://github.com/containernetworking/plugins/releases/download/v1.1.0/cni-plugins-linux-amd64-v1.1.0.tgz
mkdir -p /opt/cni/bin
tar -xvf cni-plugins-linux-amd64-v1.1.0.tgz -C /opt/cni/bin
 
# 配置nerdctl命令补全
source <(nerdctl completion bash)
echo "source <(nerdctl completion bash)" >> ~/.bashrc

上传镜像测试

 # 登录harbor仓库，添加参数--insecure-registry
nerdctl login --insecure-registry harbor.chu.net
#下载镜像
nerdctl pull centos:7.9.2009
# 打tag
nerdctl tag centos:7.9.2009 harbor.chu.net/baseimages/centos:7.9.2009
# 上传镜像至harbor仓库，添加--insecure-registry参数
nerdctl --insecure-registry push harbor.chu.net/baseimages/centos:7.9.2009

‍

4.3 分发harbor证书

 # 镜像构建服务器创建harbor域名目录
[root@k8s-master2 ~]#mkdir -p /etc/containerd/certs.d/harbor.chu.net
 
# harbor证书分发
## 格式转换
[root@harbor1 harbor]#cd /apps/harbor/certs/
[root@harbor1 certs]#openssl x509 -inform PEM -in chu.net.crt -out chu.net.cert 
[root@harbor1 certs]#ls
ca.crt  ca.key  ca.srl  chu.net.cert  chu.net.crt  chu.net.csr  chu.net.key  v3.ext
 
## 复制证书至镜像构建服务器
[root@harbor1 certs]# scp ca.crt chu.net.cert chu.net.key 10.0.0.12:/etc/containerd/certs.d/harbor.chu.net/
 
## 查看镜像服务器证书
[root@k8s-master2 ~]#ls /etc/containerd/certs.d/harbor.chu.net/
ca.crt  chu.net.cert  chu.net.key

登录harbor

 [root@k8s-master2 ~]#nerdctl login harbor.chu.net
Enter Username: admin
Enter Password: 
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 
Login Succeeded

‍

4.4 构建镜像

4.4.1 编写Dockerfile

 FROM ubuntu:22.04
ADD sources.list /etc/apt/sources.list
RUN apt update && apt  install -y iproute2  ntpdate  tcpdump telnet traceroute nfs-kernel-server nfs-common  lrzsz tree  openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute  gcc openssh-server lrzsz tree  openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump telnet traceroute iotop unzip zip make
 
ADD nginx-1.22.0.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.22.0 && ./configure --prefix=/apps/nginx && make && make install  && ln -s /apps/nginx/sbin/nginx /usr/bin
RUN groupadd  -g 2088 nginx && useradd  -g nginx -s /usr/sbin/nologin -u 2088 nginx && chown -R nginx:nginx /apps/nginx
ADD nginx.conf /apps/nginx/conf/
ADD frontend.tar.gz /apps/nginx/html/
 
 
EXPOSE 80 443
#ENTRYPOINT ["nginx"]
CMD ["nginx","-g","daemon off;"]

4.4.2 准备文件

 [root@k8s-master2 ubuntu]#ls
Dockerfile  frontend.tar.gz  nginx-1.22.0.tar.gz  nginx.conf  sources.list

nginx源码

 # 下载nginx
wget http://nginx.org/download/nginx-1.20.2.tar.gz

nginx.conf配置文件

 worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    upstream tomcat {
        server 10.0.0.101:8080;
        server 10.0.0.102:8080;
    }
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
	location /myapp {
           proxy_pass http://tomcat;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

source.list（清华镜像源）

 # 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
# deb-sr https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
# deb-sr https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
 
# 预发布软件源，不建议启用
# deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse
# deb-src http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse

frontend.tar.gz（web文件）

 [root@k8s-master2 ubuntu]#tar tf frontend.tar.gz 
./
./images/
./images/1.jpg
./index.html

4.4.3 构建本地镜像

 # 构建本地镜像
nerdctl build -t harbor.chu.net/baseimages/nginx:1.20.2 .

构建流程

 [root@k8s-master2 ubuntu]#nerdctl build -t harbor.chu.net/baseimages/nginx:1.20.2 .
[+] Building 156.6s (12/13)                                                                                                                                                                                 
[+] Building 156.7s (13/13) FINISHED                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                                     0.0s
 => => transferring dockerfile: 885B                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                                                          7.4s
 => [1/8] FROM docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a91d098f7f3f9285415a9                                                                                   16.3s
 => => resolve docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a91d098f7f3f9285415a9                                                                                    0.0s
 => => sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797 30.43MB / 30.43MB                                                                                                        14.7s
 => => extracting sha256:6e3729cf69e0ce2de9e779575a1fec8b7fb5efdfa822829290ab6d5d1bc3e797                                                                                                                1.5s
 => [internal] load build context                                                                                                                                                                        0.1s
 => => transferring context: 1.12MB                                                                                                                                                                      0.0s
 => [2/8] ADD sources.list /etc/apt/sources.list                                                                                                                                                         0.2s
 => [3/8] RUN apt update && apt  install -y iproute2  ntpdate  tcpdump telnet traceroute nfs-kernel-server nfs-common  lrzsz tree  openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev ntpdate tcpdump  64.4s 
 => [4/8] ADD nginx-1.22.0.tar.gz /usr/local/src/                                                                                                                                                        0.3s 
 => [5/8] RUN cd /usr/local/src/nginx-1.22.0 && ./configure --prefix=/apps/nginx && make && make install  && ln -sv /apps/nginx/sbin/nginx /usr/bin                                                     32.3s 
 => [6/8] RUN groupadd  -g 2088 nginx && useradd  -g nginx -s /usr/sbin/nologin -u 2088 nginx && chown -R nginx.nginx /apps/nginx                                                                        0.3s 
 => [7/8] ADD nginx.conf /apps/nginx/conf/                                                                                                                                                               0.1s 
 => [8/8] ADD frontend.tar.gz /apps/nginx/html/                                                                                                                                                          0.1s 
 => exporting to docker image format                                                                                                                                                                    35.3s 
 => => exporting layers                                                                                                                                                                                 23.0s 
 => => exporting manifest sha256:1fd16223228c63b12abf40319de112e5bd5cad4f713466109f75b581ae9a9ee5                                                                                                        0.0s
 => => exporting config sha256:bc338d3676045d6e665338bff41a59be13abe417a62a4b4d7e360659ea7b2277                                                                                                          0.0s
 => => sending tarball                                                                                                                                                                                  12.3s
Loaded image: harbor.chu.net/baseimages/nginx:1.20.2

4.4.4 镜像测试

 # 创建容器，需确认安装CNI插件
[root@k8s-master2 bin]#nerdctl run -d -p 80:80 harbor.chu.net/baseimages/nginx:1.20.2
a3e0cfc5118bdc7aa067ff9402d2db36cd409a750bd4230388c80ba2e8fc9d12
 
[root@k8s-master2 bin]#nerdctl ps
CONTAINER ID    IMAGE                                     COMMAND                   CREATED           STATUS    PORTS                 NAMES
a3e0cfc5118b    harbor.chu.net/baseimages/nginx:1.20.2    "nginx -g daemon off;"    16 seconds ago    Up        0.0.0.0:80->80/tcp    nginx-a3e0c

浏览器访问

4.4.5 上传镜像

 # 上传至harbor仓库
nerdctl push harbor.chu.net/baseimages/nginx:1.20.2

上传过程

 [root@k8s-master2 ~]#nerdctl push harbor.chu.net/baseimages/nginx:1.20.2
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:1fd16223228c63b12abf40319de112e5bd5cad4f713466109f75b581ae9a9ee5) 
manifest-sha256:1fd16223228c63b12abf40319de112e5bd5cad4f713466109f75b581ae9a9ee5: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:bc338d3676045d6e665338bff41a59be13abe417a62a4b4d7e360659ea7b2277:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 8.5 s                                                                    total:  5.2 Ki (623.0 B/s)

查看harbor仓库

‍

4.5 nginx代理harbor（https）

存在harbor自签名证书不被信任问题，解决方法为：将证书从harbor迁移至负载均衡服务器上，负载均衡服务至harbor为http。

‍

主机

 harbor1 10.0.0.101 VIP 10.0.0.100
harbor2 10.0.0.102 VIP 10.0.0.100

‍

4.5.1 harbor修改http协议

 [root@harbor1 ~]#cd /apps/harbor/
# 将https配置注释
[root@harbor1 harbor]#vim harbor.yml
...
#https:
#  # https port for harbor, default is 443
#  port: 443
#  # The path of cert and key files for nginx
#  certificate: /apps/harbor/certs/chu.net.crt
#  private_key: /apps/harbor/certs/chu.net.key
 
# 更新配置，重启docker-compose
[root@harbor harbor]#docker-compose stop	#或者docker-compose down -v
[root@harbor harbor]#./prepare 
[root@harbor harbor]#docker-compose start	#或者docker-compose up -d

4.5.2 nginx实现https

keepalived配置方法参考：
https://www.cnblogs.com/areke/p/16520746.html#:~:text=keeplived 结合haproxy 实现高可用

安装nginx

 mkdir -p /apps/nginx
apt install -y make
cd /usr/local/src/
wget http://nginx.org/download/nginx-1.22.1.tar.gz
tar -xvf nginx-1.22.1.tar.gz
cd nginx-1.22.1
./configure --prefix=/apps/nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module
 
make && make install

准备证书

 # 创建证书目录，并从harbor服务器上复制证书
mkdir -p /apps/nginx/certs
scp 10.0.0.101:/apps/harbor/certs/chu.net.crt 10.0.0.101:/apps/harbor/certs/chu.net.key /apps/nginx/certs/

配置https文件

 [root@k8s-ha1 nginx]#cat conf/nginx.conf|egrep -v '^\s*#|^$'
worker_processes  auto;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    client_max_body_size 1000m;
    upstream harbor {
        ip_hash;
        server 10.0.0.101;
        server 10.0.0.102;
    }
    server {
        listen       10.0.0.100:80;
        listen       10.0.0.100:443 ssl;
        ssl_certificate      /apps/nginx/certs/chu.net.crt;
        ssl_certificate_key  /apps/nginx/certs/chu.net.key;
        ssl_session_cache    shared:SSL:20m;
        ssl_session_timeout  10m;
        server_name  harbor.chu.net;
 
        location / {
            proxy_pass http://harbor;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

配置启动服务

 cat > /usr/lib/systemd/system/nginx.service <<EOF
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
 
[Service]
Type=forking
PIDFile=/apps/nginx/logs/nginx.pid
ExecStartPre=/bin/rm -f /apps/nginx/logs/nginx.pid
ExecStartPre=/apps/nginx//sbin/nginx -t
ExecStart=/apps/nginx/sbin/nginx -c /apps/nginx/conf/nginx.conf
ExecReload=/bin/kill -s HUP \$MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target
EOF

启动服务

 systemctl daemon-reload
systemctl enable --now nginx

‍

4.5.3 浏览器访问

http

‍

https

4.5.4 配置buildkitd

支持http

 mkdir -p /etc/buildkit
cat >>/etc/buildkit/buildkitd.toml <<EOF
[registry."harbor.chu.net"]
  http = true
  insecure = true
EOF
 
# 重启服务
systemctl restart buildkit.service

4.5.5 配置nerdctl

配置默认namespace，支持http方式

 mkdir -p /etc/nerdctl
cat >>/etc/nerdctl/nerdctl.toml <<EOF
namespace = "k8s.io"
debug = false
debug_full = false
insecure_registry = true
EOF

4.5.6 自定义构建镜像验证

 # 下载镜像
nerdctl pull harbor.chu.net/baseimages/nginx:1.20.2
 
# 编写Dockerfile
cat >Dockerfile <<EOF 
FROM harbor.chu.net/baseimages/nginx:1.20.2
 
CMD ["tail","-f","/etc/hosts"]
EOF
 
# 构建镜像
nerdctl build -t harbor.chu.net/test/nginx:v11 .

构建过程

 [root@k8s-master2 dockerfile]#nerdctl build -t harbor.chu.net/test/nginx:v11 .
[+] Building 12.4s (5/5)                                                                                                                                                                                  
[+] Building 12.5s (5/5) FINISHED                                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                     0.0s
 => => transferring dockerfile: 113B                                                                                                                                                                     0.0s
 => [internal] load metadata for harbor.chu.net/baseimages/nginx:1.20.2                                                                                                                                  0.0s
 => CACHED [1/1] FROM harbor.chu.net/baseimages/nginx:1.20.2@sha256:1fd16223228c63b12abf40319de112e5bd5cad4f713466109f75b581ae9a9ee5                                                                     0.0s
 => => resolve harbor.chu.net/baseimages/nginx:1.20.2@sha256:1fd16223228c63b12abf40319de112e5bd5cad4f713466109f75b581ae9a9ee5                                                                            0.0s
 => exporting to docker image format                                                                                                                                                                    12.3s
 => => exporting layers                                                                                                                                                                                  0.0s
 => => exporting manifest sha256:0ed650b501158236ab9a4c4563d697bda491790a49b7da3632496ce159ac108c                                                                                                        0.0s
 => => exporting config sha256:652aabc714544426d2b4f2c55b04e181a5533c27337393c80a7b37a53100c761                                                                                                          0.0s
 => => sending tarball                                                                                                                                                                                  12.3s
Loaded image: harbor.chu.net/test/nginx:v11

上传镜像至harbor仓库

 [root@k8s-master2 dockerfile]#nerdctl push harbor.chu.net/test/nginx:v11
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:d730a1e993a94b80a861e4e10990cdf11a14bf8c2a89d41ef023898da0a1f197) 
WARN[0000] skipping verifying HTTPS certs for "harbor.chu.net" 
manifest-sha256:d730a1e993a94b80a861e4e10990cdf11a14bf8c2a89d41ef023898da0a1f197: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:c0c6da3dad5aca02318f6d335a0f707655b31c7b5380b9e9f0d32d2d6f46cadf:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.8 s                                                                    total:  7.9 Ki (4.4 KiB/s)

登录网页查看

‍

五、自定义镜像运行Nginx及Java服务并基于NFS实现动静分离

服务流程

‍

业务镜像规划

‍

系统基础镜像制作

编写Dockerfile

 #自定义Centos 基础镜像
FROM centos:7.9.2009
MAINTAINER Areke arekestone@gmail.com
 
# filebeat日志收集：https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.1-x86_64.rpm
ADD filebeat-7.12.1-x86_64.rpm /tmp
RUN yum install -y /tmp/filebeat-7.12.1-x86_64.rpm vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop &&  rm -rf /etc/localtime /tmp/filebeat-7.12.1-x86_64.rpm && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime  && useradd nginx -u 2088

构建镜像，并上传至本地harbor仓库

 nerdctl build -t harbor.chu.net/baseimages/centos-base:7.9.2009 .
nerdctl push harbor.chu.net/baseimages/centos-base:7.9.2009

‍

5.1 Tomcat

5.1.1 JDK基础镜像制作

编写Dockerfile

 #JDK Base Image
FROM harbor.chu.net/baseimages/centos-base:7.9.2009
 
ADD jdk-8u212-linux-x64.tar.gz /usr/local/src/
RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk 
ADD profile /etc/profile
 
ENV JAVA_HOME /usr/local/jdk
ENV JRE_HOME $JAVA_HOME/jre
ENV CLASSPATH $JAVA_HOME/lib/:$JRE_HOME/lib/
ENV PATH $PATH:$JAVA_HOME/bin

构建镜像，并上传至harbor仓库

 nerdctl build -t harbor.chu.net/baseimages/jdk-base:v8.212 .
nerdctl push harbor.chu.net/baseimages/jdk-base:v8.212

‍

5.1.2 tomcat基础镜像制作

编写Dockerfile

 #Tomcat 8.5.43基础镜像
FROM harbor.chu.net/baseimages/jdk-base:v8.212
 
RUN mkdir -p /apps /data/tomcat/webapps /data/tomcat/logs
ADD apache-tomcat-8.5.43.tar.gz  /apps
RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat && chown -R tomcat:tomcat /apps /data

构建镜像，并上传至harbor仓库

 nerdctl build -t harbor.chu.net/baseimages/tomcat-base:v8.5.43 .
nerdctl push harbor.chu.net/baseimages/tomcat-base:v8.5.43

5.1.3 tomcat业务镜像app1制作

编写Dockerfile

 #tomcat web1
FROM harbor.chu.net/baseimages/tomcat-base:v8.5.43
 
ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
ADD app1.tar.gz /data/tomcat/webapps/myapp/
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
 
RUN chown -R nginx:nginx /data/ /apps/ && chmod a+x /apps/tomcat/bin/catalina.sh /apps/tomcat/bin/run_tomcat.sh
 
EXPOSE 8080 8443
 
CMD ["/apps/tomcat/bin/run_tomcat.sh"]

构建镜像，并上传至harbor仓库

 nerdctl build -t harbor.chu.net/web/tomcat-app1:v1 .
nerdctl push harbor.chu.net/web/tomcat-app1:v1

5.1.4 在kubernetes环境运行tomcat

提前创建好namespace：web

 kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: web-tomcat-app1-deployment-label
  name: web-tomcat-app1-deployment
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-tomcat-app1-selector
  template:
    metadata:
      labels:
        app: web-tomcat-app1-selector
    spec:
      containers:
      - name: web-tomcat-app1-container
        image: harbor.chu.net/web/tomcat-app1:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
 
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: web-tomcat-app1-service-label
  name: web-tomcat-app1-service
  namespace: web
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30030
  selector:
    app: web-tomcat-app1-selector

访问测试

 [root@k8s-master2 tomcat-app1]#kubectl get pod -n web
NAME                                          READY   STATUS    RESTARTS   AGE
web-tomcat-app1-deployment-5587f96995-x25d6   1/1     Running   0          14s
[root@k8s-master2 tomcat-app1]#kubectl get svc -n web
NAME                      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
web-tomcat-app1-service   NodePort   10.100.185.255   <none>        80:30030/TCP   48s

‍

5.2 Nginx

5.2.1 nginx基础镜像制作

编写Dockerfile

 FROM harbor.chu.net/baseimages/centos-base:7.9.2009
 
RUN yum install -y vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop
ADD nginx-1.22.0.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.22.0 && ./configure  && make && make install && ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx  &&rm -rf /usr/local/src/nginx-1.22.0.tar.gz

构建镜像，并上传至本地harbor仓库

 nerdctl build -t harbor.chu.net/baseimages/nginx-base:1.22.0 .
nerdctl push harbor.chu.net/baseimages/nginx-base:1.22.0

5.2.2 nginx业务镜像制作

编写Dockerfile

 FROM harbor.chu.net/baseimages/nginx-base:1.22.0
 
ADD nginx.conf /usr/local/nginx/conf/nginx.conf
ADD app1.tar.gz  /usr/local/nginx/html/webapp/
ADD index.html  /usr/local/nginx/html/index.html
 
#静态资源挂载路径
RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images
EXPOSE 80 443
 
CMD ["nginx"]

nginx.conf

 [root@k8s-master2 nginx]#egrep -v '^\s*#|^$' nginx.conf 
user  nginx nginx;
worker_processes  auto;
daemon off;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    upstream  tomcat_webserver {
        server  web-tomcat-app1-service.web.svc.cluster.local:80; 
    }
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        location /webapp {
            root   html;
            index  index.html index.htm;
        }
        location /myapp {
             proxy_pass  http://tomcat_webserver;
             proxy_set_header   Host    $host;
             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X-Real-IP $remote_addr;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

app1.tar.gz

 [root@k8s-master2 webapp]#cat index.html 
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Devops</title>
</head>
<body>
<h1>web devops v11111111</h1>
</body>
</html>
 
[root@k8s-master2 webapp]#tar czvf app1.tar.gz index.html

index.html

 [root@k8s-master2 nginx]#cat index.html 
nginx web1 v1

构建镜像，并上传至本地harbor仓库

 nerdctl build -t harbor.chu.net/web/nginx-web1:v1 .
nerdctl push harbor.chu.net/web/nginx-web1:v1

5.2.3 业务镜像测试

 [root@k8s-master2 nginx]#nerdctl run -it -p 80:80 harbor.chu.net/web/nginx-web1:v1 bash
# 单机测试无法识别service名称，需添加hosts域名解析
[root@9d2ce970a25c /]# echo "127.0.0.1 web-tomcat-app1-service.web.svc.cluster.local" >> /etc/hosts
[root@9d2ce970a25c /]# nginx &
[1] 68
 
# 访问默认网页
[root@9d2ce970a25c /]# curl 127.0.0.1
nginx web1 v1
 
# 访问webapp
[root@9d2ce970a25c /]# curl -L 127.0.0.1/webapp
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Devops</title>
</head>
<body>
<h1>web devops v11111111</h1>
</body>
</html>

5.2.4 在kubernetes中实现nginx+tomcat动静分离

编写测试yaml

 kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: web-nginx-deployment-label
  name: web-nginx-deployment
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-nginx-selector
  template:
    metadata:
      labels:
        app: web-nginx-selector
    spec:
      containers:
      - name: web-nginx-container
        image: harbor.chu.net/web/nginx-web1:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
        - containerPort: 443
          protocol: TCP
          name: https
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "20"
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: web-nginx-app1-service-label
  name: web-nginx-app1-service
  namespace: web
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30031
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30032
  selector:
    app: web-nginx-selector

验证测试

 [root@k8s-master2 nginx]#kubectl get svc -n web
NAME                      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
web-nginx-app1-service    NodePort   10.100.141.39   <none>        80:30031/TCP,443:30032/TCP   12s
web-tomcat-app1-service   NodePort   10.100.16.87    <none>        80:30030/TCP                 22s
 
[root@k8s-master2 nginx]#kubectl get ep -n web
NAME                      ENDPOINTS                              AGE
web-nginx-app1-service    10.200.169.175:443,10.200.169.175:80   16s
web-tomcat-app1-service   10.200.107.233:8080                    26s
 
[root@k8s-master2 nginx]#kubectl get pod -n web
NAME                                          READY   STATUS    RESTARTS   AGE
web-nginx-deployment-65c95bc6bf-vr2rs         1/1     Running   0          42s
web-tomcat-app1-deployment-5587f96995-ktzzx   1/1     Running   0          52s
[root@k8s-master2 nginx]#

浏览器访问

访问默认网页

访问webapp/

静态文件

访问myapp/

动态网页跳转tomcat程序处理

index.jsp测试文件内容

 <!DOCTYPE html>
<html>
<head>
    <title>JSP - Hello World</title>
</head>
<body>
<h1><%= "Hello World!" %>
</h1>
<br/>
<a href="/tomcatTest">Hello Servlet</a>
</body>
</html>

5.3 NFS

5.3.1 基于NFS实现数据共享

NFS服务器（10.0.0.101）配置

 # 创建目录
mkdir -p /data/k8sdata/web/images /data/k8sdata/web/static
# 添加配置
cat /etc/exports
...
/data/k8sdata *(rw,sync,no_root_squash,no_subtree_check)
 
# 配置生效
[root@harbor1 data]#exportfs -r
# 查看挂载状态
[root@harbor1 data]#showmount -e 10.0.0.101
Export list for 10.0.0.101:
/data/k8sdata *

tomcat业务添加NFS

 kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: web-tomcat-app1-deployment-label
  name: web-tomcat-app1-deployment
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-tomcat-app1-selector
  template:
    metadata:
      labels:
        app: web-tomcat-app1-selector
    spec:
      containers:
      - name: web-tomcat-app1-container
        image: harbor.chu.net/web/tomcat-app1:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
        #resources:
        #  limits:
        #    cpu: 1
        #    memory: "512Mi"
        #  requests:
        #    cpu: 500m
        #    memory: "512Mi"
        volumeMounts:
        - name: web-images
          mountPath: /usr/local/nginx/html/webapp/images
          readOnly: false
        - name: web-static
          mountPath: /usr/local/nginx/html/webapp/static
          readOnly: false
      volumes:
      - name: web-images
        nfs:
          server: 10.0.0.101			# NFS服务器
          path: /data/k8sdata/web/images
      - name: web-static
        nfs:
          server: 10.0.0.101
          path: /data/k8sdata/web/static
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: web-tomcat-app1-service-label
  name: web-tomcat-app1-service
  namespace: web
spec:
  #type: NodePort			# 隐藏宿主机端口
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    #nodePort: 30030			# 隐藏宿主机端口
  selector:
    app: web-tomcat-app1-selector

nginx服务添加NFS

 kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: web-nginx-deployment-label
  name: web-nginx-deployment
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-nginx-selector
  template:
    metadata:
      labels:
        app: web-nginx-selector
    spec:
      containers:
      - name: web-nginx-container
        image: harbor.chu.net/web/nginx-web1:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
        - containerPort: 443
          protocol: TCP
          name: https
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "20"
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
        volumeMounts:
        - name: web-images
          mountPath: /usr/local/nginx/html/webapp/images
          readOnly: false
        - name: web-static
          mountPath: /usr/local/nginx/html/webapp/static
          readOnly: false
      volumes:
      - name: web-images
        nfs:
          server: 10.0.0.101
          path: /data/k8sdata/web/images 
      - name: web-static
        nfs:
          server: 10.0.0.101
          path: /data/k8sdata/web/static
 
  
 
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: web-nginx-app1-service-label
  name: web-nginx-app1-service
  namespace: web
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30031
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30032
  selector:
    app: web-nginx-selector

查看

 [root@k8s-master2 tomcat-app1]#kubectl get svc -n web
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
web-nginx-app1-service    NodePort    10.100.70.126    <none>        80:30031/TCP,443:30032/TCP   19m
web-tomcat-app1-service   ClusterIP   10.100.240.205   <none>        80/TCP                       31m

配置负载均衡

增加keepalived VIP 10.0.0.35

 cat /etc/keepalived/keepalived.conf
...
    virtual_ipaddress {
        10.0.0.10/24 dev eth0 label eth0:0
        10.0.0.35/24 dev eth0 label eth0:2
    }

配置haproxy

 cat /etc/haproxy/haproxy.cfg
...
# 添加如下配置
listen web_nginx_http_80
    bind 10.0.0.35:80
    mode tcp
    server 10.0.0.11 10.0.0.11:30031 check inter 3s fall 3 rise 5
    server 10.0.0.12 10.0.0.12:30031 check inter 3s fall 3 rise 5
    server 10.0.0.13 10.0.0.13:30031 check inter 3s fall 3 rise 5
    server 10.0.0.41 10.0.0.41:30031 check inter 3s fall 3 rise 5
    server 10.0.0.42 10.0.0.42:30031 check inter 3s fall 3 rise 5
    server 10.0.0.43 10.0.0.43:30031 check inter 3s fall 3 rise 5
 
 
listen web_nginx_https_443
    bind 10.0.0.35:443
    mode tcp
    #balance source
    server 10.0.0.11 10.0.0.11:30032 check inter 3s fall 3 rise 5
    server 10.0.0.12 10.0.0.12:30032 check inter 3s fall 3 rise 5
    server 10.0.0.13 10.0.0.13:30032 check inter 3s fall 3 rise 5
    server 10.0.0.41 10.0.0.41:30032 check inter 3s fall 3 rise 5
    server 10.0.0.42 10.0.0.42:30032 check inter 3s fall 3 rise 5
    server 10.0.0.43 10.0.0.43:30032 check inter 3s fall 3 rise 5

5.3.2 在后端服务生成数据并验证访问

准备数据

 # 生成数据至NFS:/data/k8sdata/web/images、static目录下
# 下载图片至/data/k8sdata/web/images目录下
[root@harbor1 static]#echo 'web static test' > /data/k8sdata/web/static/1.html
 
[root@harbor1 static]#tree /data/k8sdata/web/
/data/k8sdata/web/
├── images
│   └── 1.jpg
└── static
    └── 1.html
 
2 directories, 2 files

访问验证

访问默认页面

跳转tomcat页面

访问后端NFS图片

访问后端NFS网页

‍

六、运行zookeeper集群

6.1 构建镜像

准备java基础镜像

 nerdctl pull elevy/slim_java:8
nerdctl tag elevy/slim_java:8 harbor.chu.net/baseimages/slim_java:8
nerdctl push harbor.chu.net/baseimages/slim_java:8

编写Dockerfile

 FROM harbor.chu.net/baseimages/slim_java:8
 
ENV ZK_VERSION 3.4.14
ADD repositories /etc/apk/repositories 
# Download Zookeeper
COPY zookeeper-3.4.14.tar.gz /tmp/zk.tgz
COPY zookeeper-3.4.14.tar.gz.asc /tmp/zk.tgz.asc
COPY KEYS /tmp/KEYS
RUN apk add --no-cache --virtual .build-deps \
      ca-certificates   \
      gnupg             \
      tar               \
      wget &&           \
    #
    # Install dependencies
    apk add --no-cache 	\
      bash &&           \
    #
    #
    # Verify the signature
    export GNUPGHOME="$(mktemp -d)" && \
    gpg -q --batch --import /tmp/KEYS && \
    gpg -q --batch --no-auto-key-retrieve --verify /tmp/zk.tgz.asc /tmp/zk.tgz && \
    #
    # Set up directories
    #
    mkdir -p /zookeeper/data /zookeeper/wal /zookeeper/log && \
    #
    # Install
    tar -x -C /zookeeper --strip-components=1 --no-same-owner -f /tmp/zk.tgz && \
    #
    # Slim down
    cd /zookeeper && \
    cp dist-maven/zookeeper-${ZK_VERSION}.jar . && \
    rm -rf \
      *.txt \
      *.xml \
      bin/README.txt \
      bin/*.cmd \
      conf/* \
      contrib \
      dist-maven \
      docs \
      lib/*.txt \
      lib/cobertura \
      lib/jdiff \
      recipes \
      src \
      zookeeper-*.asc \
      zookeeper-*.md5 \
      zookeeper-*.sha1 && \
    #
    # Clean up
    apk del .build-deps && \
    rm -rf /tmp/* "$GNUPGHOME"
 
COPY conf /zookeeper/conf/
COPY bin/zkReady.sh /zookeeper/bin/
COPY entrypoint.sh /
RUN  chmod a+x /zookeeper/bin/zkReady.sh /entrypoint.sh
 
ENV PATH=/zookeeper/bin:${PATH} \
    ZOO_LOG_DIR=/zookeeper/log \
    ZOO_LOG4J_PROP="INFO, CONSOLE, ROLLINGFILE" \
    JMXPORT=9010
 
ENTRYPOINT [ "/entrypoint.sh" ]
 
CMD [ "zkServer.sh", "start-foreground" ]
 
EXPOSE 2181 2888 3888 9010

执行构建

 nerdctl build -t harbor.chu.net/web/zookeeper:v3.4.14 .
nerdctl push harbor.chu.net/web/zookeeper:v3.4.14

测试镜像

 bash-4.3# /zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 9010
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Using config: /zookeeper/bin/../conf/zoo.cfg
Mode: follower

‍

6.2 创建PV

NFS服务器（10.0.0.101）创建目录

 mkdir -p /data/k8sdata/web/zookeeper-datadir-1
mkdir -p /data/k8sdata/web/zookeeper-datadir-2
mkdir -p /data/k8sdata/web/zookeeper-datadir-3

编写yaml文件

 apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-1
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce 
  nfs:
    server: 10.0.0.101
    path: /data/k8sdata/web/zookeeper-datadir-1 
 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-2
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 10.0.0.101
    path: /data/k8sdata/web/zookeeper-datadir-2 
 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: zookeeper-datadir-pv-3
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 10.0.0.101 
    path: /data/k8sdata/web/zookeeper-datadir-3

查看

 [root@k8s-master2 pv]#kubectl get pv
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
zookeeper-datadir-pv-1   20Gi       RWO            Retain           Available                                   14s
zookeeper-datadir-pv-2   20Gi       RWO            Retain           Available                                   14s
zookeeper-datadir-pv-3   20Gi       RWO            Retain           Available                                   14s

6.3 创建PVC

 apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-1
  namespace: web
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-1
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-2
  namespace: web
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-2
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zookeeper-datadir-pvc-3
  namespace: web
spec:
  accessModes:
    - ReadWriteOnce
  volumeName: zookeeper-datadir-pv-3
  resources:
    requests:
      storage: 10Gi

查看

 [root@k8s-master2 pv]#kubectl get pvc -n web
NAME                      STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
zookeeper-datadir-pvc-1   Bound    zookeeper-datadir-pv-1   20Gi       RWO                           18s
zookeeper-datadir-pvc-2   Bound    zookeeper-datadir-pv-2   20Gi       RWO                           17s
zookeeper-datadir-pvc-3   Bound    zookeeper-datadir-pv-3   20Gi       RWO                           17s

6.4 创建zookeeper集群

 apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  namespace: web
spec:
  ports:
    - name: client
      port: 2181
  selector:
    app: zookeeper
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper1
  namespace: web
spec:
  type: NodePort    
  ports:
    - name: client
      port: 2181
      nodePort: 32181
    - name: followers
      port: 2888
    - name: election
      port: 3888
  selector:
    app: zookeeper
    server-id: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper2
  namespace: web
spec:
  type: NodePort    
  ports:
    - name: client
      port: 2181
      nodePort: 32182
    - name: followers
      port: 2888
    - name: election
      port: 3888
  selector:
    app: zookeeper
    server-id: "2"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper3
  namespace: web
spec:
  type: NodePort    
  ports:
    - name: client
      port: 2181
      nodePort: 32183
    - name: followers
      port: 2888
    - name: election
      port: 3888
  selector:
    app: zookeeper
    server-id: "3"
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper1
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
        server-id: "1"
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: wal
          emptyDir:
            medium: Memory
      containers:
        - name: server
          image: harbor.chu.net/web/zookeeper:v3.4.14
          imagePullPolicy: Always
          env:
            - name: MYID
              value: "1"
            - name: SERVERS
              value: "zookeeper1,zookeeper2,zookeeper3"
            - name: JVMFLAGS
              value: "-Xmx2G"
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          volumeMounts:
          - mountPath: "/zookeeper/data"
            name: zookeeper-datadir-pvc-1 
      volumes:
        - name: zookeeper-datadir-pvc-1 
          persistentVolumeClaim:
            claimName: zookeeper-datadir-pvc-1
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper2
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
        server-id: "2"
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: wal
          emptyDir:
            medium: Memory
      containers:
        - name: server
          image: harbor.chu.net/web/zookeeper:v3.4.14
          imagePullPolicy: Always
          env:
            - name: MYID
              value: "2"
            - name: SERVERS
              value: "zookeeper1,zookeeper2,zookeeper3"
            - name: JVMFLAGS
              value: "-Xmx2G"
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          volumeMounts:
          - mountPath: "/zookeeper/data"
            name: zookeeper-datadir-pvc-2 
      volumes:
        - name: zookeeper-datadir-pvc-2
          persistentVolumeClaim:
            claimName: zookeeper-datadir-pvc-2
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper3
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
        server-id: "3"
    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: wal
          emptyDir:
            medium: Memory
      containers:
        - name: server
          image: harbor.chu.net/web/zookeeper:v3.4.14
          imagePullPolicy: Always
          env:
            - name: MYID
              value: "3"
            - name: SERVERS
              value: "zookeeper1,zookeeper2,zookeeper3"
            - name: JVMFLAGS
              value: "-Xmx2G"
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          volumeMounts:
          - mountPath: "/zookeeper/data"
            name: zookeeper-datadir-pvc-3
      volumes:
        - name: zookeeper-datadir-pvc-3
          persistentVolumeClaim:
           claimName: zookeeper-datadir-pvc-3

‍

6.5 验证集群状态

查看service

 [root@k8s-master2 zookeeper]#kubectl get svc -n web
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
zookeeper                 ClusterIP   10.100.100.250   <none>        2181/TCP                                       53s
zookeeper1                NodePort    10.100.56.175    <none>        2181:32181/TCP,2888:57732/TCP,3888:62659/TCP   53s
zookeeper2                NodePort    10.100.233.22    <none>        2181:32182/TCP,2888:55598/TCP,3888:51327/TCP   53s
zookeeper3                NodePort    10.100.38.10     <none>        2181:32183/TCP,2888:56595/TCP,3888:51777/TCP   53s

查看pod、logs等状态

 [root@k8s-master2 zookeeper]#kubectl get pod -n web
NAME                                          READY   STATUS    RESTARTS   AGE
zookeeper1-56679f8f44-xh2hh                   1/1     Running   0          34s
zookeeper2-5cd9f77979-xbsmt                   1/1     Running   0          34s
zookeeper3-5b75f6546b-d6k8q                   1/1     Running   0          34s

查看zookeeper容器状态

 # 进入容器，查看集群状态
[root@k8s-master2 zookeeper]#kubectl exec -it zookeeper1-56679f8f44-xh2hh -n web bash
bash-4.3# /zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 9010
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Using config: /zookeeper/bin/../conf/zoo.cfg
Mode: follower			#集群状态，leader主、follower从
   
bash-4.3# cat /zookeeper/conf/zoo.cfg 
tickTime=10000				# 通信心跳数，建议设置10000（毫秒）
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

数据写入

 #! /bin/env python
# kazoo 2.9.0版本有问题，安装指定版本号kazoo==2.8.0
from kazoo.client import KazooClient
 
# 服务器地址、端口号
zk = KazooClient(hosts='10.0.0.41:32183')
 
# 创建连接
zk.start()
 
# makepath=True递归创建目录
zk.create('/web/kafka/nodes/id-1', b'10.0.0.41', makepath=True)
 
# 查看所有数据
node = zk.get_children('/')
print(node)
 
# 关闭连接
zk.stop()

‍

登录ZooInspector图形工具查看

查看写入数据

posted @ 2023-02-02 17:32 areke 阅读(187) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· docker

· kubernetes（三）

· 云原生第四周--kubernetes组件详解(中)

· 云原生学习笔记-DAY4

· 使用探针对pod进行状态监测

阅读排行：
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布：重大改进与新特性概览！
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 单线程的Redis速度为什么快？

公告

昵称： areke
园龄： 3年4个月
粉丝： 5
关注： 0

+加关注

2025年3月

日

一

二

三

四

五

六

	apiVersion: apps/v1
	kind: StatefulSet
	metadata:
	name: myserver-myapp
	namespace: myserver
	spec:
	replicas: 3 # 默认值是 1
	serviceName: "myserver-myapp-service" # service名称
	selector:
	matchLabels:
	app: myserver-myapp-frontend # 必须匹配 .spec.template.metadata.labels
	template:
	metadata:
	labels:
	app: myserver-myapp-frontend # 必须匹配 .spec.selector.matchLabels
	spec:
	containers:
	- name: myserver-myapp-frontend
	image: harbor.chu.net/baseimages/nginx:1.20.0
	ports:
	- containerPort: 80

	---
	apiVersion: v1
	kind: Service
	metadata:
	name: myserver-myapp-service
	namespace: myserver
	spec:
	clusterIP: None # 名为 myserver-myapp-service 的 Headless Service 用来控制网络域名。
	ports:
	- name: http
	port: 80
	selector:
	app: myserver-myapp-frontend

	kubectl apply -f statefulset.yaml
	# statefulset信息
	[root@k8s-deploy case12-Statefulset]#kubectl get sts -n myserver -owide
	NAME READY AGE CONTAINERS IMAGES
	myserver-myapp 3/3 2m54s myserver-myapp-frontend harbor.chu.net/baseimages/nginx:1.20.0

	# service信息
	[root@k8s-deploy case12-Statefulset]#kubectl get svc -n myserver -owide
	NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
	myserver-myapp-service ClusterIP None <none> 80/TCP 2m47s app=myserver-myapp-frontend

	# service对应后端pod ip：端口
	[root@k8s-deploy case12-Statefulset]#kubectl get ep -n myserver
	NAME ENDPOINTS AGE
	myserver-myapp-service 10.200.107.222:80,10.200.169.161:80,10.200.36.80:80 7m45s

	# pod名称唯一，且按顺序命名
	[root@k8s-deploy case12-Statefulset]#kubectl get pod -n myserver -owide
	NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
	myserver-myapp-0 1/1 Running 0 2m58s 10.200.107.222 10.0.0.43 <none> <none>
	myserver-myapp-1 1/1 Running 0 2m56s 10.200.169.161 10.0.0.42 <none> <none>
	myserver-myapp-2 1/1 Running 0 2m54s 10.200.36.80 10.0.0.41 <none> <none>

	# 按顺序创建pod
	[root@k8s-deploy yaml]#kubectl describe sts myserver-myapp -n myserver
	......
	Events:
	Type Reason Age From Message
	---- ------ ---- ---- -------
	Normal SuccessfulCreate 2m statefulset-controller create Pod myserver-myapp-0 in StatefulSet myserver-myapp successful
	Normal SuccessfulCreate 2m statefulset-controller create Pod myserver-myapp-1 in StatefulSet myserver-myapp successful
	Normal SuccessfulCreate 2m statefulset-controller create Pod myserver-myapp-2 in StatefulSet myserver-myapp successful

	# 创建测试pod
	kubectl run net-test1 --image=alpine sleep 10000 -n myserver

	# 进入pod测试网络，ping无头服务myserver-myapp-service，直接解析后端pod IP
	[root@k8s-deploy yaml]#kubectl exec -it net-test1 -n myserver sh
	kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
	/ # ping myserver-myapp-service
	PING myserver-myapp-service (10.200.169.161): 56 data bytes
	64 bytes from 10.200.169.161: seq=0 ttl=62 time=0.401 ms
	64 bytes from 10.200.169.161: seq=1 ttl=62 time=0.389 ms

	/ # ping myserver-myapp-service
	PING myserver-myapp-service (10.200.36.80): 56 data bytes
	64 bytes from 10.200.36.80: seq=0 ttl=62 time=0.287 ms
	64 bytes from 10.200.36.80: seq=1 ttl=62 time=0.290 ms

	/ # ping myserver-myapp-service
	PING myserver-myapp-service (10.200.107.222): 56 data bytes
	64 bytes from 10.200.107.222: seq=0 ttl=63 time=0.057 ms
	64 bytes from 10.200.107.222: seq=1 ttl=63 time=0.101 ms

	# 缩减pod数量时，从最后往前缩减
	[root@k8s-deploy yaml]#kubectl scale sts myserver-myapp -n myserver --replicas=2
	statefulset.apps/myserver-myapp scaled
	[root@k8s-deploy yaml]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-0 1/1 Running 0 32m
	myserver-myapp-1 1/1 Running 0 32m


	# 增加pod数量时，依次往后增加
	[root@k8s-deploy yaml]#kubectl scale sts myserver-myapp -n myserver --replicas=4
	statefulset.apps/myserver-myapp scaled
	[root@k8s-deploy yaml]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-0 1/1 Running 0 32m
	myserver-myapp-1 1/1 Running 0 32m
	myserver-myapp-2 1/1 Running 0 2s
	myserver-myapp-3 0/1 ContainerCreating 0 1s

	apiVersion: apps/v1
	kind: DaemonSet
	metadata:
	name: node-exporter
	namespace: monitoring
	labels:
	k8s-app: node-exporter
	spec:
	selector:
	matchLabels:
	k8s-app: node-exporter
	template:
	metadata:
	labels:
	k8s-app: node-exporter
	spec:
	tolerations:
	# 这些容忍度设置是为了让该守护进程集在控制平面节点上运行
	# 如果不希望自己的控制平面节点运行 Pod，可以删除它们
	- effect: NoSchedule
	key: node-role.kubernetes.io/master #执行kubectl get node --show-labels命令，可查看节点label
	containers:
	- image: harbor.chu.net/baseimages/prom-node-exporter:v1.3.1
	imagePullPolicy: IfNotPresent # 只有当镜像在本地不存在时才会拉取。
	name: prometheus-node-exporter
	ports:
	- containerPort: 9100
	hostPort: 9100
	protocol: TCP
	name: metrics
	volumeMounts:
	- mountPath: /host/proc
	name: proc
	- mountPath: /host/sys
	name: sys
	- mountPath: /host
	name: rootfs
	args:
	- --path.procfs=/host/proc
	- --path.sysfs=/host/sys
	- --path.rootfs=/host
	volumes:
	# 将宿主机目录挂载至pod中
	- name: proc
	hostPath:
	path: /proc
	- name: sys
	hostPath:
	path: /sys
	- name: rootfs
	hostPath:
	path: /
	hostNetwork: true # 使用宿主机网络
	hostPID: true
	---
	apiVersion: v1
	kind: Service
	metadata:
	annotations:
	prometheus.io/scrape: "true"
	labels:
	k8s-app: node-exporter
	name: node-exporter
	namespace: monitoring
	spec:
	type: NodePort
	ports:
	- name: http
	port: 9100
	nodePort: 30022
	protocol: TCP
	selector:
	k8s-app: node-exporter

	# 先创建namespace
	kubectl create ns monitoring
	kubectl apply -f daemonset.yaml

	[root@k8s-deploy yaml]#kubectl get daemonset -n monitoring
	NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
	node-exporter 6 6 6 6 6 <none> 106s

	[root@k8s-deploy yaml]#kubectl get pod -n monitoring -owide
	NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
	node-exporter-77lmq 1/1 Running 0 2m2s 10.0.0.13 10.0.0.13 <none> <none>
	node-exporter-jf6xz 1/1 Running 0 2m2s 10.0.0.11 10.0.0.11 <none> <none>
	node-exporter-k6z4z 1/1 Running 0 2m2s 10.0.0.41 10.0.0.41 <none> <none>
	node-exporter-mpfwv 1/1 Running 0 2m2s 10.0.0.43 10.0.0.43 <none> <none>
	node-exporter-xtkfs 1/1 Running 0 2m2s 10.0.0.12 10.0.0.12 <none> <none>
	node-exporter-zzszd 1/1 Running 0 2m2s 10.0.0.42 10.0.0.42 <none> <none>

	[root@k8s-deploy yaml]#kubectl get svc -n monitoring -owide
	NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
	node-exporter NodePort 10.100.70.108 <none> 9100:30022/TCP 4m19s k8s-app=node-exporter

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: myserver-myapp-frontend-deployment
	namespace: myserver
	spec:
	replicas: 1
	selector:
	matchLabels:
	app: myserver-myapp-frontend-label
	template:
	metadata:
	labels:
	app: myserver-myapp-frontend-label
	spec:
	containers:
	- name: myserver-myapp-frontend-label
	image: harbor.chu.net/baseimages/nginx:1.20.0
	ports:
	- containerPort: 80
	#readinessProbe:
	livenessProbe: # 存活探针
	httpGet: # 检测方式：http
	path: /index.html # 访问http服务的路径
	port: 80 # 访问容器的端口号或端口名，gRPC 探测不能使用命名端口或定制主机。
	initialDelaySeconds: 5
	periodSeconds: 3
	timeoutSeconds: 5
	successThreshold: 1
	failureThreshold: 3


	---
	apiVersion: v1
	kind: Service
	metadata:
	name: myserver-myapp-frontend-service
	namespace: myserver
	spec:
	ports:
	- name: http
	port: 81
	targetPort: 80
	nodePort: 30023
	protocol: TCP
	type: NodePort
	selector:
	app: myserver-myapp-frontend-label

	[root@k8s-deploy case3-Probe]#kubectl get svc -n myserver
	NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
	myserver-myapp-frontend-service NodePort 10.100.120.93 <none> 81:30023/TCP 95s


	# pod状态running
	[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-frontend-deployment-777fbb9c56-wnf49 1/1 Running 0 8s

	# 删除index.html文件，
	[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-777fbb9c56-wnf49 -n myserver bash
	kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
	root@myserver-myapp-frontend-deployment-777fbb9c56-wnf49:/# rm /usr/share/nginx/html/index.html
	# 存活探测失败，kubelet 会杀死容器，退出当前连接
	root@myserver-myapp-frontend-deployment-777fbb9c56-wnf49:/# command terminated with exit code 137

	# pod自动重启1次，状态恢复正常
	[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-frontend-deployment-777fbb9c56-wnf49 1/1 Running 1 (4s ago) 2m13s

	# pod重启后，容器恢复初始化状态，丢失的index.html文件恢复正常。
	[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-777fbb9c56-wnf49 -n myserver bash
	kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
	root@myserver-myapp-frontend-deployment-777fbb9c56-wnf49:/# ls /usr/share/nginx/html/index.html
	/usr/share/nginx/html/index.html

	# 进入容器，删除http检测的index.html文件
	[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-f68bdc86d-rcgbp -n myserver bash
	kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
	root@myserver-myapp-frontend-deployment-f68bdc86d-rcgbp:/# mv /usr/share/nginx/html/index.html /tmp

	# 查看endpoint地址为空
	[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
	NAME ENDPOINTS AGE
	myserver-myapp-frontend-service 59s

	# pod 中容器就绪状态数量为0
	[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-frontend-deployment-f68bdc86d-rcgbp 0/1 Running 0 65s

	# 网页无法正常访问
	[root@k8s-deploy case3-Probe]#curl 10.0.0.42:30023
	curl: (7) Failed to connect to 10.0.0.42 port 30023: Connection refused

	root@myserver-myapp-frontend-deployment-f68bdc86d-rcgbp:/# echo 'readinessProbe test' >> /usr/share/nginx/html/index.html

	# pod状态恢复就绪
	[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-frontend-deployment-f68bdc86d-rcgbp 1/1 Running 0 111s

	# endpoint地址恢复正常
	[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
	NAME ENDPOINTS AGE
	myserver-myapp-frontend-service 10.200.169.165:80 113s

	# 网页访问恢复正常
	[root@k8s-deploy case3-Probe]#curl 10.0.0.42:30023
	readinessProbe test

	# 添加my-index.html文件
	[root@k8s-deploy yaml]#kubectl exec -it myserver-myapp-frontend-deployment-f9774f8b8-lb2jb -n myserver bash
	kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
	root@myserver-myapp-frontend-deployment-f9774f8b8-lb2jb:/# cp /usr/share/nginx/html/index.html /usr/share/nginx/html/my-index.html

	# ready数量为1
	[root@k8s-deploy case3-Probe]#kubectl get pod -n myserver
	NAME READY STATUS RESTARTS AGE
	myserver-myapp-frontend-deployment-f9774f8b8-lb2jb 1/1 Running 0 36s
	net-test1 1/1 Running 2 (30m ago) 6h4m

	[root@k8s-deploy case3-Probe]#kubectl get ep -n myserver
	NAME ENDPOINTS AGE
	myserver-myapp-frontend-service 10.200.169.166:80 40s

	# 获取http响应码
	[root@k8s-deploy yaml]#curl -Is 10.0.0.42:30023/my-index.html\|grep HTTP
	HTTP/1.1 200 OK

	buildkitd(服务端)，支持runc和containerd作为镜像构建环境，默认是runc，可以更换为containerd
	buildctl(客户端)，负责解析Dockerfile文件，并向服务端buildkitd发送构建请求

	cd /usr/local/src
	# 下载并解tar
	wget https://github.com/moby/buildkit/releases/download/v0.11.0/buildkit-v0.11.0.linux-amd64.tar.gz
	tar xvf buildkit-v0.11.0.linux-amd64.tar.gz
	mv bin/buildctl bin/buildkitd /usr/local/bin/

	# 准备buildkit socket
	cat >> /lib/systemd/system/buildkit.socket <<EOF
	[Unit]
	Description=BuildKit
	Documentation=https://github.com/moby/buildkit

	[Socket]
	ListenStream=%t/buildkit/buildkitd.sock

	[Install]
	WantedBy=sockets.target
	EOF

	# 准备buildkit service
	cat >> /lib/systemd/system/buildkit.service <<EOF
	[Unit]
	Description=BuildKit
	Requires=buildkit.socket
	After=buildkit.socket
	Documentation=https://github.com/moby/buildkit

	[Service]
	ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true

	[Install]
	WantedBy=multi-user.target
	EOF

	# 启动服务
	systemctl daemon-reload
	systemctl enable buildkit
	systemctl start buildkit

	[root@k8s-master2 src]#systemctl is-active buildkit.service
	active

	# 安装nerdctl
	wget https://github.com/containerd/nerdctl/releases/download/v1.0.0/nerdctl-1.0.0-linux-amd64.tar.gz
	tar -xvf nerdctl-1.0.0-linux-amd64.tar.gz
	mv nerdctl /usr/bin/

	# 安装CNI插件
	wget https://github.com/containernetworking/plugins/releases/download/v1.1.0/cni-plugins-linux-amd64-v1.1.0.tgz
	mkdir -p /opt/cni/bin
	tar -xvf cni-plugins-linux-amd64-v1.1.0.tgz -C /opt/cni/bin

	# 配置nerdctl命令补全
	source <(nerdctl completion bash)
	echo "source <(nerdctl completion bash)" >> ~/.bashrc

	# 登录harbor仓库，添加参数--insecure-registry
	nerdctl login --insecure-registry harbor.chu.net
	#下载镜像
	nerdctl pull centos:7.9.2009
	# 打tag
	nerdctl tag centos:7.9.2009 harbor.chu.net/baseimages/centos:7.9.2009
	# 上传镜像至harbor仓库，添加--insecure-registry参数
	nerdctl --insecure-registry push harbor.chu.net/baseimages/centos:7.9.2009

	# 镜像构建服务器创建harbor域名目录
	[root@k8s-master2 ~]#mkdir -p /etc/containerd/certs.d/harbor.chu.net

	# harbor证书分发
	## 格式转换
	[root@harbor1 harbor]#cd /apps/harbor/certs/
	[root@harbor1 certs]#openssl x509 -inform PEM -in chu.net.crt -out chu.net.cert
	[root@harbor1 certs]#ls
	ca.crt ca.key ca.srl chu.net.cert chu.net.crt chu.net.csr chu.net.key v3.ext

	## 复制证书至镜像构建服务器
	[root@harbor1 certs]# scp ca.crt chu.net.cert chu.net.key 10.0.0.12:/etc/containerd/certs.d/harbor.chu.net/

	## 查看镜像服务器证书
	[root@k8s-master2 ~]#ls /etc/containerd/certs.d/harbor.chu.net/
	ca.crt chu.net.cert chu.net.key

areke