基于k8s搭建微服务日志收集中心

由于项目组业务需求，在k8s中运行的微服务日志要进行收集。

1、需要将k8s中的每个pod微服务挂载一个卷，（每个pod都会在一个node上，在node节点上创建一个文件夹，如：/logs/ 将这个目录挂载到每个微服务的pod中，使之实现映射）

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neusipo-cloud-gateway-deployment
  namespace: gateway
spec:
  selector:
    matchLabels:
      app: neusipo-cloud-gateway
  replicas: 3
  minReadySeconds: 10
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      name: neusipo-cloud-gateway
      labels:
        app: neusipo-cloud-gateway
    spec:
      containers:
      - name: neusipo-cloud-gateway
        image: cr.registry.cnipaig1.cloud/neusipo/neusipo-cloud-gateway-test:1.0  #这里的镜像地址是阿里云镜像仓库地址
        args: ["--spring.profiles.active=test"]   #配置文件中的测试配置
        env:
        - name: SERVER_PORT
          value: "18080"
        resources:
          requests:
            cpu: 4
            memory: 4Gi
          limits:
            cpu: 4
            memory: 4Gi
        volumeMounts:   #pod中挂载
        - name: gateway-logs   #name与下方本机name保持一致
          mountPath: /sipo-data/neusipo-test/   #pod中的挂载地址
          readOnly: false
      volumes:    #本机卷
      - name: gateway-logs     
        hostPath:
          path: /logs/   #本机创建的挂载目录

2、执行命令，创建pod。

kubectl apply -f deployment-demo.yaml -n gateway

3、查看是否创建成功

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get pods -n gateway
NAME                                                READY   STATUS    RESTARTS   AGE
neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht   1/1     Running   0          4h16m
neusipo-cloud-gateway-deployment-66b6c96fbf-r7wcd   1/1     Running   0          4h16m
neusipo-cloud-gateway-deployment-66b6c96fbf-wwjpk   1/1     Running   0          4h16m

 看到STATUS状态为Running 时说明容器已启动。

4、此时验证pod中是否已经挂载上本机目录。

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl exec -it  neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht -n gateway -- bash
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht data]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          40G   17G   21G  45% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        40G   17G   21G  45% /sipo-data/neusipo-test   #这里就代表已经挂载上本机所创建的目录了
shm              64M     0   64M   0% /dev/shm
tmpfs           7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.8G     0  7.8G   0% /proc/acpi
tmpfs           7.8G     0  7.8G   0% /proc/scsi
tmpfs           7.8G     0  7.8G   0% /sys/firmware
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht data]# 

此时查看pod服务启动后是否有日志打印，通过映射到node节点上创建的目录。（查看pod在那个node节点上，登录node节点，进入创建的目录，查看是否pod日志映射出来）

[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht data]# cd /sipo-data/neusipo-test/
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht neusipo-test]# ls
gateway
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht neusipo-test]# cd gateway/
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht gateway]# ls
logs
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht gateway]# cd logs/
[root@neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht logs]# ls
error  info　

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl  get pods -n gateway -o wide
NAME                                                READY   STATUS    RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
neusipo-cloud-gateway-deployment-66b6c96fbf-9ncht   1/1     Running   0          4h26m   172.20.1.91    cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2l   <none>           <none>
neusipo-cloud-gateway-deployment-66b6c96fbf-r7wcd   1/1     Running   0          4h26m   172.20.0.198   cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2k   <none>           <none>
neusipo-cloud-gateway-deployment-66b6c96fbf-wwjpk   1/1     Running   0          4h26m   172.20.1.60    cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2j   <none>           <none>
[root@iZqj001np4kv147ax0tr54Z ~]# kubectl  get node -o wide
NAME                                         STATUS   ROLES    AGE    VERSION            INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
cn-beijing-gzj1-d01.i-qj001np4kv147ax0tr54   Ready    master   198d   v1.14.8-aliyun.1   10.160.22.68   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147ax0tr55   Ready    master   198d   v1.14.8-aliyun.1   10.160.22.69   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147ax0tr56   Ready    master   198d   v1.14.8-aliyun.1   10.160.22.70   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2j   Ready    <none>   198d   v1.14.8-aliyun.1   10.160.22.73   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2k   Ready    <none>   198d   v1.14.8-aliyun.1   10.160.22.72   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2
cn-beijing-gzj1-d01.i-qj001np4kv147gu45x2l   Ready    <none>   198d   v1.14.8-aliyun.1   10.160.22.71   <none>        CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.2

 从这里看到其中一个pod在10.160.22.73这个node节点上，登录node节点查看是否有映射的日志

[root@iZqj001np4kv147gu45x2jZ logs]# pwd
/logs
[root@iZqj001np4kv147gu45x2jZ logs]# ls
gateway
[root@iZqj001np4kv147gu45x2jZ logs]# cd gateway/
[root@iZqj001np4kv147gu45x2jZ gateway]# ls
logs
[root@iZqj001np4kv147gu45x2jZ gateway]# cd logs/
[root@iZqj001np4kv147gu45x2jZ logs]# ls
error  info

 如果node节点出现了和pod中一样的目录则说明，挂载成功并映射成功，（其他微服务同理就行）

5、日志映射出来后，需要搭建filebeat镜像，创建filebeat的pod容器，容器需要创建为DaemonSet形式的（DaemonSet所创建的容器类似于守护进程，它会在每个node节点都创建一个pod），使filebeat容器可以从所有的node节点日志目录拉取日志到pod中，并发送给elk进行日志分析。在master节点上下载filebeat压缩包，并解压。

[root@iZqj001np4kv147ax0tr54Z ~]# ls
deployment-demo.yaml              filebeat-7.13.1-linux-x86_64.tar.gz         daemonset-demo.yamlfilebeat-7.13.1-linux-x86_64      daemonset-demo.yaml        dockerfile-filebeat

 创建filebeat的docker镜像

[root@iZqj001np4kv147ax0tr54Z ~]# cat dockerfile-filebeat 
FROM cr.registry.cnipaig1.cloud/neusipo/centos:7.9.2009
ENV LANG en_US.UTF-8
ENV TZ Asia/Shanghai

ADD filebeat-7.13.1-linux-x86_64.tar.gz /data/

WORKDIR /data/filebeat-7.13.1-linux-x86_64

ENTRYPOINT ["./filebeat", "-c", "filebeat.yml", "-e"]

 上传到阿里云的镜像仓库，阿里云容器镜像服务文档中已写如何上传，照着一步步执行就可。

 

 

 构建filebeat的yaml文件

apiVersion: apps/v1
kind: DaemonSet   #这里为DaemonSet模式
metadata:
  name: neusipo-filebeat
  namespace: filebeat
spec:
  selector:
    matchLabels:
      app: neusipo-filebeat
  template:
    metadata:
      name: neusipo-filebeat
      labels:
        app: neusipo-filebeat
    spec:
      containers:
      - name: neusipo-filebeat
        image: cr.registry.cnipaig1.cloud/neusipo/filebeat:v2
        command: ["./filebeat", "-c", "/data/filebeat/conf/filebeat.yml", "-e"]   #command模块可以强制替换容器里默认的启动命令
        resources:
          requests:
            cpu: 4
            memory: 4Gi
          limits:
            cpu: 4
            memory: 4Gi
        volumeMounts:
          - name: filebeat-log
            mountPath: /sipo-data/neusipo-test/    #这里为挂载node目录到pod中的目录
            readOnly: false
          - name: neusipo-filebeat
            mountPath: /data/filebeat/conf/       #这里为pod容器内filebeat配置文件目录
            readOnly: true
      volumes:
      - name: filebeat-log
        hostPath:
          path: /logs/          #node节点目录
      - name: neusipo-filebeat
        configMap:               #将pod容器内的filebeat配置文件映射到configmap中
          name: neusipo-filebeat    #configmap内filebeat的name
          items:
            - key: filebeat.yml      #映射在configmap中的filebeat的配置文件名
              path: filebeat.yml       #pod中filebeat的配置文件名

 创建neusipo-filebeat的configmap，映射的文件是从filebeat压缩包解压出来的配置文件

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl create configmap neusipo-filebeat --from-file=filebeat.yml -n filebeat
[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get configmap -n filebeat
NAME               DATA   AGE
neusipo-filebeat   1      3h55m   #这里DATA出现1，说明文件已经写入

 创建namesapce

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl create configmap ns filebeat
[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get ns |grep filebeat
filebeat              Active   4h25m

 创建filebeat的pod容器，并查看是否创建成功

kubectl apply -f daemonset-demo.yaml -n filebeat

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl get pods -n filebeat
NAME                     READY   STATUS    RESTARTS   AGE
neusipo-filebeat-64mhf   1/1     Running   0          3h55m
neusipo-filebeat-k9ncp   1/1     Running   0          3h55m
neusipo-filebeat-lsvcb   1/1     Running   0          3h55m

 进入容器查看是否挂载上node目录，并查看是否拉取到微服务的日志

[root@iZqj001np4kv147ax0tr54Z ~]# kubectl exec -it neusipo-filebeat-k9ncp -n filebeat /bin/bash
[root@neusipo-filebeat-k9ncp filebeat-7.13.1-linux-x86_64]# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          40G   17G   21G  45% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        40G   17G   21G  45% /sipo-data/neusipo-test
shm              64M     0   64M   0% /dev/shm
tmpfs           7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.8G     0  7.8G   0% /proc/acpi
tmpfs           7.8G     0  7.8G   0% /proc/scsi
tmpfs           7.8G     0  7.8G   0% /sys/firmware
[root@neusipo-filebeat-k9ncp filebeat-7.13.1-linux-x86_64]# cd /sipo-data/neusipo-test/
[root@neusipo-filebeat-k9ncp neusipo-test]# ls
gateway
[root@neusipo-filebeat-k9ncp neusipo-test]# cd gateway/
[root@neusipo-filebeat-k9ncp gateway]# ls
logs
[root@neusipo-filebeat-k9ncp gateway]# cd logs/
[root@neusipo-filebeat-k9ncp logs]# ls
error  info
[root@neusipo-filebeat-k9ncp logs]# 

 后续只需要更改configmap中的filebeat配置文件，就可以了。

kubectl edit configmap neusipo-filebeat -n filebeat

 完成！！！