K8S-ELK日志系统之四：filebeat

1、filebeat介绍

Filebeat是用于转发和集中日志数据的轻量级传送工具。Filebeat监视您指定的日志文件或位置，收集日志事件，并将它们转发到Elasticsearch或 Logstash或者kafka等
Filebeat的工作方式如下：启动Filebeat时，它将启动一个或多个输入，这些输入将在为日志数据指定的位置中查找。对于Filebeat所找到的每个日志，Filebeat都会启动收集器。每个收集器都读取单个日志以获取新内容，并将新日志数据发送到libbeat，libbeat将聚集事件，并将聚集的数据发送到为Filebeat配置的输出。

如图：

　　

 

2、部署方式

      k8s上可以部署方式：

filebeat和应用容器运行在一个pod，作为sidercar模式，搜集日志，这样将产生较多的sidercar容器 

filebeat作为daemonSet运行在各node节点，搜集docker日志，配置简单，但是日志不好分类

filebeat和应用运行同一容器，本次使用的方式

3、制作Dockerfile

使用tomcat官方最新镜像：tomcat:latest

filebeat版本：7.16.2，下载地址 https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.2-amd64.deb

FROM tomcat:latest
ENV TIME_ZONE Asia/Shanghai
COPY  filebeat-7.16.2-amd64.deb /
COPY sources.list /etc/apt/
COPY entrypoint.sh /
RUN apt-get update -y &&  \
    apt-get install vim inetutils-ping net-tools telnet -y && \
    echo "${TIME_ZONE}" > /etc/timezone && \
    ln -sf /usr/share/zoneinfo/${TIME_ZONE} /etc/localtime && \
    chmod +x /entrypoint.sh && \
    dpkg -i /filebeat-7.16.2-amd64.deb
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/filebeat","-c","/etc/filebeat/filebeat.yml","-e"]
EXPOSE 8080

#!/bin/bash
/usr/local/tomcat/bin/catalina.sh run 1>/dev/null 2>&1 &
exec "$@"

# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free

deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free

deb https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free

4、yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: filebeat-configmap
  namespace: es
data:
  filebeat-config: |
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
      - /var/log/*.log
      fields:
        app_id: system
    - type: log
      enabled: true
      paths:
      - /usr/local/tomcat/logs/*access*
      fields:
        app_id: tomcat
    filebeat.conf.modules:
      path: ${path.conf}/modules.d/*yml
      reload.enabled: false
    setup.template.setting:
      index.number_of_shards: 1
    output.kafka:
      hosts: ["10.0.8.111:30209","10.0.8.112:30209","10.0.8.113:30209"]
      enable: true
      required_acks: 1
      topic: "%{[fields.app_id]}"
      partition.round_robin:
        reachable_only: false
      keep_alive: 10s

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: tomcat
  namespace: es
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: tomcat
  template:
    metadata:
      name: tomcat
      namespace: es
      labels:
        app: tomcat
    spec:
      containers:
      - name: tomcat
        image: harbor.myland.com/baseimages/tomcat-filebeat:7.16.0
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 300m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: file-beatconfig-file
          mountPath: /etc/filebeat/

        ports:
        - name: tomcat
          containerPort: 8080
          protocol: TCP
      volumes:
      - name: file-beatconfig-file
        configMap: 
          name: filebeat-configmap
          items:
          - key: filebeat-config
            path: filebeat.yml

5、验证

　　可以看到，kafka上产生system和tomcat两个topic，搜集到最新日志