filebeat收集k8s的容器日志需要关注的几个点

1、转载自博客：https://www.jianshu.com/p/c3709d3384a8

在使用filebeat 7.x版本采集容器日志时，推荐采用container input，并且使用autodiscover实现容器的自动发现，也就是在有新的容器运行时，filebeat会自动去采集新建的容器日志，而不需要再去修改filebeat.yml来实现对新部署的容器日志的采集。而正式使用autodiscover功能，使得限定采集源成为了可能，因为在autodiscover模式下，filebeat在启动时就会去调用kubernetes API来获取当前集群下所有的namespace、pod、container等元数据的信息，然后根据这些元数据再去指定的目录采集对应的日志。

下面给出一个可用的限定采集源的filebeat.yml：

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      templates:
        - condition:
            and:
              - or:
                  - equals:
                      kubernetes.namespace: testa
                  - equals:
                      kubernetes.namespace: testb
              - equals:
                  kubernetes.container.name: nginx
                  kubernetes.labels:
                    k8s-app: nginx
          config:
             - type: container
               paths:
                - /var/log/containers/${data.kubernetes.pod.name}_${data.kubernetes.namespace}_${data.kubernetes.container.name}-*.log
output.elasticsearch:
  hosts: ['x.x.x.x:9200']
  username: "xxx"
  password: "xxx"

上述配置中，用于限定采集源的就是condition模块下的部分，用于限定只采集testa 或者 testb命名空间下的nginx容器的日志。可以根据kubernetes元数据来限定采集源，可用的元数据有以下这些：

host
port (if exposed)
kubernetes.labels
kubernetes.annotations

kubernetes.container.id
kubernetes.container.image
kubernetes.container.name
kubernetes.namespace
kubernetes.node.name
kubernetes.pod.name
kubernetes.pod.uid

kubernetes.node.name
kubernetes.node.uid

kubernetes.namespace
kubernetes.service.name
kubernetes.service.uid
kubernetes.annotations

上述配置中，condition可以根据需求定义更复杂的限定条件，可以参考Conditions进行填写。

另外需要注意的是，上述配置中的config模块下的paths路径，需要也通过占位符对日志文件的名称进行匹配，否则就会出现采集上来的日志内容与kubernetes元数据不一致的问题。比如/var/log/containers目录下有各个pod的日志，日志文件名的命名规则为{pod_name}_{namespace}_{container_name}-{container_id}.log

nginx-6c5ff7b97b-6t5k4_default_nginx-eeecb30c81564668b1858c186099ab525431b435ed1b8fa3b25704cbbbca6a2d.log
那么 paths就需要通过$符进行规则匹配：

${data.kubernetes.pod.name}_${data.kubernetes.namespace}_${data.kubernetes.container.name}-*.log
具体参考filebeat的官方文档：https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

 

上述配置的问题就是condition条件不会生效，会全量采集所有的命名空间下的容器日志，目前没有找到较好的解决办法来限定采集源，但是可以通过定义drop_event processor来丢弃掉不需要采集的日志。实际使用中，还是建议直接使用7.2及以上版本的filebeat来采集使用containerd组件部署的kubernetes集群中的容器日志。

 

2、https://devpress.csdn.net/k8s/62ffd7087e66823466194f38.html https://devpress.csdn.net/k8s/62ffd7087e66823466194f38.html

 

https://devpress.csdn.net/cicd/62f37bbe7e66823466186e0a.html

假设您最终在官方文档页面上找到了可以与处理器一起使用的条件,并且您想同时使用“and”和“not”关键字,但这并不像听起来那么容易关于破折号和缩进。 这是一个可以帮助你的片段,我用它来只推送来自 kube-system 命名空间的日志,这些命名空间属于名为 kube-dns 的 pod: processors: - drop_event: when: and: - equals: kubernet

假设您最终在官方文档页面上找到了可以与处理器一起使用的条件,并且您想同时使用“and”和“not”关键字,但这并不像听起来那么容易关于破折号和缩进。

这是一个可以帮助你的片段,我用它来只推送来自 kube-system 命名空间的日志,这些命名空间属于名为 kube-dns 的 pod:

processors:
  - drop_event:
      when:
        and:
          - equals:
              kubernetes.namespace: "kube-system"
          - not.contains:
              kubernetes.pod.name: "kube-dns"

接下来详细的配置文件可以参考下面的配置文件：Node上部署一个日志收集程序：DaemonSet方式部署日志收集程序。对本节点/var/log和/var/lib/docker/containers/ 两个目录下的日志进行采集. 作者：赵渝强老师 https://www.bilibili.com/read/cv6015654 出处：bilibili

filebeat的配置文件如下：filebeat-kubelet-cm-kafka.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: filebeat
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         templates:
           - config:
              - type: container
                paths:
                  - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
                exclude_lines: ["^\\s+[\\-`('.|_]"]
                close_timeout: 2h
    filebeat.inputs:
    - type: log
      paths: 
        - /var/log/messages
      fields:
        kubernetes.pod.ip: ${POD_IP}
      fields_under_root: true
      include_lines: ['(atomic-openshift-node|dockerd-current)']
    processors:
      - add_cloud_metadata:
      - drop_event.when.not.or:
        - equals.input.type: "log"
        - equals.kubernetes.namespace: "default"
        - equals.kubernetes.namespace: "kube-system"
        - equals.kubernetes.namespace: "openshift-monitoring"
        - equals.kubernetes.namespace: "openshift-sdn"
        - equals.kubernetes.namespace: "openshift-node"
        - equals.kubernetes.namespace: "ccnp-system"
        - equals.kubernetes.namespace: "openshift-infra"
        - equals.kubernetes.namespace: "openshift-metrics-server"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "ocp-node-exporter"
            cluster: "cdgxsc-01"
            namespace: "openshift-monitoring"
          when:
            and:
              - equals:
                  kubernetes.namespace: "openshift-monitoring"
              - contains:
                  kubernetes.pod.name: "node-exporter"
              - equals:
                  kubernetes.container.name: "node-exporter"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "prometheus-k8s"
            cluster: "cdgxsc-01"
            namespace: "openshift-monitoring"
          when:
            and:
              - equals:
                  kubernetes.namespace: "openshift-monitoring"
              - contains:
                  kubernetes.pod.name: "prometheus-k8s"
              - equals:
                  kubernetes.container.name: "prometheus"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "k8s-ccnp-cluster-controller"
            cluster: "cdgxsc-01"
            namespace: "ccnp-system"
          when:
            and:
              - contains:
                  kubernetes.pod.name: "ccnp-cluster-controller"
              - equals:
                  kubernetes.namespace: "ccnp-system"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "ocp-etcd"
            cluster: "cdgxsc-01"
            namespace: "kube-system"
          when:
            and:
              - contains:
                  kubernetes.pod.name: "master-etcd"
              - equals:
                  kubernetes.namespace: "kube-system"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "ocp-apiserver"
            cluster: "cdgxsc-01"
            namespace: "kube-system"
          when:
            and:
              - contains:
                  kubernetes.pod.name: "master-api"
              - equals:
                  kubernetes.namespace: "kube-system"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "ocp-controller"
            cluster: "cdgxsc-01"
            namespace: "kube-system"
          when:
            and:
              - contains:
                  kubernetes.pod.name: "master-controllers"
              - equals:
                  kubernetes.namespace: "kube-system"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "ocp-ovs"
            cluster: "cdgxsc-01"
            namespace: "openshift-sdn"
          when:
            and:
              - contains:
                  kubernetes.pod.name: "ovs"
              - equals:
                  kubernetes.namespace: "openshift-sdn"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "ocp-sdn"
            cluster: "cdgxsc-01"
            namespace: "openshift-sdn"
          when:
            and:
              - contains:
                  kubernetes.pod.name: "sdn"
              - equals:
                  kubernetes.namespace: "openshift-sdn"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "atomic-openshift-node"
            cluster: "cdgxsc-01"
            namespace: ""
          when:
            and:
              - contains:
                  message: "atomic-openshift-node"
              - equals:
                  log.file.path: "/var/log/messages"
      - add_fields:
          fields:
            sysCode: "l002x0"
            middleware: "docker-ocp"
            cluster: "cdgxsc-01"
            namespace: ""
          when:
            and:
              - contains:
                  message: "dockerd-current"
              - equals:
                  log.file.path: "/var/log/messages"
      - script:
          lang: javascript
          source: >
             function process(event) {
               event.Put("host.ip", [event.Get("kubernetes.pod.ip")]);
             }
      - drop_fields:
          fields: ["agent", "input", "ecs", "host.os", "host.architecture", "host.id", "host.mac", "host.containerized", "host.hostname", "kubernetes", "node", "container"]

    output:
      kafka:
        enabled: true
        hosts: ["10.8.54.58:9092","10.8.54.59:9092","10.8.54.60:9092"]
        topic: middleware-l002x0
        username: middleware-l002x0
        password: sYkc_0104
        sasl.mechanism: "SCRAM-SHA-256"
        worker: 3
       # 最大重试次数
        max_retries: 3
        #保持连接时间
        keep_alive: 60
        partition.hash:
          reachable_only: false
        required_ack: 1

 

 

 https://blog.csdn.net/qq_27818541/article/details/108229185

 

    processors:
      - add_cloud_metadata:
      - drop_event.when.not.or:
        - equals.input.type: "log"
        - equals.kubernetes.namespace: "default"
        - equals.kubernetes.namespace: "kube-system"
        - equals.kubernetes.namespace: "openshift-monitoring"
        - equals.kubernetes.namespace: "openshift-sdn"
        - equals.kubernetes.namespace: "openshift-node"
        - equals.kubernetes.namespace: "ccnp-system"
        - equals.kubernetes.namespace: "openshift-infra"
        - equals.kubernetes.namespace: "openshift-metrics-server"

上面的这个配置表示只采集满足要求的namespace下面的集群的容器的日志

ccnp-system、

openshift-infra
可以参看博客：https://docs.elastic.co/en/integrations/winlog


add_fileds是添加自定义的标签