Kubernetes Logging operator云原生日志收集

1、前提和注意事项

1、基础环境：`Rancher v2.6.9`、`k8s v1.24.7`
2、使用`Rancher`的`Chart`仓库部署了`Logging operator 3.17.7`
3、使用`Helm`在`cattle-logging-system`名称空间部署了`Elasticsearch 8.5.1`
4、使用`Helm`在`cattle-logging-system`名称空间部署了`Kibana 8.5.1`
5、分别收集了k8s组件、应用、审计、事件、os系统日志、运行时日志，其中运行时部分尚未验证
6、本环境中`ClusterOutput`应只定义一个，为了演示用因此我在每种收集类型中均定义了一个`ClusterOutput`

2、收集k8s组件日志，k8s-components.yaml

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: k8s-components-clusteroutput
  namespace: cattle-logging-system
spec:
  elasticsearch:
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    host: elasticsearch-master.cattle-logging-system
    port: 9200
    scheme: https
    ssl_verify: true
    ca_file:
      mountFrom:
        secretKeyRef:
          key: ca.crt
          name: elasticsearch-master-certs    
    user: elastic
    password:
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password
    reconnect_on_error: true
    reload_on_failure: true
    reload_connections: false
    logstash_format: true
    logstash_prefix: k8s-components
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: k8s-components-clusterflow
  namespace: cattle-logging-system
spec:
  globalOutputRefs:
  - k8s-components-clusteroutput
  match:
  - select:
      namespaces:
        - cattle-monitoring-system
        - cattle-system
        - kube-system

3、收集k8s应用容器日志，k8s-application.yaml

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: k8s-application-clusteroutput
  namespace: cattle-logging-system
spec:
  elasticsearch:
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    host: elasticsearch-master.cattle-logging-system
    port: 9200
    scheme: https
    ssl_verify: true
    ca_file:
      mountFrom:
        secretKeyRef:
          key: ca.crt
          name: elasticsearch-master-certs    
    user: elastic
    password:
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password
    reconnect_on_error: true
    reload_on_failure: true
    reload_connections: false
    logstash_format: true
    logstash_prefix: k8s-application
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: k8s-application-clusterflow
  namespace: cattle-logging-system
spec:
  globalOutputRefs:
  - k8s-application-clusteroutput
  match:
  - exclude:
      namespaces:
        - cattle-monitoring-system
        - cattle-system
        - cattle-logging-system
        - kube-system
        - cattle-fleet-system
  - select: {}

4、收集k8s审计日志，k8s-auditlog.yaml

apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: k8s-auditlog-hosttailer
  namespace: cattle-logging-system
spec:
  fileTailers:
    - name: kubernetes-audit
      path: /var/log/kubernetes/audit.log
      buffer_max_size: 64k
      disabled: false
      skip_long_lines: "true"
      #containerOverrides:
        #image: fluent/fluent-bit:1.9.3
  workloadOverrides:
      tolerations:
      - effect: NoSchedule
        key: cattle.io/os
        operator: Equal
        value: linux
      - operator: Exists
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: k8s-auditlog-clusteroutput
  namespace: cattle-logging-system
spec:
  elasticsearch:
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    host: elasticsearch-master.cattle-logging-system
    port: 9200
    scheme: https
    ssl_verify: true
    ca_file:
      mountFrom:
        secretKeyRef:
          key: ca.crt
          name: elasticsearch-master-certs    
    user: elastic
    password:
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password
    reconnect_on_error: true
    reload_on_failure: true
    reload_connections: false
    logstash_format: true
    logstash_prefix: k8s-auditlog
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: k8s-auditlog-flow
  namespace: cattle-logging-system
spec:
  filters:
  - tag_normaliser: {}
  - parser:
      parse:
        type: json
  match:
  - select:
      labels:
        app.kubernetes.io/instance: k8s-auditlog-hosttailer-host-tailer
  globalOutputRefs:
    - k8s-auditlog-clusteroutput

5、收集k8s事件日志，k8s-events.yaml

apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: EventTailer
metadata:
  name: k8s-events-eventtailer
spec:
  controlNamespace: cattle-logging-system
  #containerOverrides:
     #image: banzaicloud/eventrouter:v0.1.0
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: k8s-events-clusteroutput
  namespace: cattle-logging-system
spec:
  elasticsearch:
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    host: elasticsearch-master.cattle-logging-system
    port: 9200
    scheme: https
    ssl_verify: true
    ca_file:
      mountFrom:
        secretKeyRef:
          key: ca.crt
          name: elasticsearch-master-certs    
    user: elastic
    password:
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password
    reconnect_on_error: true
    reload_on_failure: true
    reload_connections: false
    logstash_format: true
    logstash_prefix: k8s-events
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: k8s-events-flow
  namespace: cattle-logging-system
spec:
  filters:
  - tag_normaliser: {}
  - parser:
      parse:
        type: json
  globalOutputRefs:
  - k8s-events-clusteroutput
  match:
  - select:
      labels:
        app.kubernetes.io/instance: k8s-events-eventtailer-event-tailer

6、收集节点os系统日志，k8s-oslog.yaml

apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: k8s-oslog-hosttailer
  namespace: cattle-logging-system
spec:
  fileTailers:
    - name: system-messages
      path: /var/log/messages
      buffer_max_size: 64k
      disabled: false
      skip_long_lines: "true"
      #containerOverrides:
        #image: fluent/fluent-bit:1.9.3
  workloadOverrides:
      tolerations:
      - effect: NoSchedule
        key: cattle.io/os
        operator: Equal
        value: linux
      - operator: Exists
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: k8s-oslog-clusteroutput
  namespace: cattle-logging-system
spec:
  elasticsearch:
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    host: elasticsearch-master.cattle-logging-system
    port: 9200
    scheme: https
    ssl_verify: true
    ca_file:
      mountFrom:
        secretKeyRef:
          key: ca.crt
          name: elasticsearch-master-certs    
    user: elastic
    password:
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password
    reconnect_on_error: true
    reload_on_failure: true
    reload_connections: false
    logstash_format: true
    logstash_prefix: k8s-oslog
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: k8s-oslog-flow
  namespace: cattle-logging-system
spec:
  filters:
  - tag_normaliser: {}
  - record_modifier:
      records:
      - host: ${record.dig('kubernetes', 'host')}
      whitelist_keys: host,message
  globalOutputRefs:
  - k8s-oslog-clusteroutput
  match:
  - select:
      labels:
        app.kubernetes.io/instance: k8s-oslog-hosttailer-host-tailer

7、收集容器运行时日志，k8s-runtimelog.yaml

apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: k8s-runtimelog-hosttailer
  namespace: cattle-logging-system
spec:
  fileTailers:
    - name: runtime-tail
      path: /var/lib/rancher/rke2/agent/containerd/containerd.log
      buffer_max_size: 64k
      disabled: false
      skip_long_lines: "true"
      #containerOverrides:
        #image: fluent/fluent-bit:1.9.3
  workloadOverrides:
      tolerations:
      - effect: NoSchedule
        key: cattle.io/os
        operator: Equal
        value: linux
      - operator: Exists
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: k8s-runtimelog-clusteroutput
  namespace: cattle-logging-system
spec:
  elasticsearch:
    buffer:
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true
    host: elasticsearch-master.cattle-logging-system
    port: 9200
    scheme: https
    ssl_verify: true
    ca_file:
      mountFrom:
        secretKeyRef:
          key: ca.crt
          name: elasticsearch-master-certs    
    user: elastic
    password:
      valueFrom:
        secretKeyRef:
          name: elasticsearch-master-credentials
          key: password
    reconnect_on_error: true
    reload_on_failure: true
    reload_connections: false
    logstash_format: true
    logstash_prefix: k8s-runtimelog
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: k8s-runtimelog-flow
  namespace: cattle-logging-system
spec:
  filters:
  - tag_normaliser: {}
  - parser:
      parse:
        type: none
  match:
  - select:
      labels:
        app.kubernetes.io/instance: k8s-runtimelog-hosttailer-host-tailer
  globalOutputRefs:
    - k8s-runtimelog-clusteroutput

8、其他

# 查询自定义资源列表
kubectl get flows,clusterflows,outputs,clusteroutputs,hosttailers,eventtailers,loggings --all-namespaces

# 查询和删除索引
curl -XGET -k -u elastic:S8CiLUnwC3rgEkpE https://10.96.1.137:9200/_cat/indices
curl -XDELETE -k -u elastic:S8CiLUnwC3rgEkpE https://10.96.1.137:9200/k8s-components-2023.01.15

9、总结

1、Logging operator网上的资料不多，幸运的是有前人帮忙开路，但还需结合官网，并且查看CRD去熟悉配置
2、在页面上配置比写资源配置清单更难，因为需要对各参数项都比较熟悉才能在页面上配出来
3、需要熟悉fluentd/fluentbit的配置文件
4、需要熟悉fluentd/fluentbit的解析器、过滤器、输出输出插件，尤其是自定义日志的正则匹配
5、需要熟悉flow、clusterflow、output、clusteroutput、hosttailer、eventtailer的资源配置清单写法
6、hosttailer、eventtailer目前处于v1alpha1，不知道后面官方会怎么做，个人觉得更改的可能性很大

参考链接：
https://blog.csdn.net/RancherLabs/article/details/126617538
https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/elasticsearch/