k8s~fluentd的configmap设置es索引前缀

对于fluentd这个组件来说，你是负责抓取日志的，它可以从docker的控制台里抓取，也可以从指定文件夹里抓取，对于文件夹里存储的日志文件，我们需要先配置logback，然后再进行fluentd的configmap的配置，这样才能把持久化的日志抓取出来，并推送到elastic这种存储介质里。

logback控制存储位置

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="logPath" value="/var/log/"/>
    <springProperty scope="context" name="springAppName" source="spring.application.name"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%date-%level-%X{X-B3-TraceId:-}-%X{X-B3-SpanId:-}-[%file:%line]-%msg%n</pattern>
        </encoder>
    </appender>

    <appender name="fileInfoLog" filePermissions="rw-r--r--" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers class="net.logstash.logback.composite.loggingevent.LoggingEventJsonProviders">
                <pattern>
                    <pattern>
                        {
                        "level": "%level",
                        "application": "${springAppName:-}",
                        "trace": "%X{X-B3-TraceId:-}",
                        "span": "%X{X-B3-SpanId:-}",
                        "exportable": "%X{X-Span-Export:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "message": "%message"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder>
        <!--滚动策略-->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!--路径-->
            <fileNamePattern>${logPath}/info.%d.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
    </appender>
    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="fileInfoLog"/>
   </root>

fluentd以sidecar边车方法注册到pod里

这种sidecar设计主要为了解耦，它与pod里的容器共享存储卷，事实上就是读取容器产生的日志，然后把日志推送到存储介质里，本例是推送到elastic里，通过kibana进行查询和分析,k8s的yaml部署脚本如下

kind: Deployment
apiVersion: apps/v1
metadata:
  name: hello-world-deployment
  namespace: saas
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
        - name: hello-world
          image: 172.17.0.22:8888/saas/hello-world:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 9001
          env:
            - name: spring.profiles.active
              value: prod
          volumeMounts:
            - name: varlog
              mountPath: /var/log
        - name: fluent-sidecar
          image: registry.cn-beijing.aliyuncs.com/k8s-mqm/fluentd-elasticsearch:v2.1.0
          env:
            - name: FLUENTD_ARGS
              value: -c /etc/fluentd-config/fluentd.conf
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: config-volume
              mountPath: /etc/fluentd-config
      volumes:
        - name: varlog
          emptyDir: {}
        - name: config-volume
          configMap:
            name: fluentd-config

最后是，为fluentd添加配置，就是k8s里的configmap，注意，它是针对某个namespace来说的，这个configmap不能跨namespace访问。

一般来说，一个namespace在kibana里可以对应一个索引，你在配置fluentd时，可以使用tag和logstash_prefix来实现索引前缀的功能

<source>
type tail
format json
path /var/log/*.log
pos_file /var/log/*.log.pos
tag test.*
</source>

<match **>
@id elasticsearch
@type elasticsearch
@log_level debug
type_name fluentd
host elasticsearch.elk
port 9200
logstash_format true
logstash_prefix test #表示索引的前缀，对应source里的tag，一个namespace可以是一个，对应一组微服务，方便进行日志追踪
flush_interval 10s
</match> 	  	

最后在kibana里建立索引

Management->create index，选择test-*，保存即可