部署 Fluent Bit ( td-agent-bit )

部署 Fluent Bit ( td-agent-bit )

此处使用 yum 安装，如需要离线安装，可以参考离线安装dokcer的方式进行部署

安装

参考官方文档：https://docs.fluentbit.io/manual/installation/linux/redhat-centos

• 配置yum仓库

# cat >/etc/yum.repos.d/tb-aget-bit.repo<<EOF
[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/centos/7/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
EOF

• 安装

# yum install td-agent-bit

配置 Fluent Bit

参考官档：
https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/configuration-file#config_include_file
https://docs.fluentbit.io/manual/pipeline/inputs/systemd
https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch

# cd /etc/td-agent-bit/
# vim evescn.conf

[INPUT]
    Name         systemd
    Tag          test_evescn.*
    Systemd_Filter  _SYSTEMD_UNIT=evescn.service

[OUTPUT]
    Name  es
    Match test_evescn.*
    Host  ES_IP
    Port  9200
    HTTP_User XXXXXXX (es-xpack的账户密码信息)
    HTTP_Passwd XXXXXXXXXXXXXXXXX (es-xpack的账户密码信息)
    Index test_evescn_index_228
    Type  test_evescn_type_228
    Logstash_format true
    Logstash_Prefix test_evescn
    Logstash_DateFormat %Y-%m-%d

## 编辑主配置文件，追加引入子配置文件 ```evescn.conf```
# vim td-agent-bit.conf

@INCLUDE evescn.conf

启动 Fluent Bit

systemctl start td-agent-bit
systemctl enable td-agent-bit

最后在 kibana 上面添加索引 test_evescn

---

手动验证是否可用将日志推送到es

在td-agent-bit服务器上面进行手动推送
# /opt/td-agent-bit/bin/td-agent-bit -i  systemd\
	-p systemd_filter=_SYSTEMD_UNIT=evescn.service \
	-p tag='test_evescn.*' \
	-o es://ES_IP:9200/test_evescn_index_228/test_evescn_type_228 \
	-m '*'

在es服务器上面验证是否有索引

# curl -u 'XXXX:XXXXXXXXX' 'localhost:9200/_cat/indices?v'
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test_evescn               q0t4O-GeTOSzio49mQ8B2A   1   0    1257047            0    628.9mb        628.9mb

java日志格式非完全 json 如何转换?

• 首先添加日志过滤规则

# cat parsers.conf
[PARSER]
    Name   evescn_parsers
    Format regex
    Regex ^(?<log_time>[^\[]*)\[(?<log_url>[^\]]*)\]\[(?<log_level>[^ ]*)\]\[(?<log_tid>[^ ]*)\]\:(?<log_data>[^ ].*)

• 测试网站

https://rubular.com/r/X7BH0M4Ivm

# 测试日志
2021-01-26 05:48:58.626[com.tencent.nhccovid_19.service.GovDataServiceImpl->findAntibodyDetection#203][INFO][%PARSER_ERROR[tid]]:{"message":"查询失败","time":"0","url":"/evescn","urlName":"测试接口","userInfo":{"name":"evescn","id":"5100daxuuGxaad7433"}}

• 配置文件中影响添加的规则

# vim evescn.conf

[INPUT]
    Name         systemd
    Tag          test_evescn.*
    Systemd_Filter  _SYSTEMD_UNIT=evescn.service

[FILTER]
    Name parser
    Match test_evescn.*
    Key_Name log
    Parser evescn_parsers
    Reserve_Data On
    Preserve_Key On

[FILTER]
    Name parser
    Match test_evescn.*
    Key_Name log_data
    Parser json
    Reserve_Data On
    Preserve_Key On

[OUTPUT]
    Name  es
    Match test_evescn.*
    Host  ES_IP
    Port  9200
    HTTP_User XXXXXXX (es-xpack的账户密码信息)
    HTTP_Passwd XXXXXXXXXXXXXXXXX (es-xpack的账户密码信息)
    Index test_evescn_index_228
    Type  test_evescn_type_228
    Logstash_format true
    Logstash_Prefix test_evescn
    Logstash_DateFormat %Y-%m-%d