FileBeat多行消息Multiline
官方地址:
https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
服务日志输出样例:
根据样例可以看出,每一行开始是有相同规律:
[时间]2021-10-22 详细学习可以查阅正则表达式
'^(.{4})([0-9]{4}-[0-9]{2}-[0-9]{2})'
^(.{4}):表示以任意四个字符开始/ [时间]
[0-9]{4}-:表示4个0到9数字/ 2021-
[0-9]{2}-: 表示2个0到9数字/ 10-
[0-9]{2}: 表示2个0到9数字/ 22
filebeta的filebeat.yml配置文件:
└─# grep -v "^#" filebeat.yml | grep -v "^$" | grep -v "#"
filebeat.inputs:
filebeat.config.inputs:
path: config/*.yml
reload.enabled: true
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: "http://172.17.12.174:5601"
output.elasticsearch:
username: filebeta_log
hosts: ["http://es-cn-v0h1dthan00259sf2.elasticsearch.aliyuncs.com:9200"]
indices:
- index: "%{[fields.servicename]}-%{+yyyy.ww}"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- drop_fields:
fields: ["agent.ephemeral_id","agent.hostname","agent.id","agent.name","agent.type","agent.version","host.id","host.hostname","host.architecture","host.os.platform","host.ip","log.file.path","host.mac","host.os.build","host.os.family","host.os.kernel","ecs.version","log.offset","log.flags"]
ignore_missing: false
filebeta日志收集配置项input-win.yml配置:
- type: log
enabled: true
paths:
- D:\BetaWM\BetaWM.Server\Beta.VipManagerH5API\Logs\*.log
encoding: GB2312
multiline.pattern: '^(.{4})([0-9]{4}-[0-9]{2}-[0-9]{2})'
multiline.negate: true
multiline.match: after
fields:
servicename: "beta-vipmanagerh5api"
- type: log
enabled: true
paths:
- D:\BetaWM\BetaWM.Server\Beta.WxWorkApi\Log\*
encoding: GB2312
multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})'
multiline.negate: true
multiline.match: after
fields:
servicename: "beta-wxworkapi"
kibana日志收集展示根据配置message展示如下,和我们在服务的日志输出文件中message一致: