Filebeat介绍与使用

介绍：

Filebeat附带预构建的模块，这些模块包含收集、解析、充实和可视化各种日志文件格式数据所需的配置，每个Filebeat模块由一个或多个文件集组成，这些文件集包含摄取节点管道、Elasticsearch模板、Filebeat勘探者配置和Kibana仪表盘。

filebeat和logstash是一样的作用
ELK都是Java程序写的 
filebeat是golang写的  #速度非常快

Filebeat模块很好的入门，它是轻量级单用途的日志收集工具，用于在没有安装java的服务器上专门收集日志，可以将日志转发到logstash、elasticsearch或redis等场景中进行下一步处理。

filebeat 支持的服务

1.安装filebeat

#上传代码包
[root@logstash ~]# rz filebeat-6.6.0-x86_64.rpm

#安装
[root@logstash ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm

2.配置文件

[root@logstash ~]# rpm -qc filebeat
/etc/filebeat/filebeat.yml

3.日志

[root@logstash ~]# less /var/log/filebeat/filebeat

二、Filebeat收集单类型日志到本地文件

配置Filebeat

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml  
filebeat.inputs:
- type: log
  enabled: true                   #配置文件的格式必须遵守yml的格式写法，否则报错
  paths:
    - /var/log/messages

output.file:
  path: "/tmp"
  filename: "filebeat_message.log"    

2.启动

[root@logstash ~]# systemctl start filebeat.service
[root@logstash ~]# ps -ef | grep filebeat
[root@logstash ~]# ps -ef | grep filebeat
root      12418      1  2 16:55 ?        00:00:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root      12438   7019  0 16:56 pts/0    00:00:00 grep --color=auto filebeat

3.测试

[root@logstash ~]# echo 1111 >> /var/log/messages

#得到内容
[root@logstash ~]# tail -f /tmp/filebeat_message.log
{"@timestamp":"2020-07-21T08:58:00.373Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.6.0"},"source":"/var/log/messages","offset":230243,"prospector":{"type":"log"},"input":{"type":"log"},"beat":{"version":"6.6.0","name":"logstash","hostname":"logstash"},"host":{"name":"logstash"},"log":{"file":{"path":"/var/log/messages"}},"message":"1111"}

三、filebeat收集单个日志到ES

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]    #收集日志到elasticsearch 不需要指定索引名字
  
[root@logstash ~]# systemctl restart filebeat.service

2.访问nginx测试

3.指定ES索引名配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx_json-%{+yyyy.MM.dd}"  #filebeat 不支持自定义索引，需要添加下面的setup几项
  #index: "nginx_json-%{[beat.version]}-%{+yyyy.MM.dd}"  如果有多个filebeat版本号，可以加上filebeat版本     
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true     #注意 这几行配置必须顶个写
setup.template.enabled: false
setup.ilm.enabled: false

[root@logstash ~]# systemctl restart filebeat

4.修改kibana中日志展示格式

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx_json-%{+yyyy.MM.dd}"   
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

四、收集单个日志到reids

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.redis:
  hosts: ["172.16.1.53:6381"]
  db: "0"
  key: "nginx_json_redis"

2.启动

[root@logstash ~]# systemctl restart filebeat

3.访问nginx，查看redis

[root@db03 ~]# redis-cli -p 6381 --raw
127.0.0.1:6381> keys *
nginx_json_redis
127.0.0.1:6381> LLEN nginx_json_redis
8

4.配置将redis数据取出到ES

[root@logstash ~]# vim /etc/logstash/conf.d/filebeat_redis_es.conf
input {
  redis {
    host => "172.16.1.53"
    port => 6381
    data_type => "list"
    db => "0"
    key => "nginx_json_redis"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "filebeat_redis_es_%{+YYYY-MM-dd}"
  }
}

[root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat_redis_es.conf

五、filebeat收集日志到logstash

1.配置filebeat

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.logstash:
  hosts: ["10.0.0.54:6666"]

2.配置logstash接收数据传给ES

[root@logstash ~]# vim /etc/logstash/conf.d/filebeat_logstash_es.conf
input {
  beats {
    port => 6666
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "ngx_file_log_es_%{+YYYY-MM-dd}"
  }
}

一、filebeat收集多个日志到ES

1.配置方式一：

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx_json_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        source: "/var/log/nginx/nginx_json.log"
    - index: "nginx_access_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        source: "/var/log/nginx/access.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

2.配置方式二：

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["json"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["access"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx_json_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "json"
    - index: "nginx_access_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

二、filebeat收集java报错日志

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx_json_%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false