CentOS6.9安装Filebeat监控Nginx的访问日志发送到Kafka

一、下载地址：

官方：https://www.elastic.co/cn/downloads/beats/filebeat

百度云盘：https://pan.baidu.com/s/1dvhqb0

 

二、安装

tar zxvf filebeat-6.2.1-linux-x86_64.tar.gz
mv filebeat-6.2.1-linux-x86_64 filebeat

也可yum安装

下载和安装key文件

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

创建yum源文件

vi /etc/yum.repos.d/elastic.repo

[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

开始安装

yum install filebeat

 

三、配置

vi kafka.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/openresty/nginx/logs/access.log
output.kafka:
  hosts: ["10.10.6.225:9092"]
  topic: need_process_nginx_access_log
  partition.round_robin:
    reachable_only: false
  required_acks: 1

注：output到kafka的partitions策略：分为random、round_robin、hash，默认是hash

具体说明：https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#_literal_partition_literal

运行

./filebeat -e -c kafka.yml

 后台运行

nohup /usr/local/filebeat/filebeat -e -c /usr/local/filebeat/kafka.yml >/dev/null 2>&1 &