EFLFK部署

EFLFK=ES+Filebeat+logstash+kafka+kabana

服务器类型	系统和IP地址	需要安装的组件	硬件方面
node1节点	192.168.239.10	JDK、elasticsearch-6.7.2	2核4G
node2节点	192.168.239.20	JDK、elasticsearch-6.7.2	2核4G
apache节点	192.168.239.30	JDK、apache、logstash-6.7.2、kabana-6.7.2、filebeat-6.7.2	2核4G
kafka40节点	192.168.239.40	JDK、zookeeper-3.6.3、kafka_2.13-2.7.1	2核4G
kafka50节点	192.168.239.50	JDK、zookeeper-3.6.3、kafka_2.13-2.7.1	2核4G
Zookeeper	192.168.239.60	JDK、zookeeper-3.6.3、kafka_2.13-2.7.1	2核4G

部署 Zookeeper+Kafka 集群

部署 ELFK

apache节点修改filebeat配置文件

cd /usr/local/filebeat
vim filebeat.yml
#注释logstash添加输出到 Kafka 的配置其他配置不动
 output.kafka:
167   enabled: true
168   hosts: ["192.168.239.40:9092","192.168.239.50:9092","192.168.239.60:9092"]
169  #指定 Kafka 集群配置
topic: "apache"  #指定 Kafka 的 topic  

# 启动 filebeat
./filebeat -e -c filebeat.yml

在 Logstash 组件所在节点上新建一个 Logstash 配置文件

cd /etc/logstash/conf.d/

vim kafka.conf
input {
    kafka {
        bootstrap_servers => "192.168.239.40:9092,192.168.239.50:9092,192.168.239.60:9092" 
        topics  => "apache"
        type => "apache_access"
        codec => "json"
        auto_offset_reset => "latest"
        decorate_events => true
    }
}

output {
    elasticsearch {
        hosts => ["192.168.239.10:9200","192.168.239.20:9200"]
        index => "kafka-%{+YYYY.MM.dd}"
    }
    stdout {
        codec => rubydebug
    }
}

#启动 logstash
logstash -f filebeat.conf