ELK Nxlog->Kafka->ElasticSearch

Windows 系统下，log4日志通过kafka发送到elasticsearch; windows 下nxlog没有找到直接发送数据到kafka的插件，所以采用logstash中转下

   

Nxlog-->logstash(tcp)-->kafka(producer)-->logstash(consumer)-->elasticsearch

   

   

nxlog配置

   

<Input kafkalog4>

Module im_file

File "E:\\log\\webapi\\\kafka.txt"

SavePos TRUE                

</Input>

   

<Output kafkalog4out>

Module om_tcp

Host 127.0.0.1

Port 6666

</Output>

   

<Route jsonruby>

Path kafkalog4 => kafkalog4out

</Route>

   

Logstash配置

input {

tcp {

port=>6666

codec => multiline {

charset =>"locale"

pattern => "^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}\,\d{3}"

negate => true

what => "previous"

}

type=>"kafkain"

}

   

kafka {

zk_connect => "127.0.0.1:2181"

topic_id => "test"

codec => plain

reset_beginning => false

consumer_threads => 5

decorate_events => true

}

   

}

   

output {

 

if [type=="kafkain"]

{

kafka {

bootstrap_servers => "localhost:9092"

topic_id => "test"

codec => plain {

format => "%{message}"

}

}

}

else{

elasticsearch {

hosts => ["localhost:9200"]

index => "test-kafka-%{+YYYY-MM}"

}

}

}

   

   

Logstash 文件在使用前用

logstash -f logstash.conf --configtest --verbose

检测下是否正确

   

显示OK的话起动logstash如下命令

Logstash agent -f logstash.conf

   

kibana显示日志如下