logstash读取多个配置文件启动

多个配置文件

user.conf

input {
	kafka {
		bootstrap_servers => "hadoop003:9092"
		client_id => "user"
		group_id => "yiyang_user"
		auto_offset_reset => "latest"
		consumer_threads => 5
		decorate_events => true
		topics => "yiyang_2"
		type => "user"
		codec => "json"
	}
}

filter {
	json {
		source => "message"
	}
}

output {
	if [type] == "user" {
		elasticsearch {
			hosts => ["hadoop003:9200"]
			index => "user_%{YYYY-MM-dd}"
		}
	}
	stdout { codec => rubydebug }
}

record_traffic.conf

input {
	kafka {
		bootstrap_servers => "hadoop003:9092"
		client_id => "record_traffic"
		group_id => "yiyang_record_traffic"
		auto_offset_reset => "latest"
		consumer_threads => 5
		decorate_events => true
		topics => "yiyang"
		type => "record_traffic"
		codec => "json"
	}
}

filter {
	json {
		source => "message"
	}
}

output {
	if [type] == "record_traffic" {
		elasticsearch {
			hosts => ["hadoop003:9200"]
			index => "record_traffic-%{YYYY-MM-dd}"
		}
	}
	stdout { codec => rubydebug }
}

将这两个配置文件全部放到config目录下.
**注意：因为input是两个kafka，所以必须指定client_id **

启动logstash

[root@hadoop003 logstash-7.6.2]# bin/logstash -f config/

启动完成之后，我们开始测试

我们先发送user的数据到kafka。我们发送一条数据，logstash的日志，输出了两次
从图上可以看出，多了一个type字段

但是最终还是存储一条数据。至于日志为什么打印两次，目前还不清楚。