Flume配置Multiplexing Channel Selector

1 官网内容

上面配置的是根据不同的heder当中state值走不同的channels，如果是CZ就走c1 如果是US就走c2 c3 其他默认走c4

 

2 我的详细配置信息

　　一个监听http端口 然后 配置两个channel，根据不同的Header中的state值走不同的channel

监听source的配置

	#配置文件：
	a1.sources= r1
	a1.sinks= k1 k2
	a1.channels= c1 c2
	
	#Describe/configure the source
	a1.sources.r1.type= org.apache.flume.source.http.HTTPSource
	a1.sources.r1.port= 50000
	a1.sources.r1.host= 127.0.0.1
	a1.sources.r1.selector.type= multiplexing
	a1.sources.r1.channels= c1 c2
	
	a1.sources.r1.selector.header= state
	a1.sources.r1.selector.mapping.CZ= c1
	a1.sources.r1.selector.mapping.US= c2
	a1.sources.r1.selector.default= c1
	
	#Describe the sink
	a1.sinks.k1.type= avro
	a1.sinks.k1.channel= c1
	a1.sinks.k1.hostname= 127.0.0.1
	a1.sinks.k1.port= 50001
	
	a1.sinks.k2.type= avro
	a1.sinks.k2.channel= c2
	a1.sinks.k2.hostname= 127.0.0.1
	a1.sinks.k2.port= 50002
	# Usea channel which buffers events in memory
	a1.channels.c1.type= memory
	a1.channels.c1.capacity= 1000
	a1.channels.c1.transactionCapacity= 100
	
	a1.channels.c2.type= memory
	a1.channels.c2.capacity= 1000
	a1.channels.c2.transactionCapacity= 100
	~
~

　　sink1配置

# Name the components on this agent
	a2.sources = r1
	a2.sinks = k1
	a2.channels = c1
	
	# Describe/configure the source
	a2.sources.r1.type = avro
	a2.sources.r1.channels = c1
	a2.sources.r1.bind = 127.0.0.1
	a2.sources.r1.port = 50001
	
	# Describe the sink
	a2.sinks.k1.type = logger
	a2.sinks.k1.channel = c1
	
	# Use a channel which buffers events inmemory
	a2.channels.c1.type = memory
	a2.channels.c1.capacity = 1000
	a2.channels.c1.transactionCapacity = 100
	
	

　　sink2配置

	▽
	# Name the components on this agent
	a3.sources = r1
	a3.sinks = k1
	a3.channels = c1
	
	# Describe/configure the source
	a3.sources.r1.type = avro
	a3.sources.r1.channels = c1
	a3.sources.r1.bind = 127.0.0.1
	a3.sources.r1.port = 50002
	
	# Describe the sink
	a3.sinks.k1.type = logger
	a3.sinks.k1.channel = c1
	
	# Use a channel which buffers events inmemory
	a3.channels.c1.type = memory
	a3.channels.c1.capacity = 1000
	a3.channels.c1.transactionCapacity = 100
	

　　3 在三个终端分别启动

flume-ng agent -cconf -f sink1.conf -n a2 -Dflume.root.logger=INFO,console
flume-ng agent -cconf -f sink2.conf -n a3 -Dflume.root.logger=INFO,console
flume-ng agent -c conf -f multi_source_case.conf -n a1 -Dflume.root.logger=INFO,console
		
		

　4 再开一个终端发送消息

　　

root@hdp1 /mnt/software/flume-1.6.0/flume-conf]#curl -X POST -d '[{"headers" :{"state" : "CZ"},"body" :"TEST1"}]' http://127.0.0.1:50000
[root@hdp1 /mnt/software/flume-1.6.0/flume-conf]#curl -X POST -d '[{"headers" :{"state" : "US"},"body" :"TEST2"}]' http://127.0.0.1:50000
[root@hdp1 /mnt/software/flume-1.6.0/flume-conf]#curl -X POST -d '[{"headers" :{"state" : "SH"},"body" :"TEST3"}]' http://127.0.0.1:50000	
	
	

　　

5 观察结果

state为CZ,以及其他值，走了sink1的channel，跟配置的效果一致		
	
		19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc80ff487, /127.0.0.1:36444 => /127.0.0.1:50001] OPEN
	19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc80ff487, /127.0.0.1:36444 => /127.0.0.1:50001] BOUND: /127.0.0.1:50001
	19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc80ff487, /127.0.0.1:36444 => /127.0.0.1:50001] CONNECTED: /127.0.0.1:36444
	19/02/21 05:30:38 INFO sink.LoggerSink: Event: { headers:{state=CZ} body: 54 45 53 54 31                                  TEST1 }
	
	19/02/21 05:33:35 INFO sink.LoggerSink: Event: { headers:{state=SH} body: 54 45 53 54 33                                  TEST3 }

state为US的走了sink2的channel，配置效果一致	
	
	UND: /127.0.0.1:50002
	19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc75a3add, /127.0.0.1:37036 => /127.0.0.1:50002] CONNECTED: /127.0.0.1:37036
	
	19/02/21 05:33:13 INFO sink.LoggerSink: Event: { headers:{state=US} body: 54 45 53 54 32                                  TEST2 }