Flume配置Multiplexing Channel Selector

1 官网内容

上面配置的是根据不同的heder当中state值走不同的channels，如果是CZ就走c1 如果是US就走c2 c3 其他默认走c4

2 我的详细配置信息

　　一个监听http端口然后配置两个channel，根据不同的Header中的state值走不同的channel

监听source的配置

<em id="__mceDel">  #配置文件：
    a1.sources= r1
    a1.sinks= k1 k2
    a1.channels= c1 c2
     
    #Describe/configure the source
    a1.sources.r1.type= org.apache.flume.source.http.HTTPSource
    a1.sources.r1.port= 50000
    a1.sources.r1.host= 127.0.0.1
    a1.sources.r1.selector.type= multiplexing
    a1.sources.r1.channels= c1 c2
     
    a1.sources.r1.selector.header= state
    a1.sources.r1.selector.mapping.CZ= c1
    a1.sources.r1.selector.mapping.US= c2
    a1.sources.r1.selector.default= c1
     
    #Describe the sink
    a1.sinks.k1.type= avro
    a1.sinks.k1.channel= c1
    a1.sinks.k1.hostname= 127.0.0.1
    a1.sinks.k1.port= 50001
     
    a1.sinks.k2.type= avro
    a1.sinks.k2.channel= c2
    a1.sinks.k2.hostname= 127.0.0.1
    a1.sinks.k2.port= 50002
    # Usea channel which buffers events in memory
    a1.channels.c1.type= memory
    a1.channels.c1.capacity= 1000
    a1.channels.c1.transactionCapacity= 100
     
    a1.channels.c2.type= memory
    a1.channels.c2.capacity= 1000
    a1.channels.c2.transactionCapacity= 100
    ~
~
</em>

　　sink1配置

# Name the components on this agent
    a2.sources = r1
    a2.sinks = k1
    a2.channels = c1
     
    # Describe/configure the source
    a2.sources.r1.type = avro
    a2.sources.r1.channels = c1
    a2.sources.r1.bind = 127.0.0.1
    a2.sources.r1.port = 50001
     
    # Describe the sink
    a2.sinks.k1.type = logger
    a2.sinks.k1.channel = c1
     
    # Use a channel which buffers events inmemory
    a2.channels.c1.type = memory
    a2.channels.c1.capacity = 1000
    a2.channels.c1.transactionCapacity = 100
     
    

　　sink2配置

▽
# Name the components on this agent
a3.sources = r1
a3.sinks = k1
a3.channels = c1
 
# Describe/configure the source
a3.sources.r1.type = avro
a3.sources.r1.channels = c1
a3.sources.r1.bind = 127.0.0.1
a3.sources.r1.port = 50002
 
# Describe the sink
a3.sinks.k1.type = logger
a3.sinks.k1.channel = c1
 
# Use a channel which buffers events inmemory
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100

　　3 在三个终端分别启动

flume-ng agent -cconf -f sink1.conf -n a2 -Dflume.root.logger=INFO,console<br>flume-ng agent -cconf -f sink2.conf -n a3 -Dflume.root.logger=INFO,console
flume-ng agent -c conf -f multi_source_case.conf -n a1 -Dflume.root.logger=INFO,console

　4 再开一个终端发送消息

root@hdp1 /mnt/software/flume-1.6.0/flume-conf]#curl -X POST -d '[{"headers" :{"state" : "CZ"},"body" :"TEST1"}]' http://127.0.0.1:50000
[root@hdp1 /mnt/software/flume-1.6.0/flume-conf]#curl -X POST -d '[{"headers" :{"state" : "US"},"body" :"TEST2"}]' http://127.0.0.1:50000
[root@hdp1 /mnt/software/flume-1.6.0/flume-conf]#curl -X POST -d '[{"headers" :{"state" : "SH"},"body" :"TEST3"}]' http://127.0.0.1:50000   
     
    

5 观察结果

state为CZ,以及其他值，走了sink1的channel，跟配置的效果一致     
     
        19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc80ff487, /127.0.0.1:36444 => /127.0.0.1:50001] OPEN
    19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc80ff487, /127.0.0.1:36444 => /127.0.0.1:50001] BOUND: /127.0.0.1:50001
    19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc80ff487, /127.0.0.1:36444 => /127.0.0.1:50001] CONNECTED: /127.0.0.1:36444
    19/02/21 05:30:38 INFO sink.LoggerSink: Event: { headers:{state=CZ} body: 54 45 53 54 31                                  TEST1 }
     
    19/02/21 05:33:35 INFO sink.LoggerSink: Event: { headers:{state=SH} body: 54 45 53 54 33                                  TEST3 }
 
state为US的走了sink2的channel，配置效果一致 
     
    UND: /127.0.0.1:50002
    19/02/21 05:29:46 INFO ipc.NettyServer: [id: 0xc75a3add, /127.0.0.1:37036 => /127.0.0.1:50002] CONNECTED: /127.0.0.1:37036
     
    19/02/21 05:33:13 INFO sink.LoggerSink: Event: { headers:{state=US} body: 54 45 53 54 32                                  TEST2 }