Flume-ng 配置以及简单的例子[转]

一、简单测试配置信息

flume-ng启动agent的命令为：

$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template

添加如下配置文件

avro2log.conf

# avro2log.conf: A single-node Flume configuration

# excute next command in flumeng home dir

# bin/flume-ng agent --conf conf --conf-file conf/avro2log.conf --name agent1 -Dflume.root.logger=INFO,console

# Name the components on this agent

agent1.sources = source1

agent1.sinks = sink1

agent1.channels = channel1

# Describe/configure source1

agent1.sources.source1.type = avro

agent1.sources.source1.bind = localhost

agent1.sources.source1.port = 44444

# Describe sink1

agent1.sinks.sink1.type = logger

# Use a channel which buffers events in memory

agent1.channels.channel1.type = memory

agent1.channels.channel1.capacity = 1000

agent1.channels.channel1.transactionCapactiy = 100

# Bind the source and sink to the channel

agent1.sources.source1.channels = channel1

agent1.sinks.sink1.channel = channel1

现在我们可以启动看看我们的flume-ng是否成功。

在flume-ng的home目录下

$ bin/flume-ng agent --conf-file avro2log.conf --name agent1 -Dflume.root.logger=INFO,console --conf = conf

启动之后日志信息如下：

2012-08-08 14:53:17,522 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1

2012-08-08 14:53:17,527 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1

2012-08-08 14:53:17,530 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting

2012-08-08 14:53:17,533 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10

2012-08-08 14:53:17,531 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting

2012-08-08 14:53:17,534 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/avro2log.conf

2012-08-08 14:53:17,540 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:sink1

2012-08-08 14:53:17,541 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:sink1

2012-08-08 14:53:17,541 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: sink1 Agent: agent1

2012-08-08 14:53:17,562 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent1]

2012-08-08 14:53:17,562 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels

2012-08-08 14:53:17,622 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: channel1, registered successfully.

2012-08-08 14:53:17,622 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel channel1

2012-08-08 14:53:17,634 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: SOURCE, name: source1, registered successfully.

2012-08-08 14:53:17,646 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: sink1, type: logger

2012-08-08 14:53:17,650 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:92)] Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: { source:Avro source source1: { bindAddress: localhost, port: 44444 } }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@18f1d7e counterGroup:{ name:null counters:{} } }} channels:{channel1=org.apache.flume.channel.MemoryChannel@d9660d} }

2012-08-08 14:53:17,652 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:99)] Starting Channel channel1

2012-08-08 14:53:17,653 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: CHANNEL, name: channel1 started

2012-08-08 14:53:17,653 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] Waiting for channel: channel1 to start. Sleeping for 500 ms

2012-08-08 14:53:18,156 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:127)] Starting Sink sink1

2012-08-08 14:53:18,156 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:138)] Starting Source source1

2012-08-08 14:53:18,157 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:138)] Starting Avro source source1: { bindAddress: localhost, port: 44444 }...

2012-08-08 14:53:18,565 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: SOURCE, name: source1 started

2012-08-08 14:53:18,565 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:155)] Avro source source1 started.

打开另外一个consloe:

在目录/data/flumetmp下创建文件a

执行echo 11111111 >>a

我们再打开另外一个console：执行

bin/flume-ng avro-client --conf conf -H localhost -p 44444 -F /data/flumetmp/a

显示：

+ exec /opt/soft/jdk/bin/java -Xmx20m -cp '/data/software/flumeng/conf:/data/software/flumeng/lib/*' -Djava.library.path= org.apache.flume.client.avro.AvroCLIClient -H localhost -p 44444 -F /data/flumetmp/a

在日志控制台，我们看到如下信息

2012-08-08 14:53:18,565 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: SOURCE, name: source1 started

2012-08-08 14:53:18,565 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:155)] Avro source source1 started.

2012-08-08 15:27:30,298 (New I/O server boss #1 ([id: 0x0050a649, /127.0.0.1:44444])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:141)] [id: 0x01c208b0, /127.0.0.1:55565 => /127.0.0.1:44444] OPEN

2012-08-08 15:27:30,304 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:141)] [id: 0x01c208b0, /127.0.0.1:55565 => /127.0.0.1:44444] BOUND: /127.0.0.1:44444

2012-08-08 15:27:30,749 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:141)] [id: 0x01c208b0, /127.0.0.1:55565 :> /127.0.0.1:44444] DISCONNECTED

2012-08-08 15:27:33,227 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 31 31 31 31 31 111111 }

avro source执行成功！

posted on 2013-09-11 13:55 乔峰的乔阅读(8339) 评论(0) 编辑收藏举报