flume1.7.0的安装与使用

1. 官网下载flume1.7.0
2. 安装Java，与前文hadoop安装的方法相同
3. 解压并进入flume1.7.0

$ tar -zxvf apache-flume-1.7.0-bin.tar.gz

$ cd apache-flume-1.7.0-bin

4、创建flume-env.sh文件

   cp conf/flume-env.sh.template conf/flume-env.sh

5、配置java的环境变量

vim  conf/flume-env.sh

export  JAVA_HOME=/usr/java/latest/

6、创建配置文件

$ cp conf/flume-conf.properties.template conf/flume.conf

7、编辑文件flume.conf：

$ vim conf/flume.conf

a1.sources = s1

a1.channels = c1

a1.sinks = k1

 

# define the source

a1.sources.s1.type = spooldir  #上传目录类型

a1.sources.s1.spoolDir = /ttt   #上传该目录下的所有文件

a1.sources.files_source.fileHeader = true

a1.sources.files_source.fileHeaderKey = fileName

 

#define the channel

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 1000

 

# define the sink

a1.sinks.k1.type = hdfs

a1.sinks.k1.hdfs.path = hdfs://master:9000/aaa  #上传的文件保存在hdfs的/aaa目录下

a1.sinks.k1.hdfs.fileType = SequenceFile

a1.sinks.hdfs_sink.hdfs.filePrefix = %{fileName}

 

# zuhe

a1.sources.s1.channels = c1

a1.sinks.k1.channel = c1

 

8、拷贝jar包

将/home/hadoop/hadoop-2.7.2/中的以下文件拷贝到目录/apache-flume-1.7.0-bin/lib下

share/hadoop/common/hadoop-common-2.7.2.jar

share/hadoop/common/lib/commons-configuration-1.6.jar

share/hadoop/common/lib/hadoop-auth-2.7.2.jar

share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar

share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar

share/hadoop/common/lib/common-io-2.4.jar

9、启动flume1.7.0

bin/flume-ng agent --conf ./conf/ -f ./conf/flume.conf --name a1 -Dflume.root.logger=DEBUG,console     注意该命令的a1,要与配置文件中的a1相同