hadoop flume

flume将所有主机hadoop的日志同步到hdfs里

(4.5.7.8)

flume 需要设置第四章hadoop配置文件，启动hadoop，zookeeper集群设置，hbase配置

上传 flume 文件

解压到/usr/local/src 并改名

tar  xf apache-flume-1.6.0-bin.tar.gz -C  /usr/local/src/
cd /usr/local/src/
mv apache-flume-1.6.0-bin/ flume
chown -R hadoop:hadoop  flume

添加环境变量

vi /etc/profile
export FLUME_HOME=/usr/local/src/flume
export PATH=$PATH:$FLUME_HOME/bin

切换到hadoop操作

su - hadoop
source /etc/profile
cd /usr/local/src/flume/conf
cp flume-env.sh.template flume-env.sh
vi /usr/local/src/flume/conf/flume-env.sh   #添加变量
    export JAVA_HOME=/usr/local/src/jdk1.8.0_152

查询版本

[hadoop@master conf]$ flume-ng version
Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f
[hadoop@master conf]$ 

在 Flume 安装目录中创建 simple-hdfs-flume.conf 文件

[hadoop@master flume]$ cat simple-hdfs-flume.conf 
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/usr/local/src/hadoop/logs/
a1.sources.r1.fileHeader=true
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://master:9000/tmp/flume
a1.sinks.k1.hdfs.rollsize=1048760
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.rollInterval=900
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.channels.c1.type=file
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

注释 /usr/local/src/hbase/conf/hbase-env.sh 里的一行

#export HBASE_CLASSPATH=/usr/local/src/hadoop/etc/hadoop/

启动日志

关闭hdfs安全模式  hdfs dfsadmin -safemode leave
hdfs dfs -mkdir /tmp/f1ume   #创建文件夹
flume-ng agent --conf-file simple-hdfs-flume.conf  --name a1
hdfs dfs -ls /tmp/f1ume   #查看该目录
可以改权限在web上访问
[hadoop@master flume]$ hdfs dfs -chmod 777 /tmp/flume
[hadoop@master flume]$ hdfs dfs -chmod 777 /tmp
[hadoop@master flume]$ 

这样的结果算成功