Flume

• 服务：启动后监听某个端口，等待客户端连接，并处理客户端请求。客户端断开连接后，服务程序依然运行。

• 工具：需要使用工具的功能时，启动程序。使用完毕后，工具程序可以直接关闭。

• 框架：一个半成品软件，需要开发人员根据业务逻辑填写核心代码，组成完成的程序，提供工具或者服务的功能。

是什么

• 在数据处理场景中，数据的产生往往分散在时间空间不同的各个服务器上，需要将各个服务器的数据自动化采集到同一个HDFS集群，就需要使用自动化采集工具

• flume是一个hadoop生态中的专用用来进行海量日志，收集、聚合、移动的工具

核心概念

• agent 代理 flume启动的一个Jvm实例，称之为一个Agent

• source 源 是agent中的一个组件，负责读取原始数据(文件，其他协议的网络连接) 将原始数据封装为event对象 将event对象发送到channel中

• channel 通道 通过内存或者磁盘文件构建一个用来缓存event的队列

• sink 出口 读取channel中缓存的event对象 拆解出event中的数据并发送到下游组件(HDFS)

• event 事件 flume中自定义的对象用于封装数据(比如一行日志文件封装为一个event)

flume安装

1.上传解压

2.修改配置文件

• flume-env.sh

JAVA_HOME=/opt/java/modul/jdk1.8.0_291

1. 配置环境变量

echo 'export FLUME_HOME=/opt/java/modul/flume-1.9.0' >> /etc/profile
echo 'export PATH=$PATH:$FLUME_HOME/bin' >> /etc/profile
source /etc/profile

Flume操作

• Flume通过解耦的设计将数据收集端和数据存储端以及中间的缓存拆分为了三个模块

• 使用flume进行数据采集时，只需要根据需要选择对应功能source、channel、sink，编写agent的配置文件，将三个模块组合成一个agent，启动程序即可

常用source

对比	exec	spooldir	taildir
功能	通过执行一个shell命令读取数据， 通常配合tail -F file实现单文件的实时读取	监控整个目录，读取目录中所有的文件，不支持递归	可以监控多个文件或者文件夹的变化
优势	性能最优	可以监控整个文件夹	可以监控文件或者文件夹，并支持记录读取进度
劣势	只能监控单文件，不支持断点续传	不支持断点续传，也不支持读取文件的改动	-

常用channel

对比	memory	file
性能	优	略弱
安全性	略弱	优

常用sink

• logger 通常用于测试，可以方便将数据打印到控制台

• HDFS sink 将数据写入到HDFS集群

案例一：

给agent_test_1.conf 添加配置

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动监听

flume-ng agent --conf /opt/module/flume-1.9.0/conf/ --conf-file agent_test_1.conf --name a1 -Dflume.root.logger=INFO,console

案例二：

给agent_test_1.conf 添加配置

案例三

hdfs

flume常见架构

flume2：

在idea创建项目如下

文件中配置如下

# example.conf: A single-node Flume configuration

# Name the components on this agent
#设置各组件名称
#设置资源别名r1
exec2logger.sources = r1
#设置出口 k1
exec2logger.sinks = k1
#设置通道 c1
exec2logger.channels = c1

# Describe/configure the source
#数据源的类型
exec2logger.sources.r1.type = exec
#数据源的地址
exec2logger.sources.r1.command = tail -F /opt/log.txt
#exec2logger.sources.r1.port = 44444

# Describe the sink
#输出的方式
exec2logger.sinks.k1.type = logger

# Use a channel which buffers events in memory
#缓存
exec2logger.channels.c1.type = memory
#容量
exec2logger.channels.c1.capacity = 200000
#流速
exec2logger.channels.c1.transactionCapacity = 1024

# Bind the source and sink to the channel
exec2logger.sources.r1.channels = c1
exec2logger.sinks.k1.channel = c1

将文件exec2logger 拷贝到 liunx flume 安装目录下/opt/module/flume-1.9.0/agents

启动agent

jar 冲突

案例二

编写spooldir2logger.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
#设置各组件名称
#设置资源别名r1

spooldir2logger.sources = r1
#设置出口 k1
spooldir2logger.sinks = k1
#设置通道 c1
spooldir2logger.channels = c1

# Describe/configure the source
#数据源的类型
spooldir2logger.sources.r1.type = spooldir
#数据源的地址
spooldir2logger.sources.r1.spoolDir = /opt/log
#spooldir2logger.sources.r1.port = 44444

# Describe the sink
#输出的方式
spooldir2logger.sinks.k1.type = logger

# Use a channel which buffers events in memory
#缓存
spooldir2logger.channels.c1.type = memory
#容量
spooldir2logger.channels.c1.capacity = 200000
#流速
spooldir2logger.channels.c1.transactionCapacity = 1024

# Bind the source and sink to the channel
spooldir2logger.sources.r1.channels = c1
spooldir2logger.sinks.k1.channel = c1

注意:优先创建出/opt/log 文件夹

启动agent

将log.txt 移动到log文件

给文件中添加信息 没有反应

修改后缀 数据写入

案例三

taildir2logger.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
taildir2logger.sources = r1
taildir2logger.sinks = k1
taildir2logger.channels = c1

# Describe/configure the source
taildir2logger.sources.r1.type = TAILDIR
taildir2logger.sources.r1.filegroups = g1 g2 g3
taildir2logger.sources.r1.filegroups.g1 =/opt/1.txt
taildir2logger.sources.r1.filegroups.g2 =/opt/.*.txt
taildir2logger.sources.r1.filegroups.g3 =/opt/2.txt
#taildir2logger.sources.r1.port = 44444

# Describe the sink
taildir2logger.sinks.k1.type = logger

# Use a channel which buffers events in memory
taildir2logger.channels.c1.type = memory
taildir2logger.channels.c1.capacity = 1000
taildir2logger.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
taildir2logger.sources.r1.channels = c1
taildir2logger.sinks.k1.channel = c1

检查~家目录下的.flume

案例四

taildir2hdfs.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
#设置各组件名称
#设置资源别名r1

taildir2hdfs.sources = r1
#设置出口 k1
taildir2hdfs.sinks = k1
#设置通道 c1
taildir2hdfs.channels = c1

# Describe/configure the source
#数据源的类型
taildir2hdfs.sources.r1.type = TAILDIR
#数据源的地址
taildir2hdfs.sources.r1.filegroups = g1 g2 g3
taildir2hdfs.sources.r1.filegroups.g1=/opt/1.txt
taildir2hdfs.sources.r1.filegroups.g2=/opt/log/.*.txt
taildir2hdfs.sources.r1.filegroups.g3=/opt/2.txt
#taildir2hdfs.sources.r1.port = 44444

# Describe the sink
#输出的方式
taildir2hdfs.sinks.k1.hdfs.useLocalTimeStamp=true
taildir2hdfs.sinks.k1.type = hdfs
taildir2hdfs.sinks.k1.hdfs.path = hdfs://bd0801:8020/flume/%Y-%m-%d-%H-%M
taildir2hdfs.sinks.k1.hdfs.fileType = DataStream
taildir2hdfs.sinks.k1.hdfs.writeFormat = Text
taildir2hdfs.sinks.k1.hdfs.rollInterval = 300
taildir2hdfs.sinks.k1.hdfs.rollSize = 13000000
taildir2hdfs.sinks.k1.hdfs.rollCount=0


# Use a channel which buffers events in memory
#缓存
taildir2hdfs.channels.c1.type = memory
#容量
taildir2hdfs.channels.c1.capacity = 200000
#流速
taildir2hdfs.channels.c1.transactionCapacity = 1024

# Bind the source and sink to the channel
taildir2hdfs.sources.r1.channels = c1
taildir2hdfs.sinks.k1.channel = c1

运行结果

模拟存储 vi printer.sh

#!/bin/bash


while true
do
 echo '298444.812] (--) Found matching XKB configuration "Chinese (Simplified)"
[298444.812] (--) Model = "pc105" Layout = "cn" Variant = "none" Options = "none"
[298444.812] Rules = "base" Model = "pc105" Layout = "cn" Variant = "none" Options = "none"' >> /opt/2.txt
done

执行脚本

avro.apache.org

将flume在bd0802安装

解放终端执行

启动服务端

avro2hdfs.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
#设置各组件名称
#设置资源别名r1

avro2hdfs.sources = r1
#设置出口 k1
avro2hdfs.sinks = k1
#设置通道 c1
avro2hdfs.channels = c1

# Describe/configure the source
#数据源的类型
avro2hdfs.sources.r1.type = avro
avro2hdfs.sources.r1.bind=bd0802
avro2hdfs.sources.r1.port=12345

#avro2hdfs.sources.r1.port = 44444

# Describe the sink
#输出的方式
avro2hdfs.sinks.k1.hdfs.useLocalTimeStamp=true
avro2hdfs.sinks.k1.type = hdfs
avro2hdfs.sinks.k1.hdfs.path = hdfs://bd0801:8020/flume/%Y-%m-%d
avro2hdfs.sinks.k1.hdfs.fileType = DataStream
avro2hdfs.sinks.k1.hdfs.writeFormat = Text
avro2hdfs.sinks.k1.hdfs.rollInterval = 300
avro2hdfs.sinks.k1.hdfs.rollSize = 13000000
avro2hdfs.sinks.k1.hdfs.rollCount=0


# Use a channel which buffers events in memory
#缓存
avro2hdfs.channels.c1.type = memory
#容量
avro2hdfs.channels.c1.capacity = 200000
#流速
avro2hdfs.channels.c1.transactionCapacity = 1024

# Bind the source and sink to the channel
avro2hdfs.sources.r1.channels = c1
avro2hdfs.sinks.k1.channel = c1

启动客户端

taildir2avro.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
#设置各组件名称
#设置资源别名r1

taildir2avro.sources = r1
#设置出口 k1
taildir2avro.sinks = k1
#设置通道 c1
taildir2avro.channels = c1

# Describe/configure the source
#数据源的类型
taildir2avro.sources.r1.type = TAILDIR
#数据源的地址
taildir2avro.sources.r1.filegroups = g1 g2 g3
taildir2avro.sources.r1.filegroups.g1=/opt/1.txt
taildir2avro.sources.r1.filegroups.g2=/opt/log/.*.txt
taildir2avro.sources.r1.filegroups.g3=/opt/2.txt
#taildir2avro.sources.r1.port = 44444

# Describe the sink
#输出的方式

taildir2avro.sinks.k1.type = avro
taildir2avro.sinks.k1.hostname=bd0802
taildir2avro.sinks.k1.port=12345


# Use a channel which buffers events in memory
#缓存
taildir2avro.channels.c1.type = memory
#容量
taildir2avro.channels.c1.capacity = 200000
#流速
taildir2avro.channels.c1.transactionCapacity = 1024

# Bind the source and sink to the channel
taildir2avro.sources.r1.channels = c1
taildir2avro.sinks.k1.channel = c1