各Flume的source、channel、sink解析和实例用法
常用：去官网一顿撸，有很多哦
最右侧有目录导航哟=================>>>>>>>>
一、Source:
avro
多用于复制(a1.sources.r1.selector.type = replicating)、多路复用(a1.sources.r1.selector.type = multiplexing)、负载均衡、故障转移(a1.sinkgroups.g1.processor.type = failover)
exec
一般监测启动一类的，hive.log启动日志
netcat
监控端口数据
spooling directory
监测目录下的多个文件，上传完成后文件结尾为COMPLETED，但是在上传完成后不能修改文件，否则会报错
taildir
描述：
实时监测某些文件，支持持续修改文件，支持正则表达式
Taildir Source维护了一个json格式的position File，其会定期的往position File中更新每个文件读取到的最新的位置，因此能够实现断点续传)
二、Channel:
Memory Channel
Kafka Channel(直接干到Kafka)
File Channel
三、Sink:
avro
发送到下游的avro source？都可以啊
File Roll
Stores events on the local filesystem（将事件存储在本地文件系统上）
HBase
hdfs
输出到hdfs
logger
输出到控制台
四、案例
1. 监控端口数据
案例需求：使用Flume监听一个端口，收集该端口数据，并打印到控制台。
（ source:netcat channel:memory channel sink:logger）
a1.sources = r1 a1.channels = c1 a1.sinks = k1 a1.sources.r1.type = exec a1.sources.r1.command = tail -F /opt/module/hive-3.1.2/logs/hive.log a1.channels.c1.type = memory a1.channels.c1.capacity = 10000 a1.channels.c1.transactionCapacity = 10000 a1.sinks.k1.type = hdfs a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 #设置文件上传到hdfs的路径 a1.sinks.k1.hdfs.path = hdfs://Linux201:8020/flume/hive-events/%y-%m-%d/%H #设置文件前缀 a1.sinks.k1.hdfs.filePrefix = logs- #设置每个文件每60s滚动 a1.sinks.k1.hdfs.rollInterval = 60 #设置每个文件到达128M时滚动 a1.sinks.k1.hdfs.rollSize = 134217728 #设置每多少个event就滚动一个文件（此设置就是不依据event） a1.sinks.k1.hdfs.rollCount = 0 #设置每多少个event就写入hdfs（不是文件滚动的意思） a1.sinks.k1.hdfs.batchSize = 100 #设置文件格式，此格式不会压缩（但是支持压缩？） a1.sinks.k1.hdfs.fileType = DataStream #设置时间戳四舍五入 a1.sinks.k1.hdfs.round = true #设置多长时间创建一个文件夹 a1.sinks.k1.hdfs.roundValue = 1 #设置四舍五入的值的单位 a1.sinks.k1.hdfs.roundUnit = hour #设置使用本地时间，而不是事件标头中的时间戳 a1.sinks.k1.hdfs.useLocalTimeStamp = true
2. 实时监控单个追加文件
案例需求：实时监控Hive日志，并上传到HDFS中
（source: exec channel:memory sink: hdfs）
a1.sources = r1 a1.channels = c1 a1.sinks = k1 a1.sources.r1.type = exec a1.sources.r1.command = tail -F /opt/module/hive-3.1.2/logs/hive.log a1.channels.c1.type = memory a1.channels.c1.capacity = 10000 a1.channels.c1.transactionCapacity = 10000 a1.sinks.k1.type = hdfs a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 #设置文件上传到hdfs的路径 a1.sinks.k1.hdfs.path = hdfs://Linux201:8020/flume/hive-events/%y-%m-%d/%H #设置文件前缀 a1.sinks.k1.hdfs.filePrefix = logs- #设置每个文件每60s滚动 a1.sinks.k1.hdfs.rollInterval = 60 #设置每个文件到达128M时滚动 a1.sinks.k1.hdfs.rollSize = 134217728 #设置每多少个event就滚动一个文件（此设置就是不依据event） a1.sinks.k1.hdfs.rollCount = 0 #设置每多少个event就写入hdfs（不是文件滚动的意思） a1.sinks.k1.hdfs.batchSize = 100 #设置文件格式，此格式不会压缩（但是支持压缩？） a1.sinks.k1.hdfs.fileType = DataStream #设置时间戳四舍五入 a1.sinks.k1.hdfs.round = true #设置多长时间创建一个文件夹 a1.sinks.k1.hdfs.roundValue = 1 #设置四舍五入的值的单位 a1.sinks.k1.hdfs.roundUnit = hour #设置使用本地时间，而不是事件标头中的时间戳 a1.sinks.k1.hdfs.useLocalTimeStamp = true
3. 实时监控目录下多个新文件
案例需求：使用Flume监听整个目录的新文件，并上传至HDFS
（source: spooldir channel: memory sink: hdfs）
a3.sources = r3 a3.sinks = k3 a3.channels = c3 # Describe/configure the source a3.sources.r3.type = spooldir a3.sources.r3.spoolDir = /opt/module/flume/upload a3.sources.r3.fileSuffix = .COMPLETED a3.sources.r3.fileHeader = true #忽略所有以.tmp结尾的文件，不上传 a3.sources.r3.ignorePattern = ([^ ]*\.tmp) # Describe the sink a3.sinks.k3.type = hdfs a3.sinks.k3.hdfs.path = hdfs://Linux201:8020/flume/upload/%Y%m%d/%H #上传文件的前缀 a3.sinks.k3.hdfs.filePrefix = upload- #是否按照时间滚动文件夹 a3.sinks.k3.hdfs.round = true #多少时间单位创建一个新的文件夹 a3.sinks.k3.hdfs.roundValue = 1 #重新定义时间单位 a3.sinks.k3.hdfs.roundUnit = hour #是否使用本地时间戳 a3.sinks.k3.hdfs.useLocalTimeStamp = true #积攒多少个Event才flush到HDFS一次 a3.sinks.k3.hdfs.batchSize = 100 #设置文件类型，可支持压缩 a3.sinks.k3.hdfs.fileType = DataStream #多久生成一个新的文件 a3.sinks.k3.hdfs.rollInterval = 60 #设置每个文件的滚动大小大概是128M a3.sinks.k3.hdfs.rollSize = 134217700 // ？ 134217728 #文件的滚动与Event数量无关 a3.sinks.k3.hdfs.rollCount = 0 # Use a channel which buffers events in memory a3.channels.c3.type = memory a3.channels.c3.capacity = 1000 a3.channels.c3.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r3.channels = c3 a3.sinks.k3.channel = c3
4. 实时监控目录下的多个追加文件
案例需求：使用Flume监听整个目录的实时追加文件，并上传至HDFS
（source: taildir channel: memory sink: hdfs）
a3.sources = r3 a3.sinks = k3 a3.channels = c3 # Describe/configure the source a3.sources.r3.type = TAILDIR a3.sources.r3.positionFile = /opt/module/flume/tail_dir.json a3.sources.r3.filegroups = f1 f2 a3.sources.r3.filegroups.f1 = /opt/module/flume/files/.*file.* a3.sources.r3.filegroups.f2 = /opt/module/flume/files/.*log.* # Describe the sink a3.sinks.k3.type = hdfs a3.sinks.k3.hdfs.path = hdfs://Linux201:8020/flume/upload2/%Y%m%d/%H #上传文件的前缀 a3.sinks.k3.hdfs.filePrefix = upload- #是否按照时间滚动文件夹 a3.sinks.k3.hdfs.round = true #多少时间单位创建一个新的文件夹 a3.sinks.k3.hdfs.roundValue = 1 #重新定义时间单位 a3.sinks.k3.hdfs.roundUnit = hour #是否使用本地时间戳 a3.sinks.k3.hdfs.useLocalTimeStamp = true #积攒多少个Event才flush到HDFS一次 a3.sinks.k3.hdfs.batchSize = 100 #设置文件类型，可支持压缩 a3.sinks.k3.hdfs.fileType = DataStream #多久生成一个新的文件 a3.sinks.k3.hdfs.rollInterval = 60 #设置每个文件的滚动大小大概是128M a3.sinks.k3.hdfs.rollSize = 134217700 //？ 134217728 #文件的滚动与Event数量无关 a3.sinks.k3.hdfs.rollCount = 0 # Use a channel which buffers events in memory a3.channels.c3.type = memory a3.channels.c3.capacity = 1000 a3.channels.c3.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r3.channels = c3 a3.sinks.k3.channel = c3
5. 复制和多路复用
案例需求
使用Flume-1监控文件变动，Flume-1将变动内容传递给Flume-2，Flume-2负责存储到HDFS。同时Flume-1将变动内容传递给Flume-3，Flume-3负责输出到Local FileSystem。
flume1:（source: exec channel: memory sink: avro）
flume2:（source: avro channel: memory sink: hdfs）
flume3:（source: avro channel: memory sink: File Roll Sink）
flume1:
# Name the components on this agent a1.sources = r1 a1.sinks = k1 k2 a1.channels = c1 c2 # 将数据流复制给所有channel a1.sources.r1.selector.type = replicating # Describe/configure the source a1.sources.r1.type = exec a1.sources.r1.command = tail -F /opt/module/hive/logs/hive.log a1.sources.r1.shell = /bin/bash -c # Describe the sink # sink端的avro是一个数据发送者 a1.sinks.k1.type = avro a1.sinks.k1.hostname = Linux201 a1.sinks.k1.port = 4141 a1.sinks.k2.type = avro a1.sinks.k2.hostname = Linux201 a1.sinks.k2.port = 4142 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.channels.c2.type = memory a1.channels.c2.capacity = 1000 a1.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 c2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c2
flume2:
# Name the components on this agent a2.sources = r1 a2.sinks = k1 a2.channels = c1 # Describe/configure the source # source端的avro是一个数据接收服务 a2.sources.r1.type = avro a2.sources.r1.bind = Linux201 a2.sources.r1.port = 4141 # Describe the sink a2.sinks.k1.type = hdfs a2.sinks.k1.hdfs.path = hdfs://Linux201:8020/flume2/%Y%m%d/%H #上传文件的前缀 a2.sinks.k1.hdfs.filePrefix = flume2- #是否按照时间滚动文件夹 a2.sinks.k1.hdfs.round = true #多少时间单位创建一个新的文件夹 a2.sinks.k1.hdfs.roundValue = 1 #重新定义时间单位 a2.sinks.k1.hdfs.roundUnit = hour #是否使用本地时间戳 a2.sinks.k1.hdfs.useLocalTimeStamp = true #积攒多少个Event才flush到HDFS一次 a2.sinks.k1.hdfs.batchSize = 100 #设置文件类型，可支持压缩 a2.sinks.k1.hdfs.fileType = DataStream #多久生成一个新的文件 a2.sinks.k1.hdfs.rollInterval = 600 #设置每个文件的滚动大小大概是128M a2.sinks.k1.hdfs.rollSize = 134217700 #文件的滚动与Event数量无关 a2.sinks.k1.hdfs.rollCount = 0 # Describe the channel a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a2.sources.r1.channels = c1 a2.sinks.k1.channel = c1
flume3:
注意：输出的本地目录必须是已经存在的目录，如果该目录不存在，并不会创建新的目录。
# Name the components on this agent a3.sources = r1 a3.sinks = k1 a3.channels = c2 # Describe/configure the source a3.sources.r1.type = avro a3.sources.r1.bind = Linux201 a3.sources.r1.port = 4142 # Describe the sink a3.sinks.k1.type = file_roll a3.sinks.k1.sink.directory = /opt/module/datas/flume3 # Describe the channel a3.channels.c2.type = memory a3.channels.c2.capacity = 1000 a3.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r1.channels = c2 a3.sinks.k1.channel = c2
6. 负载均衡和故障转移
案例需求：使用Flume1监控一个端口，其sink组中的sink分别对接Flume2和Flume3，采用FailoverSinkProcessor，实现故障转移的功能。
flume1:（source: netcat channel: memory sink: avro）
flume2:（source: avro channel: memory sink: logger）
flume3:（source: avro channel: memory sink: logger）
flume1:
# Name the components on this agent a1.sources = r1 a1.channels = c1 a1.sinkgroups = g1 a1.sinks = k1 k2 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 a1.sinkgroups.g1.processor.type = failover a1.sinkgroups.g1.processor.priority.k1 = 5 a1.sinkgroups.g1.processor.priority.k2 = 10 a1.sinkgroups.g1.processor.maxpenalty = 10000 # Describe the sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = Linux201 a1.sinks.k1.port = 4141 a1.sinks.k2.type = avro a1.sinks.k2.hostname = Linux201 a1.sinks.k2.port = 4142 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinkgroups.g1.sinks = k1 k2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c1
flume2:
# Name the components on this agent a2.sources = r1 a2.sinks = k1 a2.channels = c1 # Describe/configure the source a2.sources.r1.type = avro a2.sources.r1.bind = Linux201 a2.sources.r1.port = 4141 # Describe the sink a2.sinks.k1.type = logger # Describe the channel a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a2.sources.r1.channels = c1 a2.sinks.k1.channel = c1
flume3:
# Name the components on this agent a3.sources = r1 a3.sinks = k1 a3.channels = c2 # Describe/configure the source a3.sources.r1.type = avro a3.sources.r1.bind = Linux201 a3.sources.r1.port = 4142 # Describe the sink a3.sinks.k1.type = logger # Describe the channel a3.channels.c2.type = memory a3.channels.c2.capacity = 1000 a3.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r1.channels = c2 a3.sinks.k1.channel = c2
使用jps -ml查看Flume进程。
7. 聚合
案例需求：
Linux201上的Flume-1监控文件/opt/module/group.log，
Linux202上的Flume-2监控某一个端口的数据流，
Flume-1与Flume-2将数据发送给Linux203上的Flume-3，Flume-3将最终数据打印到控制台
flume1:（source: exec channel: memory sink: avro）
flume2:（source: netcat channel: memory sink: avro）
flume3:（source: avro channel: memory sink: logger）
flume1:
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = exec a1.sources.r1.command = tail -F /opt/module/group.log a1.sources.r1.shell = /bin/bash -c # Describe the sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = Linux203 a1.sinks.k1.port = 4141 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
flume2:
# Name the components on this agent a2.sources = r1 a2.sinks = k1 a2.channels = c1 # Describe/configure the source a2.sources.r1.type = netcat a2.sources.r1.bind = Linux202 a2.sources.r1.port = 44444 # Describe the sink a2.sinks.k1.type = avro a2.sinks.k1.hostname = Linux203 a2.sinks.k1.port = 4141 # Use a channel which buffers events in memory a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a2.sources.r1.channels = c1 a2.sinks.k1.channel = c1
flume3:
# Name the components on this agent a3.sources = r1 a3.sinks = k1 a3.channels = c1 # Describe/configure the source a3.sources.r1.type = avro a3.sources.r1.bind = Linux203 a3.sources.r1.port = 4141 # Describe the sink # Describe the sink a3.sinks.k1.type = logger # Describe the channel a3.channels.c1.type = memory a3.channels.c1.capacity = 1000 a3.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r1.channels = c1 a3.sinks.k1.channel = c1
五、自定义Interceptor
案例需求
使用Flume采集服务器本地日志，需要按照日志类型的不同，将不同种类的日志发往不同的分析系统。此案例中，将字母开头和数字开头的数据发送到不同的控制台中
需求分析
在实际的开发中，一台服务器产生的日志类型可能有很多种，不同类型的日志可能需要发送到不同的分析系统。此时会用到Flume拓扑结构中的Multiplexing结构，Multiplexing的原理是，根据event中Header的某个key的值，将不同的event发送到不同的Channel中，所以我们需要自定义一个Interceptor，为不同类型的event的Header中的value赋予不同的值。
在该案例中，我们以端口数据模拟日志，以数字（单个）和字母（单个）模拟不同类型的日志，我们需要自定义interceptor区分数字和字母，将其分别发往不同的分析系统（Channel）。
实现步骤
（1）创建一个maven项目，并引入以下依赖
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>flume-ng-core</artifactId>
<version>1.9.0</version>
</dependency>
（2）定义CustomInterceptor类并实现Interceptor接口
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;
import java.io.UnsupportedEncodingException;
import java.util.List;
import java.util.Map;
/**
* 根据event的首字符是字母还是数字，添加不同header
*/
public class MyInterceptor implements Interceptor {
/**
* 初始化方法
*/
public void initialize() {
}
/**
* 对一个event进行更改：我们要插入header
*
* @param event 要更改的event
* @return 更改完的event
*/
public Event intercept(Event event) {
//取出event的header和body
Map<String, String> headers = event.getHeaders();
byte[] body = event.getBody();//body是二级制数组
//根据Body首字母不同，进行不同处理
String line = null;
try {
line = new String(body, "utf-8");//将二进制数组转化为字符串,如果不加utf-8，默认使用当前环境的编解码方式
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
char first = line.charAt(0);
if ((first >= 'a' && first <= 'z') || (first >= 'A' && first <= 'Z')) {
//是字母
headers.put("AAA", "XXX");
} else if (first >= '0' && first <= '9') {
//是数字
headers.put("AAA", "YYY");
} else {
//不是字母不是数字
headers.put("AAA", "ZZZ");
}
return event;
}
/**
* 批量对一批事件进行更改
*
* @param events 要更改的一批事件
* @return 更改完的事件
*/
public List<Event> intercept(List<Event> events) {
for (Event event : events) {
intercept(event);
}
return events;
}
/**
* 关闭资源的方法
*/
public void close() {
}
/**
* 用来构建Interceptor实体的类
*/
public static class MyBuilder implements Interceptor.Builder {
//构建方法
public Interceptor build() {
return new MyInterceptor();
}
/**
* 配置方法
*
* @param context 配置文件
*/
public void configure(Context context) {
}
}
}
将其打包放入flume中的lib包下
（3）编辑flume配置文件（a1.sources.r1.interceptors.i1.type = MyInterceptor$MyBuilder使用的全类名）
为Linux201上的Flume1配置1个netcat source，1个sink group（3个avro sink），并配置相应的ChannelSelector和interceptor
a1.sources = r1 a1.sinks = k1 k2 k3 a1.channels = c1 c2 c3 a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 a1.sources.r1.interceptors = i1 a1.sources.r1.interceptors.i1.type = MyInterceptor$MyBuilder a1.sources.r1.selector.type = multiplexing a1.sources.r1.selector.header = AAA a1.sources.r1.selector.mapping.XXX = c1 a1.sources.r1.selector.mapping.YYY = c2 a1.sources.r1.selector.mapping.ZZZ = c3 a1.sinks.k1.type = avro a1.sinks.k1.hostname = Linux201 a1.sinks.k1.port = 4141 a1.sinks.k2.type=avro a1.sinks.k2.hostname = Linux202 a1.sinks.k2.port = 4242 a1.sinks.k3.type=avro a1.sinks.k3.hostname = Linux203 a1.sinks.k3.port = 4343 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.channels.c2.type = memory a1.channels.c2.capacity = 1000 a1.channels.c2.transactionCapacity = 100 a1.channels.c3.type = memory a1.channels.c3.capacity = 1000 a1.channels.c3.transactionCapacity = 100 a1.sources.r1.channels = c1 c2 c3 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c2 a1.sinks.k3.channel = c3
为Linux201上的Flume2配置一个avro source和一个logger sink
a2.sources = r1 a2.sinks = k1 a2.channels = c1 a2.sources.r1.type = avro a2.sources.r1.bind = Linux201 a2.sources.r1.port = 4141 a2.sinks.k1.type = logger a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 a2.sinks.k1.channel = c1 a2.sources.r1.channels = c1
为Linux202上的Flume3配置一个avro source和一个logger sink
a3.sources = r1 a3.sinks = k1 a3.channels = c1 a3.sources.r1.type = avro a3.sources.r1.bind = Linux202 a3.sources.r1.port = 4242 a3.sinks.k1.type = logger a3.channels.c1.type = memory a3.channels.c1.capacity = 1000 a3.channels.c1.transactionCapacity = 100 a3.sinks.k1.channel = c1 a3.sources.r1.channels = c1
为Linux203上的Flume4配置一个avro source和一个logger sink
a4.sources = r1 a4.sinks = k1 a4.channels = c1 a4.sources.r1.type = avro a4.sources.r1.bind = Linux203 a4.sources.r1.port = 4343 a4.sinks.k1.type = logger a4.channels.c1.type = memory a4.channels.c1.capacity = 1000 a4.channels.c1.transactionCapacity = 100 a4.sinks.k1.channel = c1 a4.sources.r1.channels = c1
（4）在Linux201(启两flume)，Linux202，Linux203上启动flume进程，注意先后顺序
Linux201: bin/flume-ng agent -c conf/ -n a1 -f job/flume-file-flume bin/flume-ng agent -c conf/ -n a2 -f job/flume-flume-console1 -Dflume.root.logger=INFO,console Linux202: bin/flume-ng agent -c conf/ -n a3 -f job/flume-flume-console2 -Dflume.root.logger=INFO,console Linux203: bin/flume-ng agent -c conf/ -n a4 -f job/flume-flume-console3 -Dflume.root.logger=INFO,console
（5）在Linux201使用netcat向localhost:44444发送字母和数字
（6）观察Linux201、Linux202和Linux203打印的日志
六、自定义Source
官方也提供了自定义source的接口：
https://flume.apache.org/FlumeDeveloperGuide.html#source根据官方说明自定义MySource需要继承AbstractSource类并实现Configurable和PollableSource接口。
使用场景：读取MySQL数据或者其他文件系统。
1、需求：
使用flume接收数据，并给每条数据添加前缀，输出到控制台。前缀可从flume配置文件中配置。
2. 实现相应方法：
getBackOffSleepIncrement()//失败后每次递增的时间
getMaxBackOffSleepInterval()//最多多长时间后就不source了
configure(Context context)//初始化context（读取配置文件内容）
process()//获取数据封装成event并写入channel，这个方法将被循环调用。
3. 编码
（1）导入pom依赖
<dependencies>
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>flume-ng-core</artifactId>
<version>1.9.0</version>
</dependency>
</dependencies>
（2）编写代码
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.PollableSource;
import org.apache.flume.channel.ChannelProcessor;
import org.apache.flume.conf.Configurable;
import org.apache.flume.event.SimpleEvent;
import org.apache.flume.source.AbstractSource;
/**
* @version 1.0
* @Author: zls
* @Date: 2020/10/10 12:40
* @Desc:
*/
public class MySource extends AbstractSource implements Configurable, PollableSource {
//前缀
private String prefix;
//间隔
private Long interval;
/**
* 框架调用该方法来拉取数据并处理
*
* @return 事件处理的状态
* @throws EventDeliveryException
*/
public Status process() throws EventDeliveryException {
Status status = null;
//获取ChannelProcessor
ChannelProcessor channelProcessor = getChannelProcessor();
try {
//处理事件
Event e = getSomeData();
channelProcessor.processEvent(e);
status = Status.READY;
} catch (Exception e) {
//处理异常
e.printStackTrace();
status = Status.BACKOFF;
}
return status;
}
/**
* 对于一个自定义源，获取数据的方式
*
* @return 获取到的事件
*/
private Event getSomeData() throws InterruptedException {
Event event = new SimpleEvent();
event.setBody((prefix + "Test content").getBytes());
Thread.sleep(interval);
return event;
}
/**
* 如果出现异常，停止调用Source的递增时间
*
* @return 递增时间
*/
public long getBackOffSleepIncrement() {
return 1000;
}
/**
* 停止调用Source的最大时间
*
* @return
*/
public long getMaxBackOffSleepInterval() {
return 10000;
}
/**
* 定义方法，可以用来配置我们的自定义Source
*
* @param context 配置文件
*/
public void configure(Context context) {
prefix = context.getString("XXX", "DD");
interval = context.getLong("YYY", 500L);
}
}
4. 测试
（1）打包
将写好的代码打包，并放到flume的lib目录下。
（2）编写配置文件（a1.sources.r1.type = MySource使用的全类名）
a1.sources = r1 a1.channels = c1 a1.sinks = k1 a1.sources.r1.type = MySource a1.sources.r1.XXX = Myprefix a1.sources.r1.YYY = 1000 a1.sinks.k1.type = logger a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
（3）开启flume，测试一下
[zls@Linux201 flume]$ bin/flume-ng agent -c conf/ -f job/mysource.conf -n a1 -Dflume.root.logger=INFO,console
结果：每隔1S输出以下数据
七、 自定义Sink
官方提供了自定义sink的接口：
https://flume.apache.org/FlumeDeveloperGuide.html#sink根据官方说明自定义MySink需要继承AbstractSink类并实现Configurable接口。
实现相应方法：
configure(Context context)//初始化context（读取配置文件内容）
process()//从Channel读取获取数据（event），这个方法将被循环调用。
使用场景：读取Channel数据写入MySQL或者其他文件系统。
需求：
使用flume接收数据，并在Sink端给每条数据添加前缀和后缀，输出到控制台。前后缀可在flume任务配置文件中配置。
编码
import org.apache.flume.*;
import org.apache.flume.conf.Configurable;
import org.apache.flume.sink.AbstractSink;
import java.io.IOException;
/**
* @version 1.0
* @Author: zls
* @Date: 2020/10/10 15:07
* @Desc: 将所有拿到的数据打印到控制台
*/
public class MySink extends AbstractSink implements Configurable {
private String prefix;
private String suffix;
/**
* Sink会从channel中拉取数据并处理
*
* @return
* @throws EventDeliveryException
*/
public Status process() throws EventDeliveryException {
Status status = null;
//获取Sink对应的Channel
Channel channel = getChannel();
//事务
Transaction transaction = channel.getTransaction();
transaction.begin();
try {
//做事情
//1. 从channel中拿数据
Event take = channel.take();
//2. 将数据写入到对应的sink
storeSomeData(take);
status = Status.READY;
transaction.commit();
} catch (Exception e) {
//处理异常
status = Status.BACKOFF;
transaction.rollback();
} finally {
transaction.close();
}
return status;
}
/**
* 将event数据进行处理（储存或者消费）
*
* @param take 拿到的数据
*/
private void storeSomeData(Event take) throws IOException, InterruptedException {
if (take != null) {
System.out.print(prefix);
System.out.write(take.getBody());
System.out.println(suffix);
} else {
Thread.sleep(5000L);
}
}
/**
* 配置方法：用来配置我们的sink
*
* @param context
*/
public void configure(Context context) {
prefix = context.getString("XXX", "DPre");
suffix = context.getString("YYY", "DSuf");
}
}
测试
（1）打包
将写好的代码打包，并放到flume的lib目录下。
（2）配置文件
a1.sources = r1 a1.sinks = k1 a1.channels = c1 a1.sources.r1.type = netcat a1.sources.r1.bind = 0.0.0.0 a1.sources.r1.port = 44444 a1.sinks.k1.type = MySink a1.sinks.k1.XXX = zls: a1.sinks.k1.YYY = :zls a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
（3）开启任务
[zls@Linux201 flume]$ bin/flume-ng agent -c conf/ -f job/mysink.conf -n a1 -Dflume.root.logger=INFO,console [zls@Linux201 ~]$ nc localhost 44444 1 OK hello OK
结果
