flink定时器使用问题
flink定时器的使用,需要涉及flink time、water mark、keyStream、keyState等概述,尽管关于flink time和water mark的文章烂大街,但还是有必要先简单介绍一下,有助于解释下面flink定时器使用遇到的问题。
时间模型
flink在streaming程序中支持三种不同的时间模型
- event time:事件发生时间。根据事件时间处理,可能需要等待一定时间的延迟事件和处理无序事件,事件时间常常跟处理时间操作一起使用。基于事件时间处理的优势在于,无论是在处理实时的数据还是重新处理历史的数据,基于事件时间创建的流计算应用都能保证结果是一样的。
- ingestion time:进入flink的时间(source operator分配的时间)。不能处理任何无序事件或者延迟事件,优点是程序无需指定如何产生水印。
- processing time:flink执行window操作的时间。处理时间最简单,有最好的性能和最低的延迟,缺点是无法处理事件乱序问题。
底层实现其实就两种:event time和processing time,ingestion time也算是processing time的一种,同一个事件的时间先后顺序:event time、ingestion time、processing time。
- event time
- event time
- processing time
- ingestion time
- processing time
water mark
使用event time会有乱序问题,解决时间乱序问题需要依赖于water mark,water mark的生成分两种
- 周期性水印:分配时间戳并定期生成水印(这可能依赖于流元素,或者纯粹基于处理时间。
- AssignerWithPeriodicWatermarks
- AscendingTimestampExtractor:递增时间戳的分配器
- BoundedOutOfOrdernessTimestampExtractor:允许固定时间延迟的时间戳分配器
- 带断点水印:当某一事件到达需要创建新的water mark时,使用AssignerWithPunctuatedWatermarks。
定时器使用
flink定时器最常见的使用是配合KeyedProcessFunction使用,在其processElement()方法中注册定时器,onTimer()方法作为Timer触发时的回调逻辑。
如果是周期性处理,在onTimer()方法内再注册定时器,这样只要有第一个事件进入之后,processElement()注册了定时器,到时间触发onTimer()回调,后面每到onTimer()设置的时间都会继续触发onTimer()回调。
根据时间特征不同分为两种:
- 处理时间——调用Context.timerService().registerProcessingTimeTimer()注册,在系统时间戳达到Timer设定的时间戳时触发调用onTimer()
- 事件时间——调用Context.timerService().registerEventTimeTimer()注册;在水印达到或超过Timer设定的时间戳时触发onTimer()
示例代码:
- import org.apache.flink.api.common.state.ValueState;
- import org.apache.flink.api.common.state.ValueStateDescriptor;
- import org.apache.flink.api.java.tuple.Tuple;
- import org.apache.flink.api.java.tuple.Tuple2;
- import org.apache.flink.configuration.Configuration;
- import org.apache.flink.streaming.api.TimeCharacteristic;
- import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
- import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks;
- import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
- import org.apache.flink.streaming.api.functions.source.SourceFunction;
- import org.apache.flink.streaming.api.watermark.Watermark;
- import org.apache.flink.util.Collector;
-
- import java.util.Arrays;
- import java.util.List;
- import java.util.Random;
- import java.util.concurrent.TimeUnit;
-
-
- public class TimerApp {
- public static class Counter {
- private Long lastTime = 0L;
-
- public Long getLastTime() {
- return lastTime;
- }
-
- public void setLastTime(Long lastTime) {
- this.lastTime = lastTime;
- }
- }
- public static void main(String[] args) throws Exception {
- StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
- // 如果不指定EventTime,flink默认使用ProcessingTime
- env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
-
- // 如果使用TimeCharacteristic.ProcessingTime,则需要设置:env.getConfig().setAutoWatermarkInterval(300L);
- // env.getConfig().setAutoWatermarkInterval(300L);
-
- env.setParallelism(1)
- .addSource(new SourceFunction<Tuple2<String, Long>>() {
- private Random random = new Random();
- private List<String> names = Arrays.asList("A", "B", "C", "D");
- private Boolean isRunning = true;
-
- @Override
- public void run(SourceContext<Tuple2<String, Long>> ctx) throws Exception {
- while (true) {
- int index = random.nextInt(1);
- ctx.collect(new Tuple2(names.get(index), 1L){});
- // 睡眠1小时,用于模拟长时间没有事件
- TimeUnit.SECONDS.sleep(3600);
- }
- }
-
- @Override
- public void cancel() {
- isRunning = false;
- }
- })
- .assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks<Tuple2<String, Long>>() {
- private Long maxLateness = 200L;
-
- @Override
- public long extractTimestamp(Tuple2<String, Long> element, long previousElementTimestamp) {
- return System.currentTimeMillis();
- }
-
-
- // 如果需要强制定时器生效,不管定时时间范围内有没有数据到达,则必须实际这个方法
- @Override
- public Watermark getCurrentWatermark() {
- // return the watermark as current time minus the maximum time lag
- return new Watermark(System.currentTimeMillis() - maxLateness);
- }
- })
- .keyBy(0)
- .process(new KeyedProcessFunction<Tuple, Tuple2<String, Long>, Tuple2<String, Long>>() {
- private ValueState<Counter> state;
- private Long INTERVAL = 2000L;
-
- @Override
- public void open(Configuration parameters) throws Exception {
- state = getRuntimeContext().getState(new ValueStateDescriptor<>("myState", Counter.class));
- }
-
- @Override
- public void onTimer(long timestamp, OnTimerContext ctx, Collector<Tuple2<String, Long>> out) throws Exception {
- System.out.println("onTimer:" + timestamp);
- Counter counter = state.value();
- counter.setLastTime(counter.getLastTime() + INTERVAL);
- ctx.timerService().registerEventTimeTimer(counter.getLastTime());
- state.value();
- }
-
- @Override
- public void processElement(Tuple2<String, Long> value, Context ctx, Collector<Tuple2<String, Long>> out) throws Exception {
- System.out.println("processElement:" + ctx.timestamp());
- Counter counter = state.value();
- if (counter == null) {
- counter = new Counter();
- counter.setLastTime(System.currentTimeMillis() + INTERVAL);
- }
-
- state.update(counter);
- ctx.timerService().registerEventTimeTimer(counter.getLastTime());
- }
- })
- .print("处理结果")
- ;
- env.execute();
- }
- }
- 如果不设置:env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime),flink默认使用的是TimeCharacteristic.ProcessingTime
- 如果不设置或者设置了:env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime),必须设置:env.getConfig().setAutoWatermarkInterval(xxx)
- 如果设置了env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime),则不需要设置:env.getConfig().setAutoWatermarkInterval(xxx)
- 使用周期性水印,如果没有实现getCurrentWatermark()方法,且长时间没有数据注入,定时器不生效,即使定时器设置的时间到了也不回调onTimer()方法。
- event time用registerEventTimeTimer注册定时器,processing time用registerProcessingTimeTimer注册定时器,不要混用!
- 使用ProcessingTime,但注册的时候是EventTime,如果不调用env.getConfig().setAutoWatermarkInterval()方法进行设置,那么定时器也不生效
示例代码:
- import org.apache.flink.api.common.state.ValueState;
- import org.apache.flink.api.common.state.ValueStateDescriptor;
- import org.apache.flink.api.java.tuple.Tuple;
- import org.apache.flink.api.java.tuple.Tuple2;
- import org.apache.flink.configuration.Configuration;
- import org.apache.flink.streaming.api.TimeCharacteristic;
- import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
- import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks;
- import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
- import org.apache.flink.streaming.api.functions.source.SourceFunction;
- import org.apache.flink.streaming.api.watermark.Watermark;
- import org.apache.flink.util.Collector;
-
- import java.util.Arrays;
- import java.util.List;
- import java.util.Random;
- import java.util.concurrent.TimeUnit;
-
-
- public class TimerApp {
- public static class Counter {
- private Long lastTime = 0L;
-
- public Long getLastTime() {
- return lastTime;
- }
-
- public void setLastTime(Long lastTime) {
- this.lastTime = lastTime;
- }
- }
- public static void main(String[] args) throws Exception {
- StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
- // env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime);
- // env.getConfig().setAutoWatermarkInterval(300L);
- env.setParallelism(1)
- .addSource(new SourceFunction<Tuple2<String, Long>>() {
- private Random random = new Random();
- private List<String> names = Arrays.asList("A", "B", "C", "D");
- private Boolean isRunning = true;
-
- @Override
- public void run(SourceContext<Tuple2<String, Long>> ctx) throws Exception {
- while (true) {
- int index = random.nextInt(1);
- ctx.collect(new Tuple2(names.get(index), 1L){});
- TimeUnit.SECONDS.sleep(4);
- }
- }
-
- @Override
- public void cancel() {
- isRunning = false;
- }
- })
- .assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks<Tuple2<String, Long>>() {
-
- @Override
- public long extractTimestamp(Tuple2<String, Long> element, long previousElementTimestamp) {
- return System.currentTimeMillis();
- }
-
- @Override
- public Watermark getCurrentWatermark() {
- // return the watermark as current time minus the maximum time lag
- return new Watermark(System.currentTimeMillis() - 0);
- }
- })
- .keyBy(0)
- .process(new KeyedProcessFunction<Tuple, Tuple2<String, Long>, Tuple2<String, Long>>() {
- private ValueState<Counter> state;
- private Long INTERVAL = 2000L;
-
- @Override
- public void open(Configuration parameters) throws Exception {
- state = getRuntimeContext().getState(new ValueStateDescriptor<>("myState", Counter.class));
- }
-
- @Override
- public void onTimer(long timestamp, OnTimerContext ctx, Collector<Tuple2<String, Long>> out) throws Exception {
- System.out.println("onTimer:" + timestamp);
- Counter counter = state.value();
- counter.setLastTime(counter.getLastTime() + INTERVAL);
- ctx.timerService().registerEventTimeTimer(counter.getLastTime());
- state.value();
- }
-
- @Override
- public void processElement(Tuple2<String, Long> value, Context ctx, Collector<Tuple2<String, Long>> out) throws Exception {
- System.out.println("currentWatermark:" + ctx.timerService().currentWatermark());
-
- Counter counter = state.value();
- if (counter == null) {
- counter = new Counter();
- counter.setLastTime(System.currentTimeMillis() + INTERVAL);
- }
-
- state.update(counter);
- ctx.timerService().registerEventTimeTimer(counter.getLastTime());
- }
- })
- .print("处理结果")
- ;
- env.execute();
- }
- }
输出如下
currentWatermark:-9223372036854775808
currentWatermark:-9223372036854775808
currentWatermark:-9223372036854775808
currentWatermark:-9223372036854775808……
为什么是-9223372036854775808这个值(Long的最小值)?为什么用ProcessingTime且不设置env.getConfig().setAutoWatermarkInterval()的话,但调用registerEventTimeTimer注册定时器,定时器就不生效呢?跟默认获取water mark的默认时间间隔有莫大干系:
- ProcessingTime:0ms
- EventTime:200ms
具体分析可参考这篇文章:Flink WaterMark的生成以及获取_lvwenyuan_1的博客-CSDN博客,欢迎讨论。
后续
重新讨论下长时间没有事件与定时器的关系
以下面简单的代码为例,MyTimerProcess注册和实现定时逻辑,processElement和onTimer方法分别注册定时器,实现固定间隔回调
env.addSource(new MySource()) .map(new MyMap()) .assignTimestampsAndWatermarks(new MyWaterMark()) .keyBy(0) .process(new MyTimerProcess()) .addSink(new MySink()) ;
对于长时间没有事件的定时器,分两种情况:
- 来了第一次事件后,长时没有事件进入:只要有一次事件进入之后,后面的定时触发逻辑就会生效。
- 从始至终都没有事件到达,定时器:只要从开始到现在都没有事件进入,那么后面的定时触发逻辑就不会生效。