Flink – window operator
参考，
http://wuchong.me/blog/2016/05/25/flink-internals-window-mechanism/
http://wuchong.me/blog/2016/06/06/flink-internals-session-window/
WindowOperator
window operator通过WindowAssigner和Trigger来实现它的逻辑
当一个element到达时，通过KeySelector先assign一个key，并且通过WindowAssigner assign若干个windows，这样这个element会被放入若干个pane
一个pane会存放所有相同key和相同window的elements
/** * An operator that implements the logic for windowing based on a {@link WindowAssigner} and * {@link Trigger}. * * <p> * When an element arrives it gets assigned a key using a {@link KeySelector} and it gets * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element * is put into panes. A pane is the bucket of elements that have the same key and same * {@code Window}. An element can be in multiple panes if it was assigned to multiple windows by the * {@code WindowAssigner}. * * <p> * Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when * the contents of the pane should be processed to emit results. When a trigger fires, * the given {@link InternalWindowFunction} is invoked to produce the results that are emitted for * the pane to which the {@code Trigger} belongs. * * @param <K> The type of key returned by the {@code KeySelector}. * @param <IN> The type of the incoming elements. * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}. * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns. */ @Internal public class WindowOperator<K, IN, ACC, OUT, W extends Window> extends AbstractUdfStreamOperator<OUT, InternalWindowFunction<ACC, OUT, K, W>> implements OneInputStreamOperator<IN, OUT>, Triggerable, InputTypeConfigurable { // ------------------------------------------------------------------------ // Configuration values and user functions // ------------------------------------------------------------------------ protected final WindowAssigner<? super IN, W> windowAssigner; protected final KeySelector<IN, K> keySelector; protected final Trigger<? super IN, ? super W> trigger; protected final StateDescriptor<? extends AppendingState<IN, ACC>, ?> windowStateDescriptor; /** * The allowed lateness for elements. This is used for: * <ul> * <li>Deciding if an element should be dropped from a window due to lateness. * <li>Clearing the state of a window if the system time passes the * {@code window.maxTimestamp + allowedLateness} landmark. * </ul> */ protected final long allowedLateness; //允许late多久，即当watermark已经触发后 /** * To keep track of the current watermark so that we can immediately fire if a trigger * registers an event time callback for a timestamp that lies in the past. */ protected transient long currentWatermark = Long.MIN_VALUE; protected transient Context context = new Context(null, null); //Trigger Context protected transient WindowAssigner.WindowAssignerContext windowAssignerContext; //只为获取getCurrentProcessingTime // ------------------------------------------------------------------------ // State that needs to be checkpointed // ------------------------------------------------------------------------ /** * Processing time timers that are currently in-flight. */ protected transient PriorityQueue<Timer<K, W>> processingTimeTimersQueue; //Timer用于存储timestamp，key，window, queue按时间排序 /** * Current waiting watermark callbacks. */ protected transient Set<Timer<K, W>> watermarkTimers; protected transient PriorityQueue<Timer<K, W>> watermarkTimersQueue; // protected transient Map<K, MergingWindowSet<W>> mergingWindowsByKey; //用于记录merge后的stateWindow和window的对应关系
对于window operator而已，最关键的是WindowAssigner和Trigger
WindowAssigner
WindowAssigner，用于指定一个tuple应该被分配到那些windows去
借用个图，可以看出有多少种WindowAssigner
对于WindowAssigner，最关键的接口是，assignWindows
为一个element，分配一组windows， Collection<W>
@PublicEvolving public abstract class WindowAssigner<T, W extends Window> implements Serializable { private static final long serialVersionUID = 1L; /** * Returns a {@code Collection} of windows that should be assigned to the element. * * @param element The element to which windows should be assigned. * @param timestamp The timestamp of the element. * @param context The {@link WindowAssignerContext} in which the assigner operates. */ public abstract Collection<W> assignWindows(T element, long timestamp, WindowAssignerContext context); /** * Returns the default trigger associated with this {@code WindowAssigner}. */ public abstract Trigger<T, W> getDefaultTrigger(StreamExecutionEnvironment env); /** * Returns a {@link TypeSerializer} for serializing windows that are assigned by * this {@code WindowAssigner}. */ public abstract TypeSerializer<W> getWindowSerializer(ExecutionConfig executionConfig);
实际看下，具体WindowAssigner的实现
public class TumblingProcessingTimeWindows extends WindowAssigner<Object, TimeWindow> { @Override public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerContext context) { final long now = context.getCurrentProcessingTime(); long start = now - (now % size); return Collections.singletonList(new TimeWindow(start, start + size)); //很简单，分配一个TimeWindow } @Override public Trigger<Object, TimeWindow> getDefaultTrigger(StreamExecutionEnvironment env) { return ProcessingTimeTrigger.create(); //默认给出的是ProcessingTimeTrigger，如其名 }
public class SlidingEventTimeWindows extends WindowAssigner<Object, TimeWindow> { private final long size; private final long slide; @Override public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerContext context) { if (timestamp > Long.MIN_VALUE) { List<TimeWindow> windows = new ArrayList<>((int) (size / slide)); long lastStart = timestamp - timestamp % slide; for (long start = lastStart; start > timestamp - size; start -= slide) { windows.add(new TimeWindow(start, start + size)); //可以看到这里会assign多个TimeWindow，因为是slide } return windows; } else { } } @Override public Trigger<Object, TimeWindow> getDefaultTrigger(StreamExecutionEnvironment env) { return EventTimeTrigger.create(); }
Trigger, Evictor
下面看看3个主要的接口，分别触发，onElement，onEventTime，onProcessingTime
processElement
处理element到达的逻辑，触发onElement
public void processElement(StreamRecord<IN> element) throws Exception { Collection<W> elementWindows = windowAssigner.assignWindows( //通过WindowAssigner为element分配一系列windows element.getValue(), element.getTimestamp(), windowAssignerContext); final K key = (K) getStateBackend().getCurrentKey(); if (windowAssigner instanceof MergingWindowAssigner) { //如果是MergingWindow //....... } else { //如果是普通window for (W window: elementWindows) { // drop if the window is already late if (isLate(window)) { //late data的处理，默认是丢弃 continue; } AppendingState<IN, ACC> windowState = getPartitionedState( //从backend中取出该window的状态，就是buffer的element window, windowSerializer, windowStateDescriptor); windowState.add(element.getValue()); //把当前的element加入buffer state context.key = key; context.window = window; //context的设计相当tricky和晦涩 TriggerResult triggerResult = context.onElement(element); //触发onElment，得到triggerResult if (triggerResult.isFire()) { //对triggerResult做各种处理 ACC contents = windowState.get(); if (contents == null) { continue; } fire(window, contents); //如果fire，真正去计算窗口中的elements } if (triggerResult.isPurge()) { cleanup(window, windowState, null); //purge，即去cleanup elements } else { registerCleanupTimer(window); } } } }
判断是否是late data的逻辑
protected boolean isLate(W window) { return (windowAssigner.isEventTime() && (cleanupTime(window) <= currentWatermark)); } private long cleanupTime(W window) { long cleanupTime = window.maxTimestamp() + allowedLateness; //allowedLateness; return cleanupTime >= window.maxTimestamp() ? cleanupTime : Long.MAX_VALUE; }
fire逻辑
private void fire(W window, ACC contents) throws Exception { timestampedCollector.setAbsoluteTimestamp(window.maxTimestamp()); userFunction.apply(context.key, context.window, contents, timestampedCollector); }
processWatermark
处理watermark，onEvent触发
@Override public void processWatermark(Watermark mark) throws Exception { boolean fire; do { Timer<K, W> timer = watermarkTimersQueue.peek(); //这叫watermarkTimersQueue，是否有些歧义，叫eventTimerQueue更好理解些 if (timer != null && timer.timestamp <= mark.getTimestamp()) { fire = true; watermarkTimers.remove(timer); watermarkTimersQueue.remove(); context.key = timer.key; context.window = timer.window; setKeyContext(timer.key); //stateBackend.setCurrentKey(key); AppendingState<IN, ACC> windowState; MergingWindowSet<W> mergingWindows = null; if (windowAssigner instanceof MergingWindowAssigner) { //MergingWindow mergingWindows = getMergingWindowSet(); W stateWindow = mergingWindows.getStateWindow(context.window); if (stateWindow == null) { // then the window is already purged and this is a cleanup // timer set due to allowed lateness that has nothing to clean, // so it is safe to just ignore continue; } windowState = getPartitionedState(stateWindow, windowSerializer, windowStateDescriptor); } else { //普通window windowState = getPartitionedState(context.window, windowSerializer, windowStateDescriptor); //取得window的state } ACC contents = windowState.get(); if (contents == null) { // if we have no state, there is nothing to do continue; } TriggerResult triggerResult = context.onEventTime(timer.timestamp); //触发onEvent if (triggerResult.isFire()) { fire(context.window, contents); } if (triggerResult.isPurge() || (windowAssigner.isEventTime() && isCleanupTime(context.window, timer.timestamp))) { cleanup(context.window, windowState, mergingWindows); } } else { fire = false; } } while (fire); //如果fire为true，继续看下个waterMarkTimer是否需要fire output.emitWatermark(mark); //把waterMark传递下去 this.currentWatermark = mark.getTimestamp(); //更新currentWaterMark }
trigger
首先，这个函数的命名有问题，为何和前面的process…不匹配
这个是用来触发onProcessingTime，这个需要依赖系统时间的定时器来触发，逻辑和processWatermark基本等同，只是触发条件不一样
@Override public void trigger(long time) throws Exception { boolean fire; //Remove information about the triggering task processingTimeTimerFutures.remove(time); processingTimeTimerTimestamps.remove(time, processingTimeTimerTimestamps.count(time)); do { Timer<K, W> timer = processingTimeTimersQueue.peek(); if (timer != null && timer.timestamp <= time) { fire = true; processingTimeTimers.remove(timer); processingTimeTimersQueue.remove(); context.key = timer.key; context.window = timer.window; setKeyContext(timer.key); AppendingState<IN, ACC> windowState; MergingWindowSet<W> mergingWindows = null; if (windowAssigner instanceof MergingWindowAssigner) { mergingWindows = getMergingWindowSet(); W stateWindow = mergingWindows.getStateWindow(context.window); if (stateWindow == null) { // then the window is already purged and this is a cleanup // timer set due to allowed lateness that has nothing to clean, // so it is safe to just ignore continue; } windowState = getPartitionedState(stateWindow, windowSerializer, windowStateDescriptor); } else { windowState = getPartitionedState(context.window, windowSerializer, windowStateDescriptor); } ACC contents = windowState.get(); if (contents == null) { // if we have no state, there is nothing to do continue; } TriggerResult triggerResult = context.onProcessingTime(timer.timestamp); if (triggerResult.isFire()) { fire(context.window, contents); } if (triggerResult.isPurge() || (!windowAssigner.isEventTime() && isCleanupTime(context.window, timer.timestamp))) { cleanup(context.window, windowState, mergingWindows); } } else { fire = false; } } while (fire); }
EvictingWindowOperator
Evicting对于WindowOperator而言，就是多了Evictor
private void fire(W window, Iterable<StreamRecord<IN>> contents) throws Exception { timestampedCollector.setAbsoluteTimestamp(window.maxTimestamp()); // Work around type system restrictions... int toEvict = evictor.evict((Iterable) contents, Iterables.size(contents), context.window); //执行evict FluentIterable<IN> projectedContents = FluentIterable .from(contents) .skip(toEvict) .transform(new Function<StreamRecord<IN>, IN>() { @Override public IN apply(StreamRecord<IN> input) { return input.getValue(); } }); userFunction.apply(context.key, context.window, projectedContents, timestampedCollector); }
关键的逻辑就是在fire的时候，在apply function之前，会先remove需要evict的elements
