DStream-04 Window函数的原理和源码
DStream 中 window 函数有两种，一种是普通 WindowedDStream，另外一种是针对 window聚合 优化的 ReducedWindowedDStream。
Demo
object SocketWordCountDstreamReduceByWindow {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf()
.setAppName("SocketWordCountDstream")
.setMaster("local[3]")
val sparkStreamContext = new StreamingContext(sparkConf,Seconds(5))
val sparkContext = sparkStreamContext.sparkContext
sparkContext.setLogLevel("WARN")
val dstream = sparkStreamContext.socketTextStream("localhost",9090)
val v = dstream.flatMap(_.split(" "))
.map((_,1))
.reduceByKeyAndWindow((p:Int,c:Int)=>{
p + c
},Durations.seconds(15),Durations.seconds(5))
v.foreachRDD(...)
sparkStreamContext.start()
sparkStreamContext.awaitTermination()
}
}
源码
DStream
前提知识
在每个DStream 中会把每个batch 产生的 Rdd 放入Map中，也就是放到内存中。
// 保存RDD的 Map
@transient
private[streaming] var generatedRDDs = new HashMap[Time, RDD[T]]()
private[streaming] final def getOrCompute(time: Time): Option[RDD[T]] = {
// If RDD was already generated, then retrieve it from HashMap,
// or else compute the RDD
// 如果map 中就直接拿着用，没有就创建
generatedRDDs.get(time).orElse {
if (isTimeValid(time)) {
val rddOption = createRDDWithLocalProperties(time, displayInnerRDDOps = false) {
SparkHadoopWriterUtils.disableOutputSpecValidation.withValue(true) {
compute(time)
}
}
rddOption.foreach { case newRDD =>
// Register the generated RDD for caching and checkpointing
if (storageLevel != StorageLevel.NONE) {
newRDD.persist(storageLevel)
logDebug(s"Persisting RDD ${newRDD.id} for time $time to $storageLevel")
}
if (checkpointDuration != null && (time - zeroTime).isMultipleOf(checkpointDuration)) {
newRDD.checkpoint()
logInfo(s"Marking RDD ${newRDD.id} for time $time for checkpointing")
}
// 放入Map中
generatedRDDs.put(time, newRDD)
}
rddOption
} else {
None
}
}
}
同时也会 batch 完成的时候去清理这个Map。
private[streaming] def clearMetadata(time: Time) {
//根据当前 batch time - rememberDuration
//time = 100030, rememberDuration = 10 , generatedRDDs = {100020->rdd}
val oldRDDs = generatedRDDs.filter(_._1 <= (time - rememberDuration))
generatedRDDs --= oldRDDs.keys
if (unpersistData) {
logDebug(s"Unpersisting old RDDs: ${oldRDDs.values.map(_.id).mkString(", ")}")
oldRDDs.values.foreach { rdd =>
rdd.unpersist(false)
// Explicitly remove blocks of BlockRDD
rdd match {
case b: BlockRDD[_] =>
logInfo(s"Removing blocks of RDD $b of time $time")
b.removeBlocks()
case _ =>
}
}
}
dependencies.foreach(_.clearMetadata(time))
}
这个清理的过程是从后往前的，先清理 子DStream 然后是父DStream
// 默认 rememberDuration = slideDuration 也就是 batchInterval 。
private[streaming] var rememberDuration: Duration = null
//所以这边 父继承子
private[streaming] def parentRememberDuration = rememberDuration
DStream 初始化时
private[streaming] def initialize(time: Time) {
var minRememberDuration = slideDuration
// checkpointDuration 一般的DStream都是空的
if (checkpointDuration != null && minRememberDuration <= checkpointDuration) {
minRememberDuration = checkpointDuration * 2
}
if (rememberDuration == null || rememberDuration < minRememberDuration) {
rememberDuration = minRememberDuration
}
// Initialize the dependencies
dependencies.foreach(_.initialize(zeroTime))
}
// 抽象方法
// slideDuration 官方的注释是：方法是 DStream生成RDD的时间间隔。
// 默认情况下 slideDuration 就是 batchInterval 。针对 WindowedDStream 就是滑动的时间 但也是批次的时间
def slideDuration: Duration
这边举个例子 interval = 1s window = 4s slide = 2s。JobGenerator 会每个隔 interval = 1s 发送 GenerateJobs 事件，然后会触发 最后一个DStream 的 getOrCompute，然后依次 compute 会优先调 parent.getOrCompute 依次递归到第一个DStream。但是 getOrCompute 会判断这个 batch 的 Time - zeroTime（zeroTime 是 Stream 开始的时间，第一个batch 就是 zeroTime + batchInterval） 是不是 slideDuration 的倍数。如果是 才会调 compute 否则 就会返回 None。
private[streaming] final def getOrCompute(time: Time): Option[RDD[T]] = {
// If RDD was already generated, then retrieve it from HashMap,
// or else compute the RDD
generatedRDDs.get(time).orElse {
// Compute the RDD if time is valid (e.g. correct time in a sliding window)
// of RDD generation, else generate nothing.
if (isTimeValid(time)) {
....
rdd
}
rddOption
} else {
None
}
}
}
校验时间。 zeroTime 就是第一个Batch的时间
private[streaming] def isTimeValid(time: Time): Boolean = {
if (!isInitialized) {
throw new SparkException (this + " has not been initialized")
} else if (time <= zeroTime || ! (time - zeroTime).isMultipleOf(slideDuration)) {
logInfo(s"Time $time is invalid as zeroTime is $zeroTime" +
s" , slideDuration is $slideDuration and difference is ${time - zeroTime}")
false
} else {
logDebug(s"Time $time is valid")
true
}
}
例如常见 MappedDStream 是父 DStream 的 slideDuration
override def slideDuration: Duration = parent.slideDuration
第一个DStream 的 slideDuration = batchDuration
override def slideDuration: Duration = {
if (ssc == null) throw new Exception("ssc is null")
if (ssc.graph.batchDuration == null) throw new Exception("batchDuration is null")
ssc.graph.batchDuration
}
InputDStream
InputDStream（输入流就是数据源） 是 DirectKafkaInputDStream 、FileInputDStream..... 的父类。 输入流都是继承这个方法。
override def slideDuration: Duration = {
if (ssc == null) throw new Exception("ssc is null")
if (ssc.graph.batchDuration == null) throw new Exception("batchDuration is null")
// 就是 new StreamContext 传入流的间隔时间
ssc.graph.batchDuration
}
默认情况下就是 slideDuration = batchInterval 批次间隔时间
也就是 rememberDuration = batchInterval ， parentRememberDuration = batchInterval 。默认只会保留上次一个batch的RDD。
进入正题。
源码的入口就是 reduceByKeyAndWindow
PairDStreamFunctions
实际就是 dstream.reduceByKey().window().reduceByKey，点开window()
def reduceByKeyAndWindow(
reduceFunc: (V, V) => V,
windowDuration: Duration,
slideDuration: Duration,
partitioner: Partitioner
): DStream[(K, V)] = ssc.withScope {
self.reduceByKey(reduceFunc, partitioner)
.window(windowDuration, slideDuration)
.reduceByKey(reduceFunc, partitioner)
}
DStream
其实只要是DStream 都有 window 函数
def window(windowDuration: Duration, slideDuration: Duration): DStream[T] = ssc.withScope {
new WindowedDStream(this, windowDuration, slideDuration)
}
WindowedDStream
跟到 WindowedDStream 类
class WindowedDStream[T: ClassTag](
parent: DStream[T],
_windowDuration: Duration,
_slideDuration: Duration)
extends DStream[T](parent.ssc) {
// Persist parent level by default, as those RDDs are going to be obviously reused.
// 默认吧 parent dsteam 进行持久化，因为 parent.dsteam中的rdd 将会被吃
parent.persist(StorageLevel.MEMORY_ONLY_SER)
// 窗口时间
def windowDuration: Duration = _windowDuration
override def dependencies: List[DStream[_]] = List(parent)
// 这边的 slideDuration 就不是parent.slideDuration 而是我们定传入 window方法的滑动间隔。
override def slideDuration: Duration = _slideDuration
// parentRememberDuration 是 slideDuration + windowDuration
override def parentRememberDuration: Duration = rememberDuration + windowDuration
override def compute(validTime: Time): Option[RDD[T]] = {
val currentWindow = new Interval(validTime - windowDuration + parent.slideDuration, validTime)
// 获取一个范围内的RDD
val rddsInWindow = parent.slice(currentWindow)
// 最后union 范围内的RDD
Some(ssc.sc.union(rddsInWindow))
}
}
DStream
def slice(fromTime: Time, toTime: Time): Seq[RDD[T]] = ssc.withScope {
if (!isInitialized) {
throw new SparkException(this + " has not been initialized")
}
// windowStream.parent 就是普通的DStream slideDuration = batchInterval
// zeroTime 是 Stream 开始的时间，第一个batch 就是 zeroTime + batchInterval
截至时间到第一个batch的时间 是不是 batchInterval 的倍数。主要是方便计算
val alignedToTime = if ((toTime - zeroTime).isMultipleOf(slideDuration)) {
toTime
} else {
logWarning(s"toTime ($toTime) is not a multiple of slideDuration ($slideDuration)")
toTime.floor(slideDuration, zeroTime)
}
// 同理 开始时间到第一个batch的时间 是不是 batchInterval 的倍数。主要是方便计算
val alignedFromTime = if ((fromTime - zeroTime).isMultipleOf(slideDuration)) {
fromTime
} else {
logWarning(s"fromTime ($fromTime) is not a multiple of slideDuration ($slideDuration)")
fromTime.floor(slideDuration, zeroTime)
}
logInfo(s"Slicing from $fromTime to $toTime" +
s" (aligned to $alignedFromTime and $alignedToTime)")
alignedFromTime.to(alignedToTime, slideDuration).flatMap { time =>
//最后将这一个时间段的 根据 slideDuration 来切分，然后得到之前 batch的Time
// 然后 到DStrem getOrCompute 中，从内存中重新取回来。
if (time >= zeroTime) getOrCompute(time) else None
}
}
例子
这边还是举个例子 interval = 1s window = 4s slice = 2s。zeroTime = 1582800004.000
isTimeValid = （Time - zeroTime % slide 是否为整数）
1s后 ，Time = 1582800005 ，先到到 WindowedDStream isTimeValid = false None
2s后 ，Time = 1582800006 ， 先到到 WindowedDStream isTimeValid = true :
1、val currentWindow = new Interval(validTime - windowDuration + parent.slideDuration, validTime)
第一个参数 1582800006 - 4 + 1 = 1582800003
第二个参数 1582800006
2、parent.slice(1582800003,1582800006 ) 也就是取 1582800003、1582800004、1582800005、1582800006 四个时间的RDD。1582800003、1582800004 是无效的时间 直接是None ，然后调用
getOrCompute（1582800005）和 getOrCompute（1582800006 ） ，这样就取到 1582800005 的 RDD，虽然1582800005 时刻的返回的是None 到了 1582800006 就会把1582800005 取到，依次类推就好了。
Demo 2
大家想，如果一个Window 的时间比较长，并且 reduceBykey().window().reduceBykey 涉及的计算比较慢。每次都需要重新计算 4个batch的RDD ，很浪费（前提是不是去重计算 ） 。
假设 batchInterval = 1s window = 4s sliace = 1s
这边的数字代表时间戳
第一次 计算 1、2、3、4 的RDD
第二次 计算 2、3、4、5 的RDD
第三次 计算 3、4、 5、6、 的RDD
规律就是 上一次计算的结果 可以被下一次所副复用，减少计算。怎么服用呢
第一个 1、2、3、4 减 1的RDD 加 5的RDD 就可以了。
object SocketWordCountDstreamReduceByWindowOptimization {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf()
.setAppName("SocketWordCountDstream")
.setMaster("local[3]")
val sparkStreamContext = new StreamingContext(sparkConf,Seconds(1))
val sparkContext = sparkStreamContext.sparkContext
sparkContext.setLogLevel("WARN")
val dstream = sparkStreamContext.socketTextStream("localhost",9090)
val v = dstream.flatMap(_.split(" "))
.map((_,1))
.reduceByKeyAndWindow((p:Int,c:Int)=>{
p + c
},(c:Int,p:Int)=>{
c - p
},Durations.seconds(4),Durations.seconds(2))
v.foreachRDD(rdd => {
....
})
sparkStreamContext.start()
sparkStreamContext.awaitTermination()
}
}
PairDStreamFunctions
需要传入两个关键的函数，第一个就是 减去 slide/batchinterval 个 RDD 的函数，第二个就是加上 slide/batchinterval 个 RDD 的函数。
def reduceByKeyAndWindow(
reduceFunc: (V, V) => V,
invReduceFunc: (V, V) => V,
windowDuration: Duration,
slideDuration: Duration,
partitioner: Partitioner,
filterFunc: ((K, V)) => Boolean
): DStream[(K, V)] = ssc.withScope {
val cleanedReduceFunc = ssc.sc.clean(reduceFunc)
val cleanedInvReduceFunc = ssc.sc.clean(invReduceFunc)
val cleanedFilterFunc = if (filterFunc != null) Some(ssc.sc.clean(filterFunc)) else None
new ReducedWindowedDStream[K, V](
self, cleanedReduceFunc, cleanedInvReduceFunc, cleanedFilterFunc,
windowDuration, slideDuration, partitioner
)
}
ReducedWindowedDStream
几个关键都是和window 一样
def windowDuration: Duration = _windowDuration
override def dependencies: List[DStream[_]] = List(reducedStream)
override def slideDuration: Duration = _slideDuration
// 这个就是 需要 checkpoint
override val mustCheckpoint = true
override def parentRememberDuration: Duration = rememberDuration + windowDuration
怎么获取上一个batch的RDD，可以之在map 中，根据 time - slideDuration 就是上一个批次的时间。
例如 batchInterval = 1s window = 4s slide = 2s
当前 时间 100006
上一个widow 的区间 [ 100001，100002，100003，100004 ]
第一个：需要减去的RDD 区间 [ 100001，100002 ] （区间的表示，下同）
第二个：需要加上的RDD 区间 [ 100005，100006 ]
期望 ： [100003，100006 ]
// 这个就是 parent DStream，这边只是做了一个转换。
private val reducedStream = parent.reduceByKey(reduceFunc, partitioner)
override def compute(validTime: Time): Option[RDD[(K, V)]] = {
val reduceF = reduceFunc
val invReduceF = invReduceFunc
val currentTime = validTime
// 这边计算后就是 [100006 - 4 + 1 ,100006] => [100003 ,100006]
val currentWindow = new Interval(currentTime - windowDuration + parent.slideDuration,
currentTime)
// 这边计算后就是 [100003-2,100006-2] => [100001,100004] 上一个Window的区间
val previousWindow = currentWindow - slideDuration
// _____________________________
// | previous window _________|___________________
// |___________________| current window | --------------> Time
// |_____________________________|
//
// |________ _________| |________ _________|
// | |
// V V
// old RDDs new RDDs
//
// Get the RDDs of the reduced values in "old time steps"
// 这边计算后就是 [100001,100004-2] => [100001,100002] 和我们之前预算的一致
val oldRDDs =
reducedStream.slice(previousWindow.beginTime, currentWindow.beginTime - parent.slideDuration)
logDebug("# old RDDs = " + oldRDDs.size)
// Get the RDDs of the reduced values in "new time steps"
//这边计算后就是 [100004+1,100006] 注意 parent.slideDuration = batchInterval
val newRDDs =
reducedStream.slice(previousWindow.endTime + parent.slideDuration, currentWindow.endTime)
logDebug("# new RDDs = " + newRDDs.size)
// Get the RDD of the reduced value of the previous window
// 获取上一个window 的RDD
val previousWindowRDD =
getOrCompute(previousWindow.endTime).getOrElse(ssc.sc.makeRDD(Seq[(K, V)]()))
// Make the list of RDDs that needs to cogrouped together for reducing their reduced values
val allRDDs = new ArrayBuffer[RDD[(K, V)]]() += previousWindowRDD ++= oldRDDs ++= newRDDs
// Cogroup the reduced RDDs and merge the reduced valuesRDD
// 将 三个RDD cogroupe也就是 合并
val cogroupedRDD = new CoGroupedRDD[K](allRDDs.toSeq.asInstanceOf[Seq[RDD[(K, _)]]],
partitioner)
// val mergeValuesFunc = mergeValues(oldRDDs.size, newRDDs.size) _
val numOldValues = oldRDDs.size
val numNewValues = newRDDs.size
val mergeValues = (arrayOfValues: Array[Iterable[V]]) => {
if (arrayOfValues.length != 1 + numOldValues + numNewValues) {
throw new Exception("Unexpected number of sequences of reduced values")
}
// Getting reduced values "old time steps" that will be removed from current window
// 拿到 需要减去oldrdd 的 值
val oldValues = (1 to numOldValues).map(i => arrayOfValues(i)).filter(!_.isEmpty).map(_.head)
// Getting reduced values "new time steps"
// 拿到 需要加上 oldrdd 的 值
val newValues =
(1 to numNewValues).map(i => arrayOfValues(numOldValues + i)).filter(!_.isEmpty).map(_.head)
// 判断上一个Window的Rdd值是不是
if (arrayOfValues(0).isEmpty) {
// If previous window's reduce value does not exist, then at least new values should exist
if (newValues.isEmpty) {
throw new Exception("Neither previous window has value for key, nor new values found. " +
"Are you sure your key class hashes consistently?")
}
// Reduce the new values
// 如果上一个window 是空的，直接只计算新值
newValues.reduce(reduceF) // return
} else {
// Get the previous window's reduced value
var tempValue = arrayOfValues(0).head
// If old values exists, then inverse reduce then from previous value
if (!oldValues.isEmpty) {
// 减去 oldRDDs，其实应该说对 上一个window 的值和 oldValues 处理
tempValue = invReduceF(tempValue, oldValues.reduce(reduceF))
}
// If new values exists, then reduce them with previous value
if (!newValues.isEmpty) {
// 加上 newRdd的值，其实应该说对 上一个window 的值和 newValues 处理
tempValue = reduceF(tempValue, newValues.reduce(reduceF))
}
tempValue // return
}
}
// 调用上面的函数 拿到当前的值
val mergedValuesRDD = cogroupedRDD.asInstanceOf[RDD[(K, Array[Iterable[V]])]]
.mapValues(mergeValues)
if (filterFunc.isDefined) {
Some(mergedValuesRDD.filter(filterFunc.get))
} else {
Some(mergedValuesRDD)
}
}
