Akka源码分析-Akka-Streams-Materializer（1）
本博客逐步分析Akka Streams的源码，当然必须循序渐进，且估计会分很多篇，毕竟Akka Streams还是比较复杂的。
implicit val system = ActorSystem("QuickStart")
implicit val materializer = ActorMaterializer()
在使用Streams相关的API时，上面两个对象是必须创建的。ActorSystem不再说了，我们来看ActorMaterializer。
/** * An ActorMaterializer takes a stream blueprint and turns it into a running stream. */ abstract class ActorMaterializer extends Materializer with MaterializerLoggingProvider
ActorMaterializer把一个流式计算的blueprint（大纲、蓝本？）转换成一个运行的流，简单来说这就是用来编译akka提供的流式API的。它继承一个类和一个特质。MaterializerLoggingProvider就不看了，就是提供日志相关的功能的。
/**
* Materializer SPI (Service Provider Interface)
*
* Binary compatibility is NOT guaranteed on materializer internals.
*
* Custom materializer implementations should be aware that the materializer SPI
* is not yet final and may change in patch releases of Akka. Please note that this
* does not impact end-users of Akka streams, only implementors of custom materializers,
* with whom the Akka team co-ordinates such changes.
*
* Once the SPI is final this notice will be removed.
*/
abstract class Materializer {
/**
* The `namePrefix` shall be used for deriving the names of processing
* entities that are created during materialization. This is meant to aid
* logging and failure reporting both during materialization and while the
* stream is running.
*/
def withNamePrefix(name: String): Materializer
/**
* This method interprets the given Flow description and creates the running
* stream. The result can be highly implementation specific, ranging from
* local actor chains to remote-deployed processing networks.
*/
def materialize[Mat](runnable: Graph[ClosedShape, Mat]): Mat
/**
* This method interprets the given Flow description and creates the running
* stream using an explicitly provided [[Attributes]] as top level (least specific) attributes that
* will be defaults for the materialized stream.
* The result can be highly implementation specific, ranging from local actor chains to remote-deployed
* processing networks.
*/
def materialize[Mat](
runnable: Graph[ClosedShape, Mat],
@deprecatedName('initialAttributes) defaultAttributes: Attributes): Mat
/**
* Running a flow graph will require execution resources, as will computations
* within Sources, Sinks, etc. This [[scala.concurrent.ExecutionContextExecutor]]
* can be used by parts of the flow to submit processing jobs for execution,
* run Future callbacks, etc.
*
* Note that this is not necessarily the same execution context the stream operator itself is running on.
*/
implicit def executionContext: ExecutionContextExecutor
/**
* Interface for operators that need timer services for their functionality. Schedules a
* single task with the given delay.
*
* @return A [[akka.actor.Cancellable]] that allows cancelling the timer. Cancelling is best effort, if the event
* has been already enqueued it will not have an effect.
*/
def scheduleOnce(delay: FiniteDuration, task: Runnable): Cancellable
/**
* Interface for operators that need timer services for their functionality. Schedules a
* repeated task with the given interval between invocations.
*
* @return A [[akka.actor.Cancellable]] that allows cancelling the timer. Cancelling is best effort, if the event
* has been already enqueued it will not have an effect.
*/
def schedulePeriodically(initialDelay: FiniteDuration, interval: FiniteDuration, task: Runnable): Cancellable
}
Materializer非常重要，所以这里贴出完整代码。Materializer是一个SPI（服务提供接口），Materializer内部不保证二进制兼容，也就是说版本可能不兼容。但这个定义比较奇怪，既然所有的方法都没有实现，那是不是用trait比较合适呢？为啥是一个抽象类呢？
Materializer一共6个方法，我们一个个看。
withNamePrefix在派生处理实体的时候用到，简单来说就是流中的每个计算实体（Source/Sink/Flow/RunnableGraph）等映射成一个actor的时候，这些actor名称需要一个前缀，withNamePrefix就是用来设置这个前缀的。
materialize方法有两个实现。这个方法用来解释给定的Flow定义，并创建运行的流的。创建的结果会高度依赖具体的实现，可能是本地的actor链也可能是远程部署的加工过程。
executionContext用来提供异步执行环境。
scheduleOnce/schedulePeriodically用来提供定时调度的功能，毕竟还是有一些操作需要时间服务的。
Materializer貌似很简单，就只有上面几个接口，但其核心的接口就是materialize方法，毕竟这是用来编译akka流的。如果你使用过storm的话，就一定知道Storm有一个拓扑编译器。其功能跟这个差不多。
我们再来看ActorMaterializer。
ActorMaterializer还提供了三个比较重要的接口：actorOf/system/supervisor。其中actorOf接收MaterializationContext、Props作为参数，创建一个Actor;system返回与ActorMaterializer关联的ActorSystem；supervisor功能后面再研究。MaterializationContext这个参数还是比较有意思的，可以理解成物理化（编译流）时的上下文。
/** * Context parameter to the `create` methods of sources and sinks. * * INTERNAL API */ @InternalApi private[akka] case class MaterializationContext( materializer: Materializer, effectiveAttributes: Attributes, islandName: String)
MaterializationContext一共三个变量，materializer不再说了，就是当前上下文关联的Materializer。effectiveAttributes就是用来提供参数的，只不过又封装成了Attributes。islandName比较有意思，单纯从命名上来翻译，它是“岛名称”。那什么是“岛”（island）呢？后面分析时，也会遇到这个“island”，大家这里稍微留意一下就行了。
另外在Materializer这个抽象类中materialize方法有一个参数需要我们研究下：runnable: Graph[ClosedShape, Mat]。
/** * Not intended to be directly extended by user classes * * @see [[akka.stream.stage.GraphStage]] */ trait Graph[+S <: Shape, +M]
Graph是一个trait，它有两个类型参数：S/M。能不能起个好点的名字，SM，哈哈。其中S是Shape的子类，从名称来看，好像是graph的形状。简单分析一下Graph的主体代码，就会发现它有两个方法特别重要：shape、traversalBuilder。其中shape返回图的形状，虽然我们还不知道形状是什么，但ClosedShape是它的一个形式；traversalBuilder
/** * INTERNAL API. * * Every materializable element must be backed by a stream layout module */ private[stream] def traversalBuilder: TraversalBuilder
traversalBuilder返回一个TraversalBuilder，注释说每一个可物理化的元素都必须被流布局模块支持。TraversalBuilder从名称来看，是一个可遍历的编译器。估计就是一个拓扑排序。
下面我们来看Shape究竟是什么。
/** * A Shape describes the inlets and outlets of a [[Graph]]. In keeping with the * philosophy that a Graph is a freely reusable blueprint, everything that * matters from the outside are the connections that can be made with it, * otherwise it is just a black box. */ abstract class Shape
一个Shape描述了Graph的入口和出口，按照Graph是自由重用的蓝本这样的哲学，从外部来看Graph是可被链接的就非常重要了，否则它就只是一个黑盒子。我们姑且认为它只是用来定义Graph的输入输出的吧。而且这个trait最重要的两个方法也就是inlets/outlets。
/** * Scala API: get a list of all input ports */ def inlets: immutable.Seq[Inlet[_]] /** * Scala API: get a list of all output ports */ def outlets: immutable.Seq[Outlet[_]]
那Inlet和Outlet又是什么呢？
final class Inlet[T] private (val s: String) extends InPort {
def carbonCopy(): Inlet[T] = {
val in = Inlet[T](s)
in.mappedTo = this
in
}
/**
* INTERNAL API.
*/
def as[U]: Inlet[U] = this.asInstanceOf[Inlet[U]]
override def toString: String = s + "(" + this.hashCode + s")" +
(if (mappedTo eq this) ""
else s" mapped to $mappedTo")
}
/**
* An input port of a StreamLayout.Module. This type logically belongs
* into the impl package but must live here due to how `sealed` works.
* It is also used in the Java DSL for “untyped Inlets” as a work-around
* for otherwise unreasonable existential types.
*/
sealed abstract class InPort { self: Inlet[_] ⇒
final override def hashCode: Int = super.hashCode
final override def equals(that: Any): Boolean = this eq that.asInstanceOf[AnyRef]
/**
* INTERNAL API
*/
@volatile private[stream] var id: Int = -1
/**
* INTERNAL API
*/
@volatile private[stream] var mappedTo: InPort = this
/**
* INTERNAL API
*/
private[stream] def inlet: Inlet[_] = this
}
Inlet好像也没有提供什么方法啊，貌似目前来看比较重要的就只有一个id字段，其他的都是变量返回、复制的。
好像分析到这里，Materializer、Shape、Graph都还比较抽象，还得来看Materializer的具体实现，毕竟它只是一个trait。
def apply(materializerSettings: ActorMaterializerSettings, namePrefix: String)(implicit context: ActorRefFactory): ActorMaterializer = {
val haveShutDown = new AtomicBoolean(false)
val system = actorSystemOf(context)
new PhasedFusingActorMaterializer(
system,
materializerSettings,
system.dispatchers,
actorOfStreamSupervisor(materializerSettings, context, haveShutDown),
haveShutDown,
FlowNames(system).name.copy(namePrefix))
}
通过分析ActorMaterializer的apply方法，我们发现最终调用了上面这个版本的apply，可以看到，最终创建了PhasedFusingActorMaterializer，而且是new出来的，所以这个类一定是一个具体类，也就是说所有的方法和字段都有对应的实现和定义。PhasedFusingActorMaterializer的名称其实还挺有意思的，直译的话就是分段熔断actor物化器。分段好理解，熔断就不知道怎么理解了，或者翻译错了？哈哈，我也不知道。
@InternalApi private[akka] case class PhasedFusingActorMaterializer( system: ActorSystem, override val settings: ActorMaterializerSettings, dispatchers: Dispatchers, supervisor: ActorRef, haveShutDown: AtomicBoolean, flowNames: SeqActorName) extends ExtendedActorMaterializer
PhasedFusingActorMaterializer居然是一个case class，那还new干啥。它继承了ExtendedActorMaterializer。ExtendedActorMaterializer源码不再贴出来，它就是重新覆盖了ActorMaterializer的几个方法，并实现了一个方法actorOf。我们就来看这个actorOf
@InternalApi private[akka] override def actorOf(context: MaterializationContext, props: Props): ActorRef = {
val effectiveProps = props.dispatcher match {
case Dispatchers.DefaultDispatcherId ⇒
props.withDispatcher(context.effectiveAttributes.mandatoryAttribute[ActorAttributes.Dispatcher].dispatcher)
case ActorAttributes.IODispatcher.dispatcher ⇒
// this one is actually not a dispatcher but a relative config key pointing containing the actual dispatcher name
props.withDispatcher(settings.blockingIoDispatcher)
case _ ⇒ props
}
actorOf(effectiveProps, context.islandName)
}
其实它就是用特定的dispatcher替换了Props中的值，然后调用另一个actorOf创建actor。
@InternalApi private[akka] def actorOf(props: Props, name: String): ActorRef = {
supervisor match {
case ref: LocalActorRef ⇒
ref.underlying.attachChild(props, name, systemService = false)
case ref: RepointableActorRef ⇒
if (ref.isStarted)
ref.underlying.asInstanceOf[ActorCell].attachChild(props, name, systemService = false)
else {
implicit val timeout = ref.system.settings.CreationTimeout
val f = (supervisor ? StreamSupervisor.Materialize(props, name)).mapTo[ActorRef]
Await.result(f, timeout.duration)
}
case unknown ⇒
throw new IllegalStateException(s"Stream supervisor must be a local actor, was [${unknown.getClass.getName}]")
}
}
这个actorOf我们也不深入分析了，反正就是在创建actor。下面我们来看materialize在PhasedFusingActorMaterializer中的实现。
override def materialize[Mat](
graph: Graph[ClosedShape, Mat],
defaultAttributes: Attributes,
defaultPhase: Phase[Any],
phases: Map[IslandTag, Phase[Any]]): Mat = {
if (isShutdown) throw new IllegalStateException("Trying to materialize stream after materializer has been shutdown")
val islandTracking = new IslandTracking(phases, settings, defaultAttributes, defaultPhase, this, islandNamePrefix = createFlowName() + "-")
var current: Traversal = graph.traversalBuilder.traversal
val attributesStack = new java.util.ArrayDeque[Attributes](8)
attributesStack.addLast(defaultAttributes and graph.traversalBuilder.attributes)
val traversalStack = new java.util.ArrayDeque[Traversal](16)
traversalStack.addLast(current)
val matValueStack = new java.util.ArrayDeque[Any](8)
if (Debug) {
println(s"--- Materializing layout:")
TraversalBuilder.printTraversal(current)
println(s"--- Start materialization")
}
// Due to how Concat works, we need a stack. This probably can be optimized for the most common cases.
while (!traversalStack.isEmpty) {
current = traversalStack.removeLast()
while (current ne EmptyTraversal) {
var nextStep: Traversal = EmptyTraversal
current match {
case MaterializeAtomic(mod, outToSlot) ⇒
if (Debug) println(s"materializing module: $mod")
val matAndStage = islandTracking.getCurrentPhase.materializeAtomic(mod, attributesStack.getLast)
val logic = matAndStage._1
val matValue = matAndStage._2
if (Debug) println(s" materialized value is $matValue")
matValueStack.addLast(matValue)
val stageGlobalOffset = islandTracking.getCurrentOffset
wireInlets(islandTracking, mod, logic)
wireOutlets(islandTracking, mod, logic, stageGlobalOffset, outToSlot)
if (Debug) println(s"PUSH: $matValue => $matValueStack")
case Concat(first, next) ⇒
if (next ne EmptyTraversal) traversalStack.add(next)
nextStep = first
case Pop ⇒
val popped = matValueStack.removeLast()
if (Debug) println(s"POP: $popped => $matValueStack")
case PushNotUsed ⇒
matValueStack.addLast(NotUsed)
if (Debug) println(s"PUSH: NotUsed => $matValueStack")
case transform: Transform ⇒
val prev = matValueStack.removeLast()
val result = transform(prev)
matValueStack.addLast(result)
if (Debug) println(s"TRFM: $matValueStack")
case compose: Compose ⇒
val second = matValueStack.removeLast()
val first = matValueStack.removeLast()
val result = compose(first, second)
matValueStack.addLast(result)
if (Debug) println(s"COMP: $matValueStack")
case PushAttributes(attr) ⇒
attributesStack.addLast(attributesStack.getLast and attr)
if (Debug) println(s"ATTR PUSH: $attr")
case PopAttributes ⇒
attributesStack.removeLast()
if (Debug) println(s"ATTR POP")
case EnterIsland(tag) ⇒
islandTracking.enterIsland(tag, attributesStack.getLast)
case ExitIsland ⇒
islandTracking.exitIsland()
case _ ⇒
}
current = nextStep
}
}
def shutdownWhileMaterializingFailure =
new IllegalStateException("Materializer shutdown while materializing stream")
try {
islandTracking.getCurrentPhase.onIslandReady()
islandTracking.allNestedIslandsReady()
if (Debug) println("--- Finished materialization")
matValueStack.peekLast().asInstanceOf[Mat]
} finally {
if (isShutdown) throw shutdownWhileMaterializingFailure
}
}
由于这个方法过于重要，所以我还是贴出了整段代码。我们发现它有两个参数没有见过：defaultPhase: Phase[Any]、phases: Map[IslandTag, Phase[Any]]。这涉及到两个类型IslandTag、Phase。Phase比较好理解就是阶段，那IslandTag是啥呢？“岛”的标签？“岛”是什么？！
@DoNotInherit private[akka] trait Phase[M] {
def apply(
settings: ActorMaterializerSettings,
effectiveAttributes: Attributes,
materializer: PhasedFusingActorMaterializer,
islandName: String): PhaseIsland[M]
}
Phase这个trait就只有一个apply，他就是根据各个参数，返回了PhaseIsland，阶段岛？
@DoNotInherit private[akka] trait PhaseIsland[M] {
def name: String
def materializeAtomic(mod: AtomicModule[Shape, Any], attributes: Attributes): (M, Any)
def assignPort(in: InPort, slot: Int, logic: M): Unit
def assignPort(out: OutPort, slot: Int, logic: M): Unit
def createPublisher(out: OutPort, logic: M): Publisher[Any]
def takePublisher(slot: Int, publisher: Publisher[Any]): Unit
def onIslandReady(): Unit
}
PhaseIsland有两个方法非常重要：createPublisher、takePublisher。为啥重要？因为它好像在创建Publisher啊，Publisher是啥？翻翻我上一篇博客喽？这是reactivestreams里面的接口，之所以重要因为它涉及到了底层。
还有一个IslandTag，这又是啥呢？
@DoNotInherit private[akka] trait IslandTag
啥元素也没有，难道仅仅是用来做类型匹配的？鬼知道。估计是给Island打标签的。
materialize方法代码很多，逻辑很复杂，但有几个重要的点需要注意。它创建了一个IslandTracking，并通过一个while循环访问了graph.traversalBuilder，对IslandTracking进行了某些操作，最后调用了islandTracking的两个方法。
islandTracking.getCurrentPhase.onIslandReady() islandTracking.allNestedIslandsReady()
所以IslandTracking和这两个方法就非常重要了，因为这个类的这两个方法调用完之后，流就可以正常运行了啊。
@InternalApi private[akka] class IslandTracking( val phases: Map[IslandTag, Phase[Any]], val settings: ActorMaterializerSettings, attributes: Attributes, defaultPhase: Phase[Any], val materializer: PhasedFusingActorMaterializer, islandNamePrefix: String)
IslandTracking岛跟踪，居然一行注释都没有，这让我怎么分析啊，麻蛋！
由于时间关系，今天就先分析到这里吧，很显然Akka Streams的代码很复杂，看源码非常有难度，希望我还能看下去。哈哈。
