Akka源码分析-故障恢复
Actor故障恢复是akka中非常重要的内容，在之前的博客中虽然有介绍，但都是杂糅在其他知识点的细节中，本博客将单独介绍这一部分的故障恢复。为了简化分析的单独，本文只研究用户的actor故障恢复的步骤，系统actor也基本差不多；另外，为了简化篇幅，不重要的源码将不再整段贴出来，感兴趣的读者可以去下载源码单独分析。
上面是官方描述actor树形结构时用到的一个图，非常重要，希望读者一定要记住这个树形的层次概念。也就是说，actor一定会归属到某个父actor下面，而actor的故障恢复就是通过这种层级来实现的。这个图之所以重要，是因为如果你能够深刻理解这个图，大概就知道如何实现故障恢复了。
有些爱钻研的读者可能会问，还有一个 / 这个actor没有父actor啊，这个actor如果失败，怎么恢复？其实吧，这算是一个好问题。这是akka设计的根基啊，如果akka本身动不动都需要故障恢复，别人还怎么用。你可以这样理解，akka设计的比较好，或者说root actor功能比较少，轻易不会出现故障。虽然这样解释有点勉强，哈哈。
还记得Actor的调度模型吗，actor只有在收到消息时，才会去调用receive方法，此时actor才会被某个线程执行，也就是“活着”的。如果某个actor没有消息，那么它仅仅是一堆数据躺在内存，也就是“死的”。actor只有在“活着”的时候才可能出现故障。所以我们要从actor处理消息入手，来看看如何实现故障恢复。
//Memory consistency is handled by the Mailbox (reading mailbox status then processing messages, then writing mailbox status
final def invoke(messageHandle: Envelope): Unit = {
val influenceReceiveTimeout = !messageHandle.message.isInstanceOf[NotInfluenceReceiveTimeout]
try {
currentMessage = messageHandle
if (influenceReceiveTimeout)
cancelReceiveTimeout()
messageHandle.message match {
case msg: AutoReceivedMessage ⇒ autoReceiveMessage(messageHandle)
case msg ⇒ receiveMessage(msg)
}
currentMessage = null // reset current message after successful invocation
} catch handleNonFatalOrInterruptedException { e ⇒
handleInvokeFailure(Nil, e)
} finally {
if (influenceReceiveTimeout)
checkReceiveTimeout // Reschedule receive timeout
}
}
上面是ActorCell里面处理用户消息的相关函数，这包括在一个try-catch里面，我们假设receiveMessage处理消息出现了异常。
final protected def handleNonFatalOrInterruptedException(thunk: (Throwable) ⇒ Unit): Catcher[Unit] = {
case e: InterruptedException ⇒
thunk(e)
Thread.currentThread().interrupt()
case NonFatal(e) ⇒
thunk(e)
}
/**
* Returns true if the provided `Throwable` is to be considered non-fatal, or false if it is to be considered fatal
*/
def apply(t: Throwable): Boolean = t match {
// VirtualMachineError includes OutOfMemoryError and other fatal errors
case _: VirtualMachineError | _: ThreadDeath | _: InterruptedException | _: LinkageError | _: ControlThrowable => false
case _ => true
}
出现异常则会执行catch代码块，其实就是执行了handleNonFatalOrInterruptedException函数，这个函数会检查是不是InterruptedException或NonFatal类型的异常，如果是，则会执行thunk也就是handleInvokeFailure。简单来说就是如果receive处理消息出现异常，则会执行handleInvokeFailure。
final def handleInvokeFailure(childrenNotToSuspend: immutable.Iterable[ActorRef], t: Throwable): Unit = {
// prevent any further messages to be processed until the actor has been restarted
if (!isFailed) try {
suspendNonRecursive()
// suspend children
val skip: Set[ActorRef] = currentMessage match {
case Envelope(Failed(_, _, _), child) ⇒ { setFailed(child); Set(child) }
case _ ⇒ { setFailed(self); Set.empty }
}
suspendChildren(exceptFor = skip ++ childrenNotToSuspend)
t match {
// tell supervisor
case _: InterruptedException ⇒
// ➡➡➡ NEVER SEND THE SAME SYSTEM MESSAGE OBJECT TO TWO ACTORS ⬅⬅⬅
parent.sendSystemMessage(Failed(self, new ActorInterruptedException(t), uid))
case _ ⇒
// ➡➡➡ NEVER SEND THE SAME SYSTEM MESSAGE OBJECT TO TWO ACTORS ⬅⬅⬅
parent.sendSystemMessage(Failed(self, t, uid))
}
} catch handleNonFatalOrInterruptedException { e ⇒
publish(Error(e, self.path.toString, clazz(actor),
"emergency stop: exception in failure handling for " + t.getClass + Logging.stackTraceFor(t)))
try children foreach stop
finally finishTerminate()
}
}
handleInvokeFailure会执行一个try语句，该语句第一句话就是执行suspendNonRecursive，从名称来看，就是要非递归的方式挂起当前actor，就是挂起mailbox对消息的处理，其实就是设置mailbox的状态。
private def suspendNonRecursive(): Unit = dispatcher suspend this
/**
* After the call to this method, the dispatcher mustn't begin any new message processing for the specified reference
*/
protected[akka] def suspend(actor: ActorCell): Unit = {
val mbox = actor.mailbox
if ((mbox.actor eq actor) && (mbox.dispatcher eq this))
mbox.suspend()
}
/**
* Increment the suspend count by one. Caller does not need to worry about whether
* status was Scheduled or not.
*
* @return true if the previous suspend count was zero
*/
@tailrec
final def suspend(): Boolean = currentStatus match {
case Closed ⇒
setStatus(Closed); false
case s ⇒
if (updateStatus(s, s + suspendUnit)) s < suspendUnit
else suspend()
}
挂起当前出现异常的actor的mailbox对消息的处理之后，又通过suspendChildren挂起了子actor对消息的处理（流程跟上面差不多，都是设置子actor的mailbox状态），最后给父actor发送了Failed消息就结束了。
总结一下当前处理逻辑，如果某个actor对消息的处理出现异常，就挂起当前actor的邮箱，然后递归地挂起所有子actor的邮箱，发送Failed消息给父actor。怎么样简单吧，就是这么简单。现在要看父actor如何处理这个消息了。
final protected def handleFailure(f: Failed): Unit = {
currentMessage = Envelope(f, f.child, system)
getChildByRef(f.child) match {
/*
* only act upon the failure, if it comes from a currently known child;
* the UID protects against reception of a Failed from a child which was
* killed in preRestart and re-created in postRestart
*/
case Some(stats) if stats.uid == f.uid ⇒
if (!actor.supervisorStrategy.handleFailure(this, f.child, f.cause, stats, getAllChildStats)) throw f.cause
case Some(stats) ⇒
publish(Debug(self.path.toString, clazz(actor),
"dropping Failed(" + f.cause + ") from old child " + f.child + " (uid=" + stats.uid + " != " + f.uid + ")"))
case None ⇒
publish(Debug(self.path.toString, clazz(actor), "dropping Failed(" + f.cause + ") from unknown child " + f.child))
}
}
父actor异步收到Failed消息后会调用handleFailure方法。这个方法有两个主要逻辑，设置当前的currentMessage值，调用当前actor也就是抛异常actor的父actor的supervisorStrategy对象的handleFailure方法，如果handleFailure不处理当前异常，则通过throw继续抛出异常，这又进入了父actor的handleInvokeFailure处理逻辑。
/**
* This is the main entry point: in case of a child’s failure, this method
* must try to handle the failure by resuming, restarting or stopping the
* child (and returning `true`), or it returns `false` to escalate the
* failure, which will lead to this actor re-throwing the exception which
* caused the failure. The exception will not be wrapped.
*
* This method calls [[akka.actor.SupervisorStrategy#logFailure]], which will
* log the failure unless it is escalated. You can customize the logging by
* setting [[akka.actor.SupervisorStrategy#loggingEnabled]] to `false` and
* do the logging inside the `decider` or override the `logFailure` method.
*
* @param children is a lazy collection (a view)
*/
def handleFailure(context: ActorContext, child: ActorRef, cause: Throwable, stats: ChildRestartStats, children: Iterable[ChildRestartStats]): Boolean = {
val directive = decider.applyOrElse(cause, escalateDefault)
directive match {
case Resume ⇒
logFailure(context, child, cause, directive)
resumeChild(child, cause)
true
case Restart ⇒
logFailure(context, child, cause, directive)
processFailure(context, true, child, cause, stats, children)
true
case Stop ⇒
logFailure(context, child, cause, directive)
processFailure(context, false, child, cause, stats, children)
true
case Escalate ⇒
logFailure(context, child, cause, directive)
false
}
}
handleFailure会根据当前actor也就是父actor的监督策略，判断当前的指令，然后调用processFailure方法。请注意processFailure方法的第一个参数是当前父actor的context值，第三个参数是出现异常的子actor的ActorRef。
/** * Applies the fault handling `Directive` (Resume, Restart, Stop) specified in the `Decider` * to the child actor that failed, as opposed to [[akka.actor.AllForOneStrategy]] that applies * it to all children. * * @param maxNrOfRetries the number of times a child actor is allowed to be restarted, negative value means no limit, * if the limit is exceeded the child actor is stopped * @param withinTimeRange duration of the time window for maxNrOfRetries, Duration.Inf means no window * @param decider mapping from Throwable to [[akka.actor.SupervisorStrategy.Directive]], you can also use a * [[scala.collection.immutable.Seq]] of Throwables which maps the given Throwables to restarts, otherwise escalates. * @param loggingEnabled the strategy logs the failure if this is enabled (true), by default it is enabled */ case class OneForOneStrategy( maxNrOfRetries: Int = -1, withinTimeRange: Duration = Duration.Inf, override val loggingEnabled: Boolean = true)(val decider: SupervisorStrategy.Decider) extends SupervisorStrategy
默认情况下，每个actor的监督策略是OneForOneStrategy。也就是遇到异常会执行Restart分支的代码。
def processFailure(context: ActorContext, restart: Boolean, child: ActorRef, cause: Throwable, stats: ChildRestartStats, children: Iterable[ChildRestartStats]): Unit = {
if (restart && stats.requestRestartPermission(retriesWindow))
restartChild(child, cause, suspendFirst = false)
else
context.stop(child) //TODO optimization to drop child here already?
}
其实就是会执行restartChild方法，当然我们还可以设置重试次数以防止无限重启。（默认是没有次数限制的）
/**
* Restart the given child, possibly suspending it first.
*
* <b>IMPORTANT:</b>
*
* If the child is the currently failing one, it will already have been
* suspended, hence `suspendFirst` must be false. If the child is not the
* currently failing one, then it did not request this treatment and is
* therefore not prepared to be resumed without prior suspend.
*/
final def restartChild(child: ActorRef, cause: Throwable, suspendFirst: Boolean): Unit = {
val c = child.asInstanceOf[InternalActorRef]
if (suspendFirst) c.suspend()
c.restart(cause)
}
restartChild比较简单，就是执行了子actor的restart函数。那么restart函数在哪里实现呢？还记得这个ActorRef的实际类型是什么吗？没错就是RepointableActorRef
def restart(cause: Throwable): Unit = underlying.restart(cause)
上面是RepointableActorRef的restart定义，就是调用了ActorCell的restart方法。
final def restart(cause: Throwable): Unit = try dispatcher.systemDispatch(this, Recreate(cause)) catch handleException
上面是ActorCell的restart方法的实现（在akka.actor.dungeon.Dispatch里面），就是又给自己发送了一个Recreate方法。读者可能会有疑问，之前不都把邮箱给suspend了么，怎么还收消息，但请注意这里发送的是系统消息。所以系统消息还是可以被路由的。
/**
* Do re-create the actor in response to a failure.
*/
protected def faultRecreate(cause: Throwable): Unit =
if (actor == null) {
system.eventStream.publish(Error(self.path.toString, clazz(actor),
"changing Recreate into Create after " + cause))
faultCreate()
} else if (isNormal) {
val failedActor = actor
if (system.settings.DebugLifecycle) publish(Debug(self.path.toString, clazz(failedActor), "restarting"))
if (failedActor ne null) {
val optionalMessage = if (currentMessage ne null) Some(currentMessage.message) else None
try {
// if the actor fails in preRestart, we can do nothing but log it: it’s best-effort
if (failedActor.context ne null) failedActor.aroundPreRestart(cause, optionalMessage)
} catch handleNonFatalOrInterruptedException { e ⇒
val ex = PreRestartException(self, e, cause, optionalMessage)
publish(Error(ex, self.path.toString, clazz(failedActor), e.getMessage))
} finally {
clearActorFields(failedActor, recreate = true)
}
}
assert(mailbox.isSuspended, "mailbox must be suspended during restart, status=" + mailbox.currentStatus)
if (!setChildrenTerminationReason(ChildrenContainer.Recreation(cause))) finishRecreate(cause, failedActor)
} else {
// need to keep that suspend counter balanced
faultResume(causedByFailure = null)
}
上面是收到Recreate时调用的函数，简单来说就是调用aroundPreRestart函数，和clearActorFields、finishRecreate。从此也可以看出aroundPreRestart、preRestart是actor实例出现异常最后执行的函数，此时actor的状态还是出现异常时候的值。
final protected def clearActorFields(actorInstance: Actor, recreate: Boolean): Unit = {
setActorFields(actorInstance, context = null, self = if (recreate) self else system.deadLetters)
currentMessage = null
behaviorStack = emptyBehaviorStack
}
clearActorFields设置了context/self/currentMessage/behaviorStack几个字段为默认值或者说空值。setChildrenTerminationReason这个函数不再分析，简单来说就是递归设置子actor的失败原因，然后返回false。这实现逻辑其实也挺扯的，不返回true，返回false，简直反人类。
其实吧，setChildrenTerminationReason一般都会成功。所以会执行finishRecreate方法。
private def finishRecreate(cause: Throwable, failedActor: Actor): Unit = {
// need to keep a snapshot of the surviving children before the new actor instance creates new ones
val survivors = children
try {
try resumeNonRecursive()
finally clearFailed() // must happen in any case, so that failure is propagated
val freshActor = newActor()
actor = freshActor // this must happen before postRestart has a chance to fail
if (freshActor eq failedActor) setActorFields(freshActor, this, self) // If the creator returns the same instance, we need to restore our nulled out fields.
freshActor.aroundPostRestart(cause)
if (system.settings.DebugLifecycle) publish(Debug(self.path.toString, clazz(freshActor), "restarted"))
// only after parent is up and running again do restart the children which were not stopped
survivors foreach (child ⇒
try child.asInstanceOf[InternalActorRef].restart(cause)
catch handleNonFatalOrInterruptedException { e ⇒
publish(Error(e, self.path.toString, clazz(freshActor), "restarting " + child))
})
} catch handleNonFatalOrInterruptedException { e ⇒
clearActorFields(actor, recreate = false) // in order to prevent preRestart() from happening again
handleInvokeFailure(survivors, PostRestartException(self, e, cause))
}
}
首先调用resumeNonRecursive，这个好像是恢复mailbox的状态的，此时就可以正常的接收消息了。重点来了：newActor。这里通过Props创建了actor的一个新的实例，一般情况下创建的actor实例与原来的都不同，不过读者是不是觉得，这里可以优化一下，弄个参数让开发者自己定义是不是重用以前的actor实例？嗯，没错，恭喜你，这就是SupervisorStrategy.Resume的作用，其实就是简单的忽略此次异常，actor实例不会有任何变化。
重建之后调用了新actor实例的aroundPostRestart、postRestart，也就是说postRestart是新实例创建成功后执行的第一个方法。新的actor实例创建成功后，后面的survivors foreach代码块就是在递归的重启子actor了。这里不再具体分析。
再总结一下，父actor收到子actor异常的消息后，根据当前的策略判断是恢复、重启、还是重用什么的，一般都是重启。如果是重启就是调用子actor的restart函数，就会给出现异常的子actor，发送Recreate系统消息，子actor收到该消息后，会执行一系列重启的操作，最后创建新的actor实例，重启成功。当然这个过程中，子actor的所有子actor也会被重启，也就是一个递归的处理过程。
其实简单的说，如果某个子actor出现异常，挂起自己的邮箱和递归子actor的邮箱，发消息通知父actor，父actor根据当前策略判断能够重启，如果能够重启，则该子actor完成重启动作，并负责递归重启它的子actor。当然还有其他策略，不同的策略这个处理逻辑会有不同。
关于Actor的生命周期，上面的图画的比较清楚，读者可以自行理解。但需要注意，Instance 1和Instance 2是两个“不同”的Actor实例，这里的不同，是指Actor内部的用户状态（即开发者自定义的字段）全部丢失，都会被重建。但两个不同实例的UID相同，这就意味着用相同的ActorRef可以跟后面重新建立的Instance 2发消息。
