Flink源码阅读（一）--Checkpoint触发机制

Checkpoint触发机制

　　Flink的checkpoint是通过定时器周期性触发的。checkpoint触发最关键的类是CheckpointCoordinator，称它为检查点协调器。

org.apache.flink.runtime.checkpoint.CheckpointCoordinator

　　CheckpointCoordinator主要作用是协调operators和state的分布式快照。它通过向相关的tasks发送触发消息和从各tasks收集确认消息（Ack）来完成checkpoint。同时，它还收集和维护各个tasks上报的状态句柄/状态引用（state handles）。

　　CheckpointCoordinator主要属性：

    /** Coordinator-wide lock to safeguard the checkpoint updates */
    private final Object lock = new Object();  //Coordinator范围的锁

    /** Lock specially to make sure that trigger requests do not overtake each other.
     * This is not done with the coordinator-wide lock, because as part of triggering,
     * blocking operations may happen (distributed atomic counters).
     * Using a dedicated lock, we avoid blocking the processing of 'acknowledge/decline'
     * messages during that phase. */
    private final Object triggerLock = new Object(); //trigger requests的专用锁，避免在获取checkpointID时阻塞对消息的处理。

    /** Tasks who need to be sent a message when a checkpoint is started */
    private final ExecutionVertex[] tasksToTrigger;

    /** Tasks who need to acknowledge a checkpoint before it succeeds */
    private final ExecutionVertex[] tasksToWaitFor;

    /** Tasks who need to be sent a message when a checkpoint is confirmed */
    private final ExecutionVertex[] tasksToCommitTo;

    /** Map from checkpoint ID to the pending checkpoint */
    private final Map<Long, PendingCheckpoint> pendingCheckpoints;//待处理的checkpoint

    /** Actor that receives status updates from the execution graph this coordinator works for */
    private JobStatusListener jobStatusListener;//Actor实例，监听Job状态变化并根据变化启停定时任务

    /** Flag whether a trigger request could not be handled immediately. Non-volatile, because only
     * accessed in synchronized scope */
    private boolean triggerRequestQueued;//标记一个触发请求是否不能被立即处理。

    /** Flag marking the coordinator as shut down (not accepting any messages any more) */
    private volatile boolean shutdown;//coordinator的关闭标志

 

　　ScheduledTrigger

　　ScheduledTrigger是检查点定时任务类，主要是调用了triggerCheckpoint方法。

    private final class ScheduledTrigger implements Runnable {
        @Override
        public void run() {
            try {
                triggerCheckpoint(System.currentTimeMillis(), true);
            }
            catch (Exception e) {
                LOG.error("Exception while triggering checkpoint.", e);
            }
        }
    }

　　下面具体看一下 triggerCheckpoint 方法的具体实现

    //触发一个新的标准检查点。timestamp为触发检查点的时间戳，isPeriodic标志是否是周期性的触发
    public boolean triggerCheckpoint(long timestamp, boolean isPeriodic) {
        return triggerCheckpoint(timestamp, checkpointProperties, checkpointDirectory, isPeriodic).isSuccess();
    }

　　触发检查点的核心逻辑：

　　　　首先进行触发Checkpoint之前的预检查，判断是否满足条件；

　　　　然后获取一个CheckpointID，创建PendingCheckpoint实例；

　　　　之后重新检查触发条件是否满足要求，防止产生竞态条件；

　　　　最后将PendingCheckpoint实例checkpoint加入到pendingCheckpoints中，并向tasks发送消息触发它们的检查点。

    CheckpointTriggerResult triggerCheckpoint(
            long timestamp,
            CheckpointProperties props,
            String targetDirectory,
            boolean isPeriodic) {

        // Sanity check 如果检查点是存储在外部系统中且targetDirectory为空，报错
        if (props.externalizeCheckpoint() && targetDirectory == null) {
            throw new IllegalStateException("No target directory specified to persist checkpoint to.");
        }

        // make some eager pre-checks 一些checkpoint之前的预检查
        synchronized (lock) {
            // abort if the coordinator has been shutdown in the meantime
            if (shutdown) {
                return new CheckpointTriggerResult(CheckpointDeclineReason.COORDINATOR_SHUTDOWN);
            }

            // Don't allow periodic checkpoint if scheduling has been disabled
            if (isPeriodic && !periodicScheduling) {
                return new CheckpointTriggerResult(CheckpointDeclineReason.PERIODIC_SCHEDULER_SHUTDOWN);
            }

            // validate whether the checkpoint can be triggered, with respect to the limit of
            // concurrent checkpoints, and the minimum time between checkpoints.
            // these checks are not relevant for savepoints
            // 验证checkpoint是否能被触发，关于并发检查点的限制和检查点之间的最小时间。
            // 判断checkpoint是否被强制。强制checkpoint不受并发检查点最大数量和检查点之间最小时间的限制。
            if (!props.forceCheckpoint()) {
                // sanity check: there should never be more than one trigger request queued
                if (triggerRequestQueued) {
                    //如果不能被立即触发，直接返回异常
                    LOG.warn("Trying to trigger another checkpoint while one was queued already");
                    return new CheckpointTriggerResult(CheckpointDeclineReason.ALREADY_QUEUED);
                }

                // if too many checkpoints are currently in progress, we need to mark that a request is queued
                if (pendingCheckpoints.size() >= maxConcurrentCheckpointAttempts) {
                    //如果未完成的检查点太多，大于配置的并发检查点最大数量，则将当前检查点的触发请求设置为不能立即执行。
                    triggerRequestQueued = true;
                    //如果定时任务已经启动，则取消定时任务的执行。
                    if (currentPeriodicTrigger != null) {
                        currentPeriodicTrigger.cancel(false);
                        currentPeriodicTrigger = null;
                    }
                    return new CheckpointTriggerResult(CheckpointDeclineReason.TOO_MANY_CONCURRENT_CHECKPOINTS);
                }

                // make sure the minimum interval between checkpoints has passed
                //检查是否满足checkpoint之间的最小时间间隔的条件
                final long earliestNext = lastCheckpointCompletionNanos + minPauseBetweenCheckpointsNanos;
                final long durationTillNextMillis = (earliestNext - System.nanoTime()) / 1_000_000;

                if (durationTillNextMillis > 0) {
                    if (currentPeriodicTrigger != null) {
                        currentPeriodicTrigger.cancel(false);
                        currentPeriodicTrigger = null;
                    }
                    // Reassign the new trigger to the currentPeriodicTrigger
                    //此时延迟时间设置为durationTillNextMillis
                    currentPeriodicTrigger = timer.scheduleAtFixedRate(
                            new ScheduledTrigger(),
                            durationTillNextMillis, baseInterval, TimeUnit.MILLISECONDS);

                    return new CheckpointTriggerResult(CheckpointDeclineReason.MINIMUM_TIME_BETWEEN_CHECKPOINTS);
                }
            }
        }

        // check if all tasks that we need to trigger are running.
        // if not, abort the checkpoint
        // 检查需要触发checkpoint的所有Tasks是否处于运行状态，如果有一个不满足条件，则不触发检查点
        Execution[] executions = new Execution[tasksToTrigger.length];
        for (int i = 0; i < tasksToTrigger.length; i++) {
            Execution ee = tasksToTrigger[i].getCurrentExecutionAttempt();
            if (ee != null && ee.getState() == ExecutionState.RUNNING) {
                executions[i] = ee;
            } else {
                LOG.info("Checkpoint triggering task {} is not being executed at the moment. Aborting checkpoint.",
                        tasksToTrigger[i].getTaskNameWithSubtaskIndex());
                return new CheckpointTriggerResult(CheckpointDeclineReason.NOT_ALL_REQUIRED_TASKS_RUNNING);
            }
        }

        // next, check if all tasks that need to acknowledge the checkpoint are running.
        // if not, abort the checkpoint
        //检查所有需要ack的tasks是否都处于运行状态，如果有一个不满足条件，则不触发检查点。
        Map<ExecutionAttemptID, ExecutionVertex> ackTasks = new HashMap<>(tasksToWaitFor.length);

        for (ExecutionVertex ev : tasksToWaitFor) {
            Execution ee = ev.getCurrentExecutionAttempt();
            if (ee != null) {
                ackTasks.put(ee.getAttemptId(), ev);
            } else {
                LOG.info("Checkpoint acknowledging task {} is not being executed at the moment. Aborting checkpoint.",
                        ev.getTaskNameWithSubtaskIndex());
                return new CheckpointTriggerResult(CheckpointDeclineReason.NOT_ALL_REQUIRED_TASKS_RUNNING);
            }
        }

        // we will actually trigger this checkpoint!

        // we lock with a special lock to make sure that trigger requests do not overtake each other.
        // this is not done with the coordinator-wide lock, because the 'checkpointIdCounter'
        // may issue blocking operations. Using a different lock than the coordinator-wide lock,
        // we avoid blocking the processing of 'acknowledge/decline' messages during that time.
        // 触发检查点，在triggerLock同步代码块中完成，而不是使用coordinator范围的锁。
        synchronized (triggerLock) {
            final long checkpointID;
            //首先获取checkpointID
            try {
                // this must happen outside the coordinator-wide lock, because it communicates
                // with external services (in HA mode) and may block for a while.
                checkpointID = checkpointIdCounter.getAndIncrement();
            }
            catch (Throwable t) {
                int numUnsuccessful = numUnsuccessfulCheckpointsTriggers.incrementAndGet();
                LOG.warn("Failed to trigger checkpoint (" + numUnsuccessful + " consecutive failed attempts so far)", t);
                return new CheckpointTriggerResult(CheckpointDeclineReason.EXCEPTION);
            }

            //创建PendingCheckpoint实例，表示待处理检查点
            final PendingCheckpoint checkpoint = new PendingCheckpoint(
                job,
                checkpointID,
                timestamp,
                ackTasks,
                props,
                targetDirectory,
                executor);

            if (statsTracker != null) {
                PendingCheckpointStats callback = statsTracker.reportPendingCheckpoint(
                    checkpointID,
                    timestamp,
                    props);

                checkpoint.setStatsCallback(callback);
            }

            // schedule the timer that will clean up the expired checkpoints
            // 针对当前checkpoints超时进行资源清理的canceller
            final Runnable canceller = new Runnable() {
                @Override
                public void run() {
                    synchronized (lock) {
                        // only do the work if the checkpoint is not discarded anyways
                        // note that checkpoint completion discards the pending checkpoint object
                        if (!checkpoint.isDiscarded()) {
                            LOG.info("Checkpoint " + checkpointID + " expired before completing.");

                            checkpoint.abortExpired();
                            pendingCheckpoints.remove(checkpointID);
                            rememberRecentCheckpointId(checkpointID);

                            triggerQueuedRequests();
                        }
                    }
                }
            };

            try {
                //重新请求coordinator-wide lock
                // re-acquire the coordinator-wide lock
                synchronized (lock) {
                    // since we released the lock in the meantime, we need to re-check
                    // that the conditions still hold.
                    // 重新检查触发条件，防止产生竞态条件。这里做二次检查的原因是，中间有一段关于获得checkpointId的代码，不在同步块中。
                    if (shutdown) {
                        return new CheckpointTriggerResult(CheckpointDeclineReason.COORDINATOR_SHUTDOWN);
                    }
                    else if (!props.forceCheckpoint()) {
                        if (triggerRequestQueued) {
                            LOG.warn("Trying to trigger another checkpoint while one was queued already");
                            return new CheckpointTriggerResult(CheckpointDeclineReason.ALREADY_QUEUED);
                        }

                        if (pendingCheckpoints.size() >= maxConcurrentCheckpointAttempts) {
                            triggerRequestQueued = true;
                            if (currentPeriodicTrigger != null) {
                                currentPeriodicTrigger.cancel(false);
                                currentPeriodicTrigger = null;
                            }
                            return new CheckpointTriggerResult(CheckpointDeclineReason.TOO_MANY_CONCURRENT_CHECKPOINTS);
                        }

                        // make sure the minimum interval between checkpoints has passed
                        final long earliestNext = lastCheckpointCompletionNanos + minPauseBetweenCheckpointsNanos;
                        final long durationTillNextMillis = (earliestNext - System.nanoTime()) / 1_000_000;

                        if (durationTillNextMillis > 0) {
                            if (currentPeriodicTrigger != null) {
                                currentPeriodicTrigger.cancel(false);
                                currentPeriodicTrigger = null;
                            }

                            // Reassign the new trigger to the currentPeriodicTrigger
                            currentPeriodicTrigger = timer.scheduleAtFixedRate(
                                    new ScheduledTrigger(),
                                    durationTillNextMillis, baseInterval, TimeUnit.MILLISECONDS);

                            return new CheckpointTriggerResult(CheckpointDeclineReason.MINIMUM_TIME_BETWEEN_CHECKPOINTS);
                        }
                    }

                    LOG.info("Triggering checkpoint " + checkpointID + " @ " + timestamp);

                    //将checkpoint加入到pendingCheckpoints中
                    pendingCheckpoints.put(checkpointID, checkpoint);

                    //启动超时canceller，延迟checkpointTimeout执行
                    ScheduledFuture<?> cancellerHandle = timer.schedule(
                            canceller,
                            checkpointTimeout, TimeUnit.MILLISECONDS);

                    if (!checkpoint.setCancellerHandle(cancellerHandle)) {
                        // checkpoint is already disposed!
                        cancellerHandle.cancel(false);
                    }

                    // trigger the master hooks for the checkpoint
                    final List<MasterState> masterStates = MasterHooks.triggerMasterHooks(masterHooks.values(),
                            checkpointID, timestamp, executor, Time.milliseconds(checkpointTimeout));
                    for (MasterState s : masterStates) {
                        checkpoint.addMasterState(s);
                    }
                }
                // end of lock scope

                CheckpointOptions checkpointOptions;
                if (!props.isSavepoint()) {
                    checkpointOptions = CheckpointOptions.forFullCheckpoint();
                } else {
                    checkpointOptions = CheckpointOptions.forSavepoint(targetDirectory);
                }

                // send the messages to the tasks that trigger their checkpoint
                // 向tasks发送消息，触发它们的检查点
                for (Execution execution: executions) {
                    execution.triggerCheckpoint(checkpointID, timestamp, checkpointOptions);
                }

                numUnsuccessfulCheckpointsTriggers.set(0);
                return new CheckpointTriggerResult(checkpoint);
            }
            catch (Throwable t) {
                // guard the map against concurrent modifications
                synchronized (lock) {
                    pendingCheckpoints.remove(checkpointID);
                }

                int numUnsuccessful = numUnsuccessfulCheckpointsTriggers.incrementAndGet();
                LOG.warn("Failed to trigger checkpoint {}. ({} consecutive failed attempts so far)",
                        checkpointID, numUnsuccessful, t);

                if (!checkpoint.isDiscarded()) {
                    checkpoint.abortError(new Exception("Failed to trigger checkpoint", t));
                }
                return new CheckpointTriggerResult(CheckpointDeclineReason.EXCEPTION);
            }

        } // end trigger lock
    }

 

　　启动定时任务方法：startCheckpointScheduler

    public void startCheckpointScheduler() {
        synchronized (lock) {
            if (shutdown) {
                throw new IllegalArgumentException("Checkpoint coordinator is shut down");
            }
            //保证所有以前的timer被取消
            stopCheckpointScheduler();

            periodicScheduling = true;
            //scheduleAtFixedRate方法是以固定延迟和固定时间间隔周期性的执行任务
            currentPeriodicTrigger = timer.scheduleAtFixedRate(
                    new ScheduledTrigger(), 
                    baseInterval, baseInterval, TimeUnit.MILLISECONDS);
        }
    }

　　停止定时任务方法：stopCheckpointScheduler

    //重置一些标记变量，释放资源
    public void stopCheckpointScheduler() {
        synchronized (lock) {
            triggerRequestQueued = false;
            periodicScheduling = false;

            if (currentPeriodicTrigger != null) {
                currentPeriodicTrigger.cancel(false);//取消当前周期的触发任务
                currentPeriodicTrigger = null;
            }

            //pendingCheckpoints中存的是待执行的检查点
            for (PendingCheckpoint p : pendingCheckpoints.values()) {
                p.abortError(new Exception("Checkpoint Coordinator is suspending."));
            }
            pendingCheckpoints.clear();//清空pendingCheckpoints
            numUnsuccessfulCheckpointsTriggers.set(0);
        }
    }

 

 

基于Actor的消息驱动的协同机制

　　启动和停止定时任务的机制是怎样的?Flink使用的是基于AKKA的Actor模型的消息驱动机制。

　　CheckpointCoordinatorDeActivator类

org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator

 

　　CheckpointCoordinatorDeActivator是actor的实现类，监听JobStatus的变化，启动和停止周期性的checkpoint调度任务。

    //actor的实现类，监听JobStatus的变化，激活和取消周期性的checkpoint调度任务。
    public class CheckpointCoordinatorDeActivator implements JobStatusListener {

        private final CheckpointCoordinator coordinator;

        public CheckpointCoordinatorDeActivator(CheckpointCoordinator coordinator) {
            this.coordinator = checkNotNull(coordinator);
        }

        @Override
        public void jobStatusChanges(JobID jobId, JobStatus newJobStatus, long timestamp, Throwable error) {
            if (newJobStatus == JobStatus.RUNNING) {
                // start the checkpoint scheduler
                // 一旦监听到JobStatus变为RUNNING，就会启动定时任务
                coordinator.startCheckpointScheduler();
            } else {
                // anything else should stop the trigger for now
                coordinator.stopCheckpointScheduler();
            }
        }
    }

 

　　CheckpointCoordinatorDeActivator的实例是在CheckpointCoordinator中被创建的，方法为createActivatorDeactivator。

    public JobStatusListener createActivatorDeactivator() {
        synchronized (lock) {
            if (shutdown) {
                throw new IllegalArgumentException("Checkpoint coordinator is shut down");
            }

            if (jobStatusListener == null) {
                jobStatusListener = new CheckpointCoordinatorDeActivator(this);
            }

            return jobStatusListener;
        }
    }

 

　　checkpoint相关Akka消息

　　AbstractCheckpointMessage ：所有checkpoint消息的基础抽象类

org.apache.flink.runtime.messages.checkpoint.AbstractCheckpointMessage

　　AbstractCheckpointMessage主要属性：

    /** The job to which this message belongs */
    private final JobID job;
    /** The task execution that is source/target of the checkpoint message */  
    private final ExecutionAttemptID taskExecutionId; //检查点的source/target task
    /** The ID of the checkpoint that this message coordinates */
    private final long checkpointId;

 

　　它有以下实现类：

　　　　TriggerCheckpoint ：JobManager向TaskManager发送的检查点触发消息；

　　　　AcknowledgeCheckpoint ：TaskManager向JobManager发送的某个独立task的检查点完成确认的消息；

　　　　DeclineCheckpoint ：TaskManager向JobManager发送的检查点还没有被处理的消息；

　　　　NotifyCheckpointComplete ：JobManager向TaskManager发送的检查点完成的消息。

　　TriggerCheckpoint消息

　　　　从JobManager发送到TaskManager，通知指定的task触发checkpoint。

　　　　发送消息

　　　　　　发送消息的逻辑是在CheckpointCoordinator中，上文提到过：

        // send the messages to the tasks that trigger their checkpoint
        for (Execution execution: executions) {
            execution.triggerCheckpoint(checkpointID, timestamp, checkpointOptions);
        }

　　　　　　其中executions是Execution[]数组，其中存储的元素是在检查点触发时需要被发送消息的Tasks的集合（即CheckpointCoordinator成员变量tasksToTrigger中的数据）。对每一个要发送的Task执行triggerCheckpoint()方法。

　　　　　　接下来，看一下Execution的triggerCheckpoint方法。

        //在该execution的task上触发一个新的checkpoint
        public void triggerCheckpoint(long checkpointId, long timestamp, CheckpointOptions checkpointOptions) {
            //获取Resource
            final SimpleSlot slot = assignedResource;//获取Slot

            if (slot != null) {
                //TaskManagerGateway是用于和TaskManager通信的抽象基础类
                final TaskManagerGateway taskManagerGateway = slot.getTaskManagerGateway();
                //通过taskManagerGateway向TaskManager发送消息
                taskManagerGateway.triggerCheckpoint(attemptId, getVertex().getJobId(), checkpointId, timestamp, checkpointOptions);
            } else {
                LOG.debug("The execution has no slot assigned. This indicates that the execution is " +
                    "no longer running.");
            }
        }

 

　　　　　　继续进入ActorTaskManagerGateway（TaskManagerGateway抽象类的Actor实现）类的triggerCheckpoint()方法：

        public void triggerCheckpoint(
                ExecutionAttemptID executionAttemptID,
                JobID jobId,
                long checkpointId,
                long timestamp,
                CheckpointOptions checkpointOptions) {

            Preconditions.checkNotNull(executionAttemptID);
            Preconditions.checkNotNull(jobId);
            //新建了一个TriggerCheckpoint消息，通过actorGateway的tell方法（异步发送，没有返回结果）发送这个消息
            //ActorGateway是基于actor通信的接口
            actorGateway.tell(new TriggerCheckpoint(jobId, executionAttemptID, checkpointId, timestamp, checkpointOptions));
        }

　　　　　　AkkaActorGateway类是ActorGateway接口一种实现，它使用 Akka 与远程的actors进行通信。看一下AkkaActorGateway的tell方法：

        @Override
        public void tell(Object message) {
            Object newMessage = decorator.decorate(message);
            //通过ActorRef实例actor发送消息，ActorRef是akka中的类。以后需要研究Akka的实现机制。
            actor.tell(newMessage, ActorRef.noSender());
        }

　　　　　　至此，发送TriggerCheckpoint消息的过程结束。下面将看一下TaskManager接收消息的过程。

　　　　接收消息

　　　　　　TaskManager接收消息的部分是用scala实现的。

org.apache.flink.runtime.taskmanager.TaskManager

　　　　　　TaskManager类的handleMessage方法是消息处理中心。

        //该方法为TaskManager的消息处理中心。接收消息，按消息的种类调用不同的方法处理。
        override def handleMessage: Receive = {
            case message: TaskMessage => handleTaskMessage(message)

            //这个就是处理checkpoints相关的消息
            case message: AbstractCheckpointMessage => handleCheckpointingMessage(message)

            case JobManagerLeaderAddress(address, newLeaderSessionID) =>
              handleJobManagerLeaderAddress(address, newLeaderSessionID)

            case message: RegistrationMessage => handleRegistrationMessage(message)

            ...
        }

 

　　　　　　接下来，看方法handleCheckpointingMessage()，主要是触发Checkpoint Barrier。

        //处理Checkpoint相关的消息
        private def handleCheckpointingMessage(actorMessage: AbstractCheckpointMessage): Unit = {

            actorMessage match {
              //触发Checkpoint消息
              case message: TriggerCheckpoint =>
                val taskExecutionId = message.getTaskExecutionId
                val checkpointId = message.getCheckpointId
                val timestamp = message.getTimestamp
                val checkpointOptions = message.getCheckpointOptions

                log.debug(s"Receiver TriggerCheckpoint $checkpointId@$timestamp for $taskExecutionId.")

                val task = runningTasks.get(taskExecutionId)
                if (task != null) {
                  //调用Task的triggerCheckpointBarrier方法，触发Checkpoint Barrier，Barrier实现机制的细节以后讨论。
                  task.triggerCheckpointBarrier(checkpointId, timestamp, checkpointOptions)
                } else {
                  log.debug(s"TaskManager received a checkpoint request for unknown task $taskExecutionId.")
                }
              //Checkpoint完成通知消息
              case message: NotifyCheckpointComplete =>
                val taskExecutionId = message.getTaskExecutionId
                val checkpointId = message.getCheckpointId
                val timestamp = message.getTimestamp

                log.debug(s"Receiver ConfirmCheckpoint $checkpointId@$timestamp for $taskExecutionId.")

                val task = runningTasks.get(taskExecutionId)
                if (task != null) {
                  //调用Task的notifyCheckpointComplete方法，进行相关处理
                  task.notifyCheckpointComplete(checkpointId)
                } else {
                  log.debug(
                    s"TaskManager received a checkpoint confirmation for unknown task $taskExecutionId.")
                }

              // unknown checkpoint message
              case _ => unhandled(actorMessage)
            }
          }

 

　　NotifyCheckpointComplete消息

　　　　JobManager发送到TaskManager，通知task它的检查点已经得到完成确认，task可以向第三方提交checkpoint。

　　　　发送消息

　　　　　　发送NotifyCheckpointComplete消息的部分在CheckpointCoordinator类的receiveAcknowledgeMessage方法中。

        //该方法接收一个AcknowledgeCheckpoint消息，返回该Message是否与一个pending checkpoint相关联
        public boolean receiveAcknowledgeMessage(AcknowledgeCheckpoint message) throws CheckpointException {
            if (shutdown || message == null) {
                return false;
            }
            if (!job.equals(message.getJob())) {
                LOG.error("Received wrong AcknowledgeCheckpoint message for job {}: {}", job, message);
                return false;
            }

            final long checkpointId = message.getCheckpointId();
            
            synchronized (lock) {
                // we need to check inside the lock for being shutdown as well, otherwise we
                // get races and invalid error log messages
                if (shutdown) {
                    return false;
                }

                final PendingCheckpoint checkpoint = pendingCheckpoints.get(checkpointId);
                
                //如果是待处理的检查点并且没有被Discarded
                if (checkpoint != null && !checkpoint.isDiscarded()) {
                    
                    //根据TaskExecutionId和SubtaskState，Acknowledges the task。确认该任务
                    switch (checkpoint.acknowledgeTask(message.getTaskExecutionId(), message.getSubtaskState(), message.getCheckpointMetrics())) {
                        //确认成功
                        case SUCCESS:
                            LOG.debug("Received acknowledge message for checkpoint {} from task {} of job {}.",
                                checkpointId, message.getTaskExecutionId(), message.getJob());
                            //如果收到了全部task的确认消息（即notYetAcknowledgedTasks为空）
                            if (checkpoint.isFullyAcknowledged()) {
                                //尝试完成PendingCheckpoint（Try to complete the given pending checkpoint）
                                //将完成的checkpointId从checkpoint中删除和一下标志修改，最后，发送notify complete消息
                                completePendingCheckpoint(checkpoint);
                            }
                            break;
                        //重复消息
                        case DUPLICATE:
                            LOG.debug("Received a duplicate acknowledge message for checkpoint {}, task {}, job {}.",
                                message.getCheckpointId(), message.getTaskExecutionId(), message.getJob());
                            break;
                        //未知消息
                        case UNKNOWN:
                            LOG.warn("Could not acknowledge the checkpoint {} for task {} of job {}, " +
                                    "because the task's execution attempt id was unknown. Discarding " +
                                    "the state handle to avoid lingering state.", message.getCheckpointId(),
                                message.getTaskExecutionId(), message.getJob());

                            discardSubtaskState(message.getJob(), message.getTaskExecutionId(), message.getCheckpointId(), message.getSubtaskState());
                            break;
                        //废弃消息
                        case DISCARDED:
                            LOG.warn("Could not acknowledge the checkpoint {} for task {} of job {}, " +
                                "because the pending checkpoint had been discarded. Discarding the " +
                                    "state handle tp avoid lingering state.",
                                message.getCheckpointId(), message.getTaskExecutionId(), message.getJob());
                            discardSubtaskState(message.getJob(), message.getTaskExecutionId(), message.getCheckpointId(), message.getSubtaskState());
                    }

                    return true;
                }
                else if (checkpoint != null) {
                    // this should not happen
                    throw new IllegalStateException(
                            "Received message for discarded but non-removed checkpoint " + checkpointId);
                }
                else {
                    boolean wasPendingCheckpoint;

                    // message is for an unknown checkpoint, or comes too late (checkpoint disposed)
                    if (recentPendingCheckpoints.contains(checkpointId)) {
                        wasPendingCheckpoint = true;
                        LOG.warn("Received late message for now expired checkpoint attempt {} from " +
                            "{} of job {}.", checkpointId, message.getTaskExecutionId(), message.getJob());
                    }
                    else {
                        LOG.debug("Received message for an unknown checkpoint {} from {} of job {}.",
                            checkpointId, message.getTaskExecutionId(), message.getJob());
                        wasPendingCheckpoint = false;
                    }

                    // try to discard the state so that we don't have lingering state lying around
                    discardSubtaskState(message.getJob(), message.getTaskExecutionId(), message.getCheckpointId(), message.getSubtaskState());

                    return wasPendingCheckpoint;
                }
            }
        }

 

　　　　　　completePendingCheckpoint方法中发送NotifyCheckpointComplete消息的代码如下：

        for (ExecutionVertex ev : tasksToCommitTo) {
            Execution ee = ev.getCurrentExecutionAttempt();
            if (ee != null) {
                ee.notifyCheckpointComplete(checkpointId, timestamp);
            }
        }

　　　　接收消息

　　　　　　在TriggerCheckpoint消息接收中的有这部分代码，主要是调用notifyCheckpointComplete方法： task.notifyCheckpointComplete(checkpointId)。

　　AcknowledgeCheckpoint消息

　　　　由TaskManager发向JobManager，告知JobManager指定task的checkpoint已完成。该消息可能携带task的状态和checkpointMetrics。

　　　　AcknowledgeCheckpoint消息类的两个属性：

	private final SubtaskState subtaskState;//任务状态
	private final CheckpointMetrics checkpointMetrics;

　　　　发送消息

　　　　　　发送消息的过程在RuntimeEnvironment类中的acknowledgeCheckpoint方法

        public void acknowledgeCheckpoint(
                long checkpointId,
                CheckpointMetrics checkpointMetrics,
                SubtaskState checkpointStateHandles) {
            //通过CheckpointResponder接口的实例checkpointResponder发送ack消息
            checkpointResponder.acknowledgeCheckpoint(
                    jobId, executionId, checkpointId, checkpointMetrics,
                    checkpointStateHandles);
        }

 

　　　　　　CheckpointResponder接口是checkpoint acknowledge and decline messages 的应答类。ActorGatewayCheckpointResponder是使用了ActorGateway的CheckpointResponder接口的实现类，包含acknowledgeCheckpoint和declineCheckpoint两个方法。

        @Override
        public void acknowledgeCheckpoint(
                JobID jobID,
                ExecutionAttemptID executionAttemptID,
                long checkpointId,
                CheckpointMetrics checkpointMetrics,
                SubtaskState checkpointStateHandles) {
            //新建一个AcknowledgeCheckpoint消息
            AcknowledgeCheckpoint message = new AcknowledgeCheckpoint(
                    jobID, executionAttemptID, checkpointId, checkpointMetrics,
                    checkpointStateHandles);
            //通过actorGateway发送出去
            actorGateway.tell(message);
        }

　　　　接收消息

　　　　　　通过receiveAcknowledgeMessage方法接收（和NotifyCheckpointComplete消息的发送过程在同一个方法）。

　　DeclineCheckpoint消息

　　　　该消息由TaskManager发送给JobManager，用于告知CheckpointCoordinator：检查点的请求还没有能够被处理。这种情况通常发生于：某task已处于RUNNING状态，但在内部可能还没有准备好执行检查点。

　　　　发送消息

　　　　　　位于task类的triggerCheckpointBarrier方法中。

org.apache.flink.runtime.taskmanager.Task

        try {
            boolean success = statefulTask.triggerCheckpoint(checkpointMetaData, checkpointOptions);
            if (!success) {
                //通过CheckpointResponder发送消息，类似发送AcknowledgeCheckpoint消息
                checkpointResponder.declineCheckpoint(
                        getJobID(), getExecutionId(), checkpointID,
                        new CheckpointDeclineTaskNotReadyException(taskName));
            }
        }

 

　　　　接收消息

　　　　　　CheckpointCoordinator中的receiveDeclineMessage方法。

        public void receiveDeclineMessage(DeclineCheckpoint message) {
            if (shutdown || message == null) {
                return;
            }
            if (!job.equals(message.getJob())) {
                throw new IllegalArgumentException("Received DeclineCheckpoint message for job " +
                    message.getJob() + " while this coordinator handles job " + job);
            }
            
            final long checkpointId = message.getCheckpointId();
            final String reason = (message.getReason() != null ? message.getReason().getMessage() : "");

            PendingCheckpoint checkpoint;

            synchronized (lock) {
                // we need to check inside the lock for being shutdown as well, otherwise we
                // get races and invalid error log messages
                if (shutdown) {
                    return;
                }

                checkpoint = pendingCheckpoints.get(checkpointId);

                if (checkpoint != null && !checkpoint.isDiscarded()) {
                    //如果是待处理的Checkpoint且没有被遗弃
                    LOG.info("Discarding checkpoint {} because of checkpoint decline from task {} : {}",
                            checkpointId, message.getTaskExecutionId(), reason);

                    pendingCheckpoints.remove(checkpointId);//将checkpointId从pendingCheckpoints中删除
                    checkpoint.abortDeclined();
                    rememberRecentCheckpointId(checkpointId);

                    // we don't have to schedule another "dissolving" checkpoint any more because the
                    // cancellation barriers take care of breaking downstream alignments
                    // we only need to make sure that suspended queued requests are resumed

                    //是否还有更多pending 的checkpoint
                    boolean haveMoreRecentPending = false;
                    for (PendingCheckpoint p : pendingCheckpoints.values()) {
                        if (!p.isDiscarded() && p.getCheckpointId() >= checkpoint.getCheckpointId()) {
                            haveMoreRecentPending = true;
                            break;
                        }
                    }
                    //
                    if (!haveMoreRecentPending) {
                        triggerQueuedRequests();
                    }
                }
                else if (checkpoint != null) {
                    // this should not happen
                    throw new IllegalStateException(
                            "Received message for discarded but non-removed checkpoint " + checkpointId);
                }
                else if (LOG.isDebugEnabled()) {
                    if (recentPendingCheckpoints.contains(checkpointId)) {
                        // message is for an unknown checkpoint, or comes too late (checkpoint disposed)
                        LOG.debug("Received another decline message for now expired checkpoint attempt {} : {}",
                                checkpointId, reason);
                    } else {
                        // message is for an unknown checkpoint. might be so old that we don't even remember it any more
                        LOG.debug("Received decline message for unknown (too old?) checkpoint attempt {} : {}",
                                checkpointId, reason);
                    }
                }
            }
        }

 

 

 

 

 

 

 

我的博客即将同步至腾讯云+社区，邀请大家一同入驻：https://cloud.tencent.com/developer/support-plan?invite_code=vq5xxge4ldlk