HadoopSourceAnalyse --- Mapreduce ApplicationMaster TaskAttempt FSM
Overview
TaskAttempt 对像将是我们的task最后真正执行的地方,前面的所有的工作都是为了让我们的Attempt服务, 在Attempt被创建出来之后,Attemp开始等待SChedule 或ReSchedule 事件, 该事件将由Taskimpl对像触发。
图 1-1
T_SCHEDULE AND T_RESCHEDULE Handle
当Attempt收到该 事件后,Attempt首先通知所有Speculator 将要申请 container, 然后 向 ContainerAlloctor 申请container来运行当前的task:
// Tell any speculator that we're requesting a container taskAttempt.eventHandler.handle (new SpeculatorEvent(taskAttempt.getID().getTaskId(), +1)); //request for container if (rescheduled) { taskAttempt.eventHandler.handle( ContainerRequestEvent.createContainerRequestEventForFailedContainer( taskAttempt.attemptId, taskAttempt.resourceCapability)); } else { taskAttempt.eventHandler.handle(new ContainerRequestEvent( taskAttempt.attemptId, taskAttempt.resourceCapability, taskAttempt.dataLocalHosts.toArray( new String[taskAttempt.dataLocalHosts.size()]), taskAttempt.dataLocalRacks.toArray( new String[taskAttempt.dataLocalRacks.size()]))); }
之后Attempt进入UNASSIGN状态,等待container的分配结果,当ContainerAllocator完成请求之后,会向Attempt发送TA_ASSIGN事件通知:
图 2-1
TA_ASSIGN Handle
收到该 事件后,Attempt保存Container相关信息,然后用该 信息创建在Container中执行的用来完task Command及所需的运行时上下文信息:
Container container = cEvent.getContainer(); taskAttempt.container = container; // this is a _real_ Task (classic Hadoop mapred flavor): taskAttempt.remoteTask = taskAttempt.createRemoteTask(); taskAttempt.jvmID = new WrappedJvmID(taskAttempt.remoteTask.getTaskID().getJobID(), taskAttempt.remoteTask.isMapTask(), taskAttempt.container.getId() .getId()); taskAttempt.taskAttemptListener.registerPendingTask( taskAttempt.remoteTask, taskAttempt.jvmID); taskAttempt.computeRackAndLocality(); //launch the container //create the container object to be launched for a given Task attempt ContainerLaunchContext launchContext = createContainerLaunchContext( cEvent.getApplicationACLs(), taskAttempt.conf, taskAttempt.jobToken, taskAttempt.remoteTask, taskAttempt.oldJobId, taskAttempt.jvmID, taskAttempt.taskAttemptListener, taskAttempt.credentials); taskAttempt.eventHandler .handle(new ContainerRemoteLaunchEvent(taskAttempt.attemptId, launchContext, container, taskAttempt.remoteTask)); // send event to speculator that our container needs are satisfied taskAttempt.eventHandler.handle (new SpeculatorEvent(taskAttempt.getID().getTaskId(), -1));
最后,提交请求给ContainerLauncher请求launch并启动该 container 来执行任伤,并通知Speculator 任务已经启动.
图 3-1
Attempt 进入ASSIGNED状态,等待TA_LAUNCHED事件。
TA_LAUNCHED Handle
收到该事件后,Attempt 保存该 Container运行时相关信息:包括:
- launchTime
- trackerName
- httpPort
然后更新Counter 信息,通知history 服务器、Speculator 服务, 最后通知 Task:
TaskAttemptContainerLaunchedEvent event = (TaskAttemptContainerLaunchedEvent) evnt; //set the launch time taskAttempt.launchTime = taskAttempt.clock.getTime(); taskAttempt.shufflePort = event.getShufflePort(); // register it to TaskAttemptListener so that it can start monitoring it. taskAttempt.taskAttemptListener .registerLaunchedTask(taskAttempt.attemptId, taskAttempt.jvmID); //TODO Resolve to host / IP in case of a local address. InetSocketAddress nodeHttpInetAddr = // TODO: Costly to create sock-addr? NetUtils.createSocketAddr(taskAttempt.container.getNodeHttpAddress()); taskAttempt.trackerName = nodeHttpInetAddr.getHostName(); taskAttempt.httpPort = nodeHttpInetAddr.getPort(); taskAttempt.sendLaunchedEvents(); taskAttempt.eventHandler.handle (new SpeculatorEvent (taskAttempt.attemptId, true, taskAttempt.clock.getTime())); //make remoteTask reference as null as it is no more needed //and free up the memory taskAttempt.remoteTask = null; //tell the Task that attempt has started taskAttempt.eventHandler.handle(new TaskTAttemptEvent( taskAttempt.attemptId, TaskEventType.T_ATTEMPT_LAUNCHED));
AttemptListenr 开始监听该任务的执行,Attempt进入RUNNING状态,等等待Task及Container执行完成退出事件。