【Kafka源码】KafkaController启动过程
目录
之前聊过了很多Kafka启动过程中的一些加载内容，也知道了broker可以分为很多的partition，每个partition内部也可以分为leader和follower，主从之间有数据的复制。那么这么多partition是谁在管理？broker内部有没有主从之分？这就是本文的主角，KafkaController，本文将细细道来。
一、入口
KafkaController的启动入口同样很简洁，在KafkaServer的start方法中。
/* start kafka controller */
kafkaController = new KafkaController(config, zkUtils, brokerState, kafkaMetricsTime, metrics, threadNamePrefix)
kafkaController.startup()
首先实例化一个KafkaController，之后启动了这个controller。
二、实例化Controller
实例化的源码，见注释：
this.logIdent = "[Controller " + config.brokerId + "]: "
private var isRunning = true
private val stateChangeLogger = KafkaController.stateChangeLogger
//实例化上下文
val controllerContext = new ControllerContext(zkUtils, config.zkSessionTimeoutMs)
//实例化partition状态机
val partitionStateMachine = new PartitionStateMachine(this)
//实例化replica状态机
val replicaStateMachine = new ReplicaStateMachine(this)
//实例化broker的leader选举器
private val controllerElector = new ZookeeperLeaderElector(controllerContext, ZkUtils.ControllerPath, onControllerFailover,
onControllerResignation, config.brokerId)
// have a separate scheduler for the controller to be able to start and stop independently of the
// kafka server
//实例化负载均衡定时器
private val autoRebalanceScheduler = new KafkaScheduler(1)
//topic删除管理器
var deleteTopicManager: TopicDeletionManager = null
//离线分区leader选择器
val offlinePartitionSelector = new OfflinePartitionLeaderSelector(controllerContext, config)
//重新分配分区leader
private val reassignedPartitionLeaderSelector = new ReassignedPartitionLeaderSelector(controllerContext)
//重新分配leader时优先选择的replica
private val preferredReplicaPartitionLeaderSelector = new PreferredReplicaPartitionLeaderSelector(controllerContext)
//controller关闭后的leader选举
private val controlledShutdownPartitionLeaderSelector = new ControlledShutdownLeaderSelector(controllerContext)
private val brokerRequestBatch = new ControllerBrokerRequestBatch(this)
//重分配监听器
private val partitionReassignedListener = new PartitionsReassignedListener(this)
//优选replica选举监听器
private val preferredReplicaElectionListener = new PreferredReplicaElectionListener(this)
//isr变化通知监听器
private val isrChangeNotificationListener = new IsrChangeNotificationListener(this)
三、Controller启动
直接上代码：
def startup() = {
inLock(controllerContext.controllerLock) {
info("Controller starting up")
registerSessionExpirationListener()
isRunning = true
controllerElector.startup
info("Controller startup complete")
}
}
这个start方法并不意味着当前的broker就是controller，只是把它注册到zk上面，后面zk会进行选举，选举出controller后，在controller机器上面会执行一系列的操作，后面我们能看到。
3.1 registerSessionExpirationListener
首先，我们的broker会注册一个session过期的监听器，我们看一下这个监听器。
private def registerSessionExpirationListener() = {
zkUtils.zkClient.subscribeStateChanges(new SessionExpirationListener())
}
class SessionExpirationListener() extends IZkStateListener with Logging {
this.logIdent = "[SessionExpirationListener on " + config.brokerId + "], "
@throws(classOf[Exception])
def handleStateChanged(state: KeeperState) {
// do nothing, since zkclient will do reconnect for us.
}
/**
* Called after the zookeeper session has expired and a new session has been created. You would have to re-create
* any ephemeral nodes here.
*
* @throws Exception
* On any error.
*/
@throws(classOf[Exception])
def handleNewSession() {
info("ZK expired; shut down all controller components and try to re-elect")
inLock(controllerContext.controllerLock) {
onControllerResignation()
controllerElector.elect
}
}
override def handleSessionEstablishmentError(error: Throwable): Unit = {
//no-op handleSessionEstablishmentError in KafkaHealthCheck should handle this error in its handleSessionEstablishmentError
}
}
可以看到，当broker到zk的session失效之后，broker并不会主动发起重连操作，而是等待zk的重连，当新的session被创建后，也就是当前broker加入到broker列表中之后，会进行两个操作：
- onControllerResignation：也就是当前controller失效
- controllerElector.elect：重新进行controller选举
下面我们分别看看做了啥。
3.1.1 onControllerResignation
从代码看会比较直观，主要就是清理一些controller的数据。
/**
* This callback is invoked by the zookeeper leader elector when the current broker resigns as the controller. This is
* required to clean up internal controller data structures
*/
def onControllerResignation() {
debug("Controller resigning, broker id %d".format(config.brokerId))
// de-register listeners 取消订阅监听器
deregisterIsrChangeNotificationListener()
deregisterReassignedPartitionsListener()
deregisterPreferredReplicaElectionListener()
// shutdown delete topic manager 关闭topic删除管理器
if (deleteTopicManager != null)
deleteTopicManager.shutdown()
// shutdown leader rebalance scheduler 关闭负载均衡定时器
if (config.autoLeaderRebalanceEnable)
autoRebalanceScheduler.shutdown()
inLock(controllerContext.controllerLock) {
// de-register partition ISR listener for on-going partition reassignment task 取消订阅ISR监听器
deregisterReassignedPartitionsIsrChangeListeners()
// shutdown partition state machine 关闭分区状态机
partitionStateMachine.shutdown()
// shutdown replica state machine 关闭replica状态机
replicaStateMachine.shutdown()
// shutdown controller channel manager 关闭控制器管道管理器
if (controllerContext.controllerChannelManager != null) {
controllerContext.controllerChannelManager.shutdown()
controllerContext.controllerChannelManager = null
}
// reset controller context
controllerContext.epoch = 0
controllerContext.epochZkVersion = 0
brokerState.newState(RunningAsBroker)//把当前broker状态从controller改为broker
info("Broker %d resigned as the controller".format(config.brokerId))
}
}
3.1.2 controllerElector.elect
这块是进行controller的重新选举。
def elect: Boolean = {
val timestamp = SystemTime.milliseconds.toString
val electString = Json.encode(Map("version" -> 1, "brokerid" -> brokerId, "timestamp" -> timestamp))
leaderId = getControllerID
/*
* We can get here during the initial startup and the handleDeleted ZK callback. Because of the potential race condition,
* it's possible that the controller has already been elected when we get here. This check will prevent the following
* createEphemeralPath method from getting into an infinite loop if this broker is already the controller.
*/
if(leaderId != -1) {
debug("Broker %d has been elected as leader, so stopping the election process.".format(leaderId))
return amILeader
}
try {
val zkCheckedEphemeral = new ZKCheckedEphemeral(electionPath,
electString,
controllerContext.zkUtils.zkConnection.getZookeeper,
JaasUtils.isZkSecurityEnabled())
zkCheckedEphemeral.create()
info(brokerId + " successfully elected as leader")
leaderId = brokerId
onBecomingLeader()
} catch {
case e: ZkNodeExistsException =>
// If someone else has written the path, then
leaderId = getControllerID
if (leaderId != -1)
debug("Broker %d was elected as leader instead of broker %d".format(leaderId, brokerId))
else
warn("A leader has been elected but just resigned, this will result in another round of election")
case e2: Throwable =>
error("Error while electing or becoming leader on broker %d".format(brokerId), e2)
resign()
}
amILeader
}
这块主要进行的是controller的选举，我们着重看下当前broker被选为controller之后的动作，也就是onBecomingLeader。这块就需要我们返回到实例化中去看下，这个动作是：onControllerFailover。
def onControllerFailover() {
if (isRunning) {
info("Broker %d starting become controller state transition".format(config.brokerId))
//read controller epoch from zk
readControllerEpochFromZookeeper()
// increment the controller epoch
incrementControllerEpoch(zkUtils.zkClient)
// before reading source of truth from zookeeper, register the listeners to get broker/topic callbacks
registerReassignedPartitionsListener()
registerIsrChangeNotificationListener()
registerPreferredReplicaElectionListener()
partitionStateMachine.registerListeners()
replicaStateMachine.registerListeners()
initializeControllerContext()
replicaStateMachine.startup()
partitionStateMachine.startup()
// register the partition change listeners for all existing topics on failover
controllerContext.allTopics.foreach(topic => partitionStateMachine.registerPartitionChangeListener(topic))
info("Broker %d is ready to serve as the new controller with epoch %d".format(config.brokerId, epoch))
brokerState.newState(RunningAsController)
maybeTriggerPartitionReassignment()
maybeTriggerPreferredReplicaElection()
/* send partition leadership info to all live brokers */
sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq)
if (config.autoLeaderRebalanceEnable) {
info("starting the partition rebalance scheduler")
autoRebalanceScheduler.startup()
autoRebalanceScheduler.schedule("partition-rebalance-thread", checkAndTriggerPartitionRebalance,
5, config.leaderImbalanceCheckIntervalSeconds.toLong, TimeUnit.SECONDS)
}
deleteTopicManager.start()
}
else
info("Controller has been shut down, aborting startup/failover")
}
这里面执行的动作很多，我们一一分析。
- 首先从zk中读取controller的epoch
- 然后将epoch+1后更新到zk中
- 注册一系列监听器
- 初始化controller上下文
- 启动两个状态机
- 订阅所有topic的分区变化监听器
- 定时检查触发分区选举
- 启动topic删除管理器
这里面的东西比较多，我们后面文章再分析。
3.2 controllerElector.startup
def startup {
inLock(controllerContext.controllerLock) {
controllerContext.zkUtils.zkClient.subscribeDataChanges(electionPath, leaderChangeListener)
elect
}
}
这里的electionPath是/controller，下面我们看下这个leaderChangeListener。
3.2.1 leaderChangeListener
class LeaderChangeListener extends IZkDataListener with Logging {
/**
* Called when the leader information stored in zookeeper has changed. Record the new leader in memory
* @throws Exception On any error.
*/
@throws(classOf[Exception])
def handleDataChange(dataPath: String, data: Object) {
inLock(controllerContext.controllerLock) {
val amILeaderBeforeDataChange = amILeader
leaderId = KafkaController.parseControllerId(data.toString)
info("New leader is %d".format(leaderId))
// The old leader needs to resign leadership if it is no longer the leader
if (amILeaderBeforeDataChange && !amILeader)
onResigningAsLeader()
}
}
/**
* Called when the leader information stored in zookeeper has been delete. Try to elect as the leader
* @throws Exception
* On any error.
*/
@throws(classOf[Exception])
def handleDataDeleted(dataPath: String) {
inLock(controllerContext.controllerLock) {
debug("%s leader change listener fired for path %s to handle data deleted: trying to elect as a leader"
.format(brokerId, dataPath))
if(amILeader)
onResigningAsLeader()
elect
}
}
}
监听对应的zk节点，如果节点发生了变化，调用handleDataChange方法，主要内容是获取当前的leaderId。如果当前broker之前是leader，而新的leader不是自己，那么就会调用onResigningAsLeader方法，清除之前的leader信息。
如果节点被删除了，就会调用handleDataDeleted方法。如果当前broker是leader，会首先调用onResigningAsLeader方法，然后发起新的leader选举。
3.2.2 elect
这边就是我们的controller即leader选举方法。与3.1.2的内容一致。
