|NO.Z.00027|——————————|BigDataEnd|——|Hadoop&kafka.V12|——|kafka.v12|消费者|消费组|心跳机制.v02|

一、心跳机制

二、消费者宕机，退出消费组，触发再平衡，重新给消费组中的消费者分配分区。

### --- 由于broker宕机，主题X的分区3宕机，此时分区3没有Leader副本，触发再平衡，

~~~     消费者4没有对应的主题分区，则消费者4闲置。

### --- Kafka 的心跳是 Kafka Consumer 和 Broker 之间的健康检查，

~~~     只有当 Broker Coordinator 正常时，Consumer 才会发送心跳。

三、Consumer 和 Rebalance 相关的 2 个配置参数：

参数	字段
session.timeout.ms	MemberMetadata.sessionTimeoutMs
max.poll.interval.ms	MemberMetadata.rebalanceTimeoutMs

### --- broker 端，sessionTimeoutMs 参数

~~~     broker 处理心跳的逻辑在 GroupCoordinator 类中：
~~~     如果心跳超期， broker coordinator 会把消费者从 group 中移除，并触发 rebalance。

private def completeAndScheduleNextHeartbeatExpiration(group:GroupMetadata, member: MemberMetadata) {
    // complete current heartbeat expectation
    member.latestHeartbeat = time.milliseconds()val memberKey = MemberKey(member.groupId, member.memberId)heartbeatPurgatory.checkAndComplete(memberKey)
    // reschedule the next heartbeat expiration deadline
    
    // 计算心跳截止时刻
    val newHeartbeatDeadline = member.latestHeartbeat + member.sessionTimeoutMsval delayedHeartbeat = 
        new DelayedHeartbeat(this, group, member,newHeartbeatDeadline, member.sessionTimeoutMs)
    heartbeatPurgatory.tryCompleteElseWatch(delayedHeartbeat,Seq(memberKey))}
    
    // 心跳过期
    def onExpireHeartbeat(group: GroupMetadata, member: MemberMetadata,heartbeatDeadline: Long) {
        group.inLock {
            if (!shouldKeepMemberAlive(member, heartbeatDeadline)) {
                info(s"Member ${member.memberId} in group ${group.groupId} has failed, removing it from the group")
                removeMemberAndUpdateGroup(group, member)
            }
        }
    }

    private def shouldKeepMemberAlive(member: MemberMetadata,heartbeatDeadline: Long) = 
        member.awaitingJoinCallback != null ||
        member.awaitingSyncCallback != null ||
        member.latestHeartbeat + member.sessionTimeoutMs > heartbeatDeadline

~~~     consumer 端：sessionTimeoutMs，rebalanceTimeoutMs 参数
~~~     如果客户端发现心跳超期，客户端会标记 coordinator 为不可用，并阻塞心跳线程；
~~~     如果超过了poll 消息的间隔超过了 rebalanceTimeoutMs，
~~~     则 consumer 告知 broker 主动离开消费组，也会触发rebalance

### --- org.apache.kafka.clients.consumer.internals.AbstractCoordinator.HeartbeatThread

                        if (coordinatorUnknown()) {
                            if (findCoordinatorFuture != null || lookupCoordinator().failed())
                                // the immediate future check ensures that we backoff properly in the case that no
                                // brokers are available to connect to.
                                AbstractCoordinator.this.wait(retryBackoffMs);
                        } else if (heartbeat.sessionTimeoutExpired(now)) {
                            // the session timeout has expired without seeing a successful heartbeat, so we should
                            // probably make sure the coordinator is still healthy.
                            markCoordinatorUnknown();
                        } else if (heartbeat.pollTimeoutExpired(now)) {
                            // the poll timeout has expired, which means that the foreground thread has stalled
                            // in between calls to poll(), so we explicitly leave the group.
                            maybeLeaveGroup();
                        } else if (!heartbeat.shouldHeartbeat(now)) {
                            // poll again after waiting for the retry backoff in case the heartbeat failed or the
                            // coordinator disconnected
                            AbstractCoordinator.this.wait(retryBackoffMs);
                        } else {
                            heartbeat.sentHeartbeat(now);

                            sendHeartbeatRequest().addListener(new RequestFutureListener<Void>() {
                                @Override
                                public void onSuccess(Void value) {
                                    synchronized (AbstractCoordinator.this) {
                                        heartbeat.receiveHeartbeat(time.milliseconds());
                                    }
                                }

                                @Override
                                public void onFailure(RuntimeException e) {
                                    synchronized (AbstractCoordinator.this) {
                                        if (e instanceof RebalanceInProgressException) {
                                            // it is valid to continue heartbeating while the group is rebalancing. This
                                            // ensures that the coordinator keeps the member in the group for as long
                                            // as the duration of the rebalance timeout. If we stop sending heartbeats,
                                            // however, then the session timeout may expire before we can rejoin.
                                            heartbeat.receiveHeartbeat(time.milliseconds());
                                        } else {
                                            heartbeat.failHeartbeat();

                                            // wake up the thread if it's sleeping to reschedule the heartbeat
                                            AbstractCoordinator.this.notify();
                                        }
                                    }
                                }
                            });
                        }

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---