Kafka replication

Kafka replication

kafka_replication_detailed_design_v2.pdf

kafka Detailed Replication Design V3

Apache Kafka中Follower如何从Leader fetch消息

Kafka深度解析，众人推荐，精彩好文！

Kafka 的集群复制设计

Kafka的Log存储解析

KIP-1 - Remove support of request.required.acks

0.8.0 Producer Example

Hands-free Kafka Replication: A lesson in operational simplicity

Kafka Replication

Intra-cluster Replication in Apache Kafka

The current leader of a partition further maintains 3 sets: AR, ISR, CUR and RAR, which correspond to the set of replicas that are assigned to the partition, in-sync with the leader, catching up with the leader, and being reassigned to other brokers. Normally, ISR ⊆ AR and AR = ISR + CUR. 

Replica

Replica { // a replica of a partition
broker_id : int
partition : Partition
isLocal : Boolean // is this replica local to this broker
log : Log // local log associated with this replica 
hw : long // offset of the last committed message 
leo : long // log end offset
}

Partition

Partition { //a partition in a topic 
    topic : string 
    partition_id : int
    leader: Replica // the leader replica of this partition
    commitQ: Queue // produce requests pending commit at the leader
    AR: Set[Replica] // replicas assigned to this partition
    ISR: Set[Replica] // In-sync replica set, maintained at the leader
    CUR: Set[Replica] // Catch-up replica set, maintained at the leader
    RAR: Set[Replica] // Reassigned replica set, maintained at the leader
}