Kafka Broker调优

3.1 Broker 核心参数配置

参数名称	描述
replica.lag.time.max.ms	ISR 中，如果 Follower 长时间未向 Leader 发送通 信请求或同步数据，则该 Follower 将被踢出 ISR。 该时间阈值，默认 30s。
auto.leader.rebalance.enable	默认是 true。 自动 Leader Partition 平衡。
leader.imbalance.per.broker.percentage	默认是 10%。每个 broker 允许的不平衡的 leader 的比率。如果每个 broker 超过了这个值，控制器 会触发 leader 的平衡。
leader.imbalance.check.interval.seconds	默认值 300 秒。检查 leader 负载是否平衡的间隔时 间。
log.segment.bytes	Kafka 中 log 日志是分成一块块存储的，此配置是 指 log 日志划分 成块的大小，默认值 1G。
log.index.interval.bytes	默认 4kb，kafka 里面每当写入了 4kb 大小的日志 （.log），然后就往 index 文件里面记录一个索引。
log.retention.hours	Kafka 中数据保存的时间，默认 7 天。
log.retention.minutes	Kafka 中数据保存的时间，分钟级别，默认关闭。
log.retention.ms	Kafka 中数据保存的时间，毫秒级别，默认关闭。
log.retention.check.interval.ms	检查数据是否保存超时的间隔，默认是 5 分钟。
log.retention.bytes	默认等于-1，表示无穷大。超过设置的所有日志总 大小，删除最早的 segment。
log.cleanup.policy	默认是 delete，表示所有数据启用删除策略； 如果设置值为 compact，表示所有数据启用压缩策略。
num.io.threads	默认是 8。负责写磁盘的线程数。整个参数值要占 总核数的 50%。
num.replica.fetchers	副本拉取线程数，这个参数占总核数的 50%的 1/3
num.network.threads	默认是 3。数据传输线程数，这个参数占总核数的 50%的 2/3 。
log.flush.interval.messages	强制页缓存刷写到磁盘的条数，默认是 long 的最 大值，9223372036854775807。一般不建议修改， 交给系统自己管理。
log.flush.interval.ms	每隔多久，刷数据到磁盘，默认是 null。一般不建 议修改，交给系统自己管理。

3.2 服役新节点/退役旧节点

（1）创建一个要均衡的主题。

[atguigu@hadoop102 scripts]$ vim topics-to-move.json

{
  "topics": [
    {
      "topic": "first"
    }
  ],
  "version": 1
}

（2）生成一个负载均衡的计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[1,2,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[2,0,1],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[2,3,0],"log_dirs":["any","any","any"]}]}

（3）创建副本存储计划（所有副本存储在 broker0、broker1、broker2、broker3 中）。

[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

{
  "version": 1,
  "partitions": [
    {
      "topic": "first",
      "partition": 0,
      "replicas": [
        2,
        3,
        0
      ],
      "log_dirs": [
        "any",
        "any",
        "any"
      ]
    },
    {
      "topic": "first",
      "partition": 1,
      "replicas": [
        3,
        0,
        1
      ],
      "log_dirs": [
        "any",
        "any",
        "any"
      ]
    },
    {
      "topic": "first",
      "partition": 2,
      "replicas": [
        0,
        1,
        2
      ],
      "log_dirs": [
        "any",
        "any",
        "any"
      ]
    }
  ]
}

（4）执行副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[1,2,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[2,0,1],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for first-0,first-1,first-2

（5）验证副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.

Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

3.3 增加分区

1）修改分区数（注意：分区数只能增加，不能减少）

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --alter --topic first --partitions 3

3.4 增加副本因子

1）创建 topic

[atguigu@hadoop102 kafka]$ ./bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --create --partitions 3 --replication-factor 1 --topic four
Created topic four.

2）手动增加副本存储

（1）创建副本存储计划（所有副本都指定存储在 broker0、broker1、broker2 中）。

[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

输入如下内容：

{
  "version": 1,
  "partitions": [
    {
      "topic": "four",
      "partition": 0,
      "replicas": [
        0,
        1,
        2
      ]
    },
    {
      "topic": "four",
      "partition": 1,
      "replicas": [
        0,
        1,
        2
      ]
    },
    {
      "topic": "four",
      "partition": 2,
      "replicas": [
        0,
        1,
        2
      ]
    }
  ]
}

（2）执行副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"four","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"four","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"four","partition":2,"replicas":[0],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for four-0,four-1,four-2

3.5 手动调整分区副本存储

（1）创建副本存储计划（所有副本都指定存储在 broker0、broker1 中）。

[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

{
  "version": 1,
  "partitions": [
    {
      "topic": "three",
      "partition": 0,
      "replicas": [
        0,
        1
      ]
    },
    {
      "topic": "three",
      "partition": 1,
      "replicas": [
        0,
        1
      ]
    },
    {
      "topic": "three",
      "partition": 2,
      "replicas": [
        1,
        0
      ]
    },
    {
      "topic": "three",
      "partition": 3,
      "replicas": [
        1,
        0
      ]
    }
  ]
}

（2）执行副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"three","partition":0,"replicas":[3,1],"log_dirs":["any","any"]},{"topic":"three","partition":1,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"three","partition":2,"replicas":[0,2],"log_dirs":["any","any"]},{"topic":"three","partition":3,"replicas":[2,3],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for three-0,three-1,three-2,three-3

（3）验证副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition three-0 is complete.
Reassignment of partition three-1 is complete.
Reassignment of partition three-2 is complete.
Reassignment of partition three-3 is complete.

Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic three

3.6 Leader Partition 负载平衡

参数名称	描述
auto.leader.rebalance.enable	默认是 true。 自动 Leader Partition 平衡。生产环境中，leader 重选举的代价比较大，可能会带来性能影响，建议设置为 false 关闭。
leader.imbalance.per.broker.percentage	默认是 10%。每个 broker 允许的不平衡的 leader 的比率。如果每个 broker 超过了这个值，控制器 会触发 leader 的平衡。
leader.imbalance.check.interval.seconds	默认值 300 秒。检查 leader 负载是否平衡的间隔时间。

3.7 自动创建主题

如果 broker 端配置参数 auto.create.topics.enable 设置为 true（默认值是 true），那么当生 产者向一个未创建的主题发送消息时，会自动创建一个分区数为 num.partitions（默认值为 1）、副本因子为 default.replication.factor（默认值为 1）的主题。除此之外，当一个消费者 开始从未知主题中读取消息时，或者当任意一个客户端向未知主题发送元数据请求时，都会 自动创建一个相应主题。这种创建主题的方式是非预期的，增加了主题管理和维护的难度。 生产环境建议将该参数设置为 false。

1）向一个没有提前创建 five 主题发送数据

[atguigu@hadoop102 kafka]$ ./bin/kafka-console-producer.sh --bootstrap-server hadoop102:9092 --topic five
>hello world
[2022-05-01 18:31:01,633] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {five=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

2）查看 five 主题的详情

[atguigu@hadoop102 kafka]$ ./bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic five
Topic: five	TopicId: 8ptdbaARTCCqEjQSxUVBGw	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: five	Partition: 0	Leader: 1	Replicas: 1	Isr: 1