Kafka Broker调优

3.1 Broker 核心参数配置

参数名称	描述
replica.lag.time.max.ms	ISR 中，如果 Follower 长时间未向 Leader 发送通信请求或同步数据，则该 Follower 将被踢出 ISR。该时间阈值，默认 30s。
auto.leader.rebalance.enable	默认是 true。自动 Leader Partition 平衡。
leader.imbalance.per.broker.percentage	默认是 10%。每个 broker 允许的不平衡的 leader 的比率。如果每个 broker 超过了这个值，控制器会触发 leader 的平衡。
leader.imbalance.check.interval.seconds	默认值 300 秒。检查 leader 负载是否平衡的间隔时间。
log.segment.bytes	Kafka 中 log 日志是分成一块块存储的，此配置是指 log 日志划分成块的大小，默认值 1G。
log.index.interval.bytes	默认 4kb，kafka 里面每当写入了 4kb 大小的日志（.log），然后就往 index 文件里面记录一个索引。
log.retention.hours	Kafka 中数据保存的时间，默认 7 天。
log.retention.minutes	Kafka 中数据保存的时间，分钟级别，默认关闭。
log.retention.ms	Kafka 中数据保存的时间，毫秒级别，默认关闭。
log.retention.check.interval.ms	检查数据是否保存超时的间隔，默认是 5 分钟。
log.retention.bytes	默认等于-1，表示无穷大。超过设置的所有日志总大小，删除最早的 segment。
log.cleanup.policy	默认是 delete，表示所有数据启用删除策略；如果设置值为 compact，表示所有数据启用压缩策略。
num.io.threads	默认是 8。负责写磁盘的线程数。整个参数值要占总核数的 50%。
num.replica.fetchers	副本拉取线程数，这个参数占总核数的 50%的 1/3
num.network.threads	默认是 3。数据传输线程数，这个参数占总核数的 50%的 2/3 。
log.flush.interval.messages	强制页缓存刷写到磁盘的条数，默认是 long 的最大值，9223372036854775807。一般不建议修改，交给系统自己管理。
log.flush.interval.ms	每隔多久，刷数据到磁盘，默认是 null。一般不建议修改，交给系统自己管理。

3.2 服役新节点/退役旧节点

（1）创建一个要均衡的主题。

[atguigu@hadoop102 scripts]$ vim topics-to-move.json

{
  "topics": [
    {
      "topic": "first"
    }
  ],
  "version": 1
}

（2）生成一个负载均衡的计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[1,2,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[2,0,1],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[2,3,0],"log_dirs":["any","any","any"]}]}

（3）创建副本存储计划（所有副本存储在 broker0、broker1、broker2、broker3 中）。

[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

{
  "version": 1,
  "partitions": [
    {
      "topic": "first",
      "partition": 0,
      "replicas": [
        2,
        3,
        0
      ],
      "log_dirs": [
        "any",
        "any",
        "any"
      ]
    },
    {
      "topic": "first",
      "partition": 1,
      "replicas": [
        3,
        0,
        1
      ],
      "log_dirs": [
        "any",
        "any",
        "any"
      ]
    },
    {
      "topic": "first",
      "partition": 2,
      "replicas": [
        0,
        1,
        2
      ],
      "log_dirs": [
        "any",
        "any",
        "any"
      ]
    }
  ]
}

（4）执行副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[1,2,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[2,0,1],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for first-0,first-1,first-2

（5）验证副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.

Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

3.3 增加分区

1）修改分区数（注意：分区数只能增加，不能减少）

[atguigu@hadoop102 kafka]$ bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --alter --topic first --partitions 3

3.4 增加副本因子

1）创建 topic

[atguigu@hadoop102 kafka]$ ./bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --create --partitions 3 --replication-factor 1 --topic four
Created topic four.

2）手动增加副本存储

（1）创建副本存储计划（所有副本都指定存储在 broker0、broker1、broker2 中）。

[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

输入如下内容：

{
  "version": 1,
  "partitions": [
    {
      "topic": "four",
      "partition": 0,
      "replicas": [
        0,
        1,
        2
      ]
    },
    {
      "topic": "four",
      "partition": 1,
      "replicas": [
        0,
        1,
        2
      ]
    },
    {
      "topic": "four",
      "partition": 2,
      "replicas": [
        0,
        1,
        2
      ]
    }
  ]
}

（2）执行副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"four","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"four","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"four","partition":2,"replicas":[0],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for four-0,four-1,four-2

3.5 手动调整分区副本存储

（1）创建副本存储计划（所有副本都指定存储在 broker0、broker1 中）。

[atguigu@hadoop102 kafka]$ vim increase-replication-factor.json

{
  "version": 1,
  "partitions": [
    {
      "topic": "three",
      "partition": 0,
      "replicas": [
        0,
        1
      ]
    },
    {
      "topic": "three",
      "partition": 1,
      "replicas": [
        0,
        1
      ]
    },
    {
      "topic": "three",
      "partition": 2,
      "replicas": [
        1,
        0
      ]
    },
    {
      "topic": "three",
      "partition": 3,
      "replicas": [
        1,
        0
      ]
    }
  ]
}

（2）执行副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"three","partition":0,"replicas":[3,1],"log_dirs":["any","any"]},{"topic":"three","partition":1,"replicas":[1,0],"log_dirs":["any","any"]},{"topic":"three","partition":2,"replicas":[0,2],"log_dirs":["any","any"]},{"topic":"three","partition":3,"replicas":[2,3],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for three-0,three-1,three-2,three-3

（3）验证副本存储计划。

[atguigu@hadoop102 kafka]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition three-0 is complete.
Reassignment of partition three-1 is complete.
Reassignment of partition three-2 is complete.
Reassignment of partition three-3 is complete.

Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic three

3.6 Leader Partition 负载平衡

参数名称	描述
auto.leader.rebalance.enable	默认是 true。自动 Leader Partition 平衡。生产环境中，leader 重选举的代价比较大，可能会带来性能影响，建议设置为 false 关闭。
leader.imbalance.per.broker.percentage	默认是 10%。每个 broker 允许的不平衡的 leader 的比率。如果每个 broker 超过了这个值，控制器会触发 leader 的平衡。
leader.imbalance.check.interval.seconds	默认值 300 秒。检查 leader 负载是否平衡的间隔时间。

3.7 自动创建主题

如果 broker 端配置参数 auto.create.topics.enable 设置为 true（默认值是 true），那么当生产者向一个未创建的主题发送消息时，会自动创建一个分区数为 num.partitions（默认值为 1）、副本因子为 default.replication.factor（默认值为 1）的主题。除此之外，当一个消费者开始从未知主题中读取消息时，或者当任意一个客户端向未知主题发送元数据请求时，都会自动创建一个相应主题。这种创建主题的方式是非预期的，增加了主题管理和维护的难度。 生产环境建议将该参数设置为 false。

1）向一个没有提前创建 five 主题发送数据

[atguigu@hadoop102 kafka]$ ./bin/kafka-console-producer.sh --bootstrap-server hadoop102:9092 --topic five
>hello world
[2022-05-01 18:31:01,633] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {five=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

2）查看 five 主题的详情

[atguigu@hadoop102 kafka]$ ./bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --describe --topic five
Topic: five	TopicId: 8ptdbaARTCCqEjQSxUVBGw	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: five	Partition: 0	Leader: 1	Replicas: 1	Isr: 1

上一篇Kafka 生产者调优参数

下一篇Kafka 消费者调优

本文作者：我係死肥宅

本文链接：https://www.cnblogs.com/iamfatotaku/p/16213619.html