|NO.Z.00054|——————————|BigDataEnd|——|Hadoop&kafka.V39|——|kafka.v39|分区重新分配.v03|

一、自定义分配分区：分区重新分配实例

### --- 查看主题

~~~     # 把hadoop02上tp_re_01的partition3/4分区重新分配到hadoop02上
[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--describe --topic tp_re_01
~~~输出参数
Topic:tp_re_01  PartitionCount:5    ReplicationFactor:1 Configs:
    Topic: tp_re_01 Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 2    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 3    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 4    Leader: 0   Replicas: 0 Isr: 0

### --- 现在我们在现有集群的基础上再添加一个Kafka节点，

~~~     # 然后使用Kafka自带的kafkareassign-partitions.sh 工具来重新分布分区。
~~~     # 该工具有三种使用模式：
~~~     generate模式，给定需要重新分配的Topic，自动生成reassign plan（并不执行） 
~~~     execute模式，根据指定的reassign plan重新分配Partition 
~~~     verify模式，验证重新分配Partition是否成功

二、将分区3和4重新分布到broker1上，

### --- 借助kafka-reassign-partitions.sh工具生成reassignplan，
~~~     不过我们先得按照要求定义一个文件，
~~~     里面说明哪些topic需要重新分区，文件内容如下：

[root@node1 ~]# cat topics-to-move.json
{
    "topics": [
      {
        "topic":"tp_re_01"
      }
    ],
    "version":1
}

### --- 然后使用kafka-reassign-partitions.sh 工具生成reassign plan

~~~     # 使用该脚本生成一个分区重新分配的计划
[root@hadoop01 ~]# kafka-reassign-partitions.sh --zookeeper localhost:2181/myKafka \
--topics-to-move-json-file topics-to-move.json --broker-list "0,1" \
--generate 
~~~参数说明
--topics-to-move-json-file topics-to-move.json        # 指定使用的脚本文件
--broker-list "0,1"                                   # 指定brokerid，若是有多个节点可以写多个
~~~输出参数:输出json串
 Current partition replica assignment                 # 当前分区副本分配
{"version":1,"partitions":[{"topic":"tp_re_01","partition":4,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":1,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":2,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":3,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":0,"replicas":
[0],"log_dirs":["any"]}]}
        
Proposed partition reassignment configuration         # 建议分区重新分配配置
{"version":1,"partitions":[{"topic":"tp_re_01","partition":4,"replicas":
[1],"log_dirs":["any"]},{"topic":"tp_re_01","partition":1,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":2,"replicas":
[1],"log_dirs":["any"]},{"topic":"tp_re_01","partition":3,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":0,"replicas":
[1],"log_dirs":["any"]}]}

~~~     # 可以使用json格式化工具查看格式化结果

https://www.bejson.com/json/format/

三、根据reassign工具推荐的方案重新分配分区

### --- Proposed partition reassignment configuration下面生成的就是将分区重新分布到broker 1上的结果。

~~~     我们将这些内容保存到名为result.json文件里面
~~~     (文件名不重要，文件格式也不一定要以json为结尾，只要保证内容是json即可)，
~~~     然后执行这些reassign plan：

[root@hadoop01 ~]# vim topics-to-execute.json
~~~将上述输出的json内容拷贝到该文件下
{
    "version": 1,
    "partitions": [{
        "topic": "tp_re_01",
        "partition": 4,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "tp_re_01",
        "partition": 1,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "tp_re_01",
        "partition": 2,
        "replicas": [1],
        "log_dirs": ["any"]
    }, {
        "topic": "tp_re_01",
        "partition": 3,
        "replicas": [0],
        "log_dirs": ["any"]
    }, {
        "topic": "tp_re_01",
        "partition": 0,
        "replicas": [1],
        "log_dirs": ["any"]
    }]
}

### --- 执行计划：

[root@hadoop01 ~]# kafka-reassign-partitions.sh --zookeeper localhost:2181/myKafka \
--reassignment-json-file topics-to-execute.json --execute
~~~输出参数
Current partition replica assignment

{"version":1,"partitions":[{"topic":"tp_re_01","partition":4,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":1,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":2,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":3,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":0,"replicas":
[0],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

### --- 这样Kafka就在执行reassign plan，我们可以校验reassign plan是否执行完成：
~~~     is still in progress显示仍在重新分配中，等待一会会分配完成

[root@hadoop01 ~]# kafka-reassign-partitions.sh --zookeeper localhost:2181/myKafka \
--reassignment-json-file topics-to-execute.json --verify
~~~输出参数
Status of partition reassignment: 
Reassignment of partition tp_re_01-1 completed successfully
Reassignment of partition tp_re_01-4 completed successfully
Reassignment of partition tp_re_01-2 completed successfully
Reassignment of partition tp_re_01-3 completed successfully
Reassignment of partition tp_re_01-0 completed successfully

### --- 查看主题的细节：

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --describe --topic tp_re_01
Topic:tp_re_01  PartitionCount:5    ReplicationFactor:1 Configs:
~~~输出参数
    Topic: tp_re_01 Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: tp_re_01 Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 2    Leader: 1   Replicas: 1 Isr: 1
    Topic: tp_re_01 Partition: 3    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 4    Leader: 1   Replicas: 1 Isr: 1

~~~     # 查看集群主题数据保存位置

[root@hadoop01 ~]# ls /opt/yanqi/servers/kafka/kafka-logs/ 
tp_re_01-1 tp_re_01-3 
[root@hadoop02 ~]# ls /opt/yanqi/servers/kafka/kafka-logs/
tp_re_01-0  tp_re_01-2  tp_re_01-4

~~~     分区的分布的确和操作之前不一样了，broker 1上已经有分区分布上去了。
~~~     使用kafka-reassignpartitions.sh 工具生成的reassign plan只是一个建议，方便大家而已。
~~~     其实我们自己完全可以编辑一个reassign plan，然后执行它，如下：

四、通过自己定义个reassign plan来重新分配分区：将分区3和4重新分布到broker1上，

### --- 定义reassignpaln.json文件

[root@hadoop01 ~]# vim my-topics-to-execute.json
{
        "version": 1,
        "partitions": [{
                "topic": "tp_re_01",
                "partition": 4,
                "replicas": [1],
                "log_dirs": ["any"]
        }, {
                "topic": "tp_re_01",
                "partition": 1,
                "replicas": [0],
                "log_dirs": ["any"]
        }, {
                "topic": "tp_re_01",
                "partition": 2,
                "replicas": [0],
                "log_dirs": ["any"]
        }, {
                "topic": "tp_re_01",
                "partition": 3,
                "replicas": [1],
                "log_dirs": ["any"]
        }, {
                "topic": "tp_re_01",
                "partition": 0,
                "replicas": [0],
                "log_dirs": ["any"]
        }]
}

### --- 将上面的json数据文件保存到my-topics-to-execute.json文件中，然后也是执行它：

~~~     # 把hadoop02上tp_re_01的partition3/4分区重新分配到hadoop02上
[root@hadoop01 ~]#  kafka-reassign-partitions.sh --zookeeper localhost:2181/myKafka \
--reassignment-json-file my-topics-to-execute.json --execute
~~~输出参数
Current partition replica assignment

{"version":1,"partitions":[{"topic":"tp_re_01","partition":4,"replicas":
[1],"log_dirs":["any"]},{"topic":"tp_re_01","partition":1,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":2,"replicas":
[1],"log_dirs":["any"]},{"topic":"tp_re_01","partition":3,"replicas":
[0],"log_dirs":["any"]},{"topic":"tp_re_01","partition":0,"replicas":
[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

### --- 校验reassign plan是否执行完成

[root@hadoop01 ~]# kafka-reassign-partitions.sh --zookeeper localhost:2181/myKafka \
--reassignment-json-file my-topics-to-execute.json --verify
~~~输出参数
Status of partition reassignment: 
Reassignment of partition tp_re_01-1 completed successfully
Reassignment of partition tp_re_01-4 completed successfully
Reassignment of partition tp_re_01-2 completed successfully
Reassignment of partition tp_re_01-3 completed successfully
Reassignment of partition tp_re_01-0 completed successfully

### --- 等这个reassign plan执行完，我们再来看看分区的分布：

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--describe --topic tp_re_01
~~~输出参数
Topic:tp_re_01  PartitionCount:5    ReplicationFactor:1 Configs:
    Topic: tp_re_01 Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 2    Leader: 0   Replicas: 0 Isr: 0
    Topic: tp_re_01 Partition: 3    Leader: 1   Replicas: 1 Isr: 1
    Topic: tp_re_01 Partition: 4    Leader: 1   Replicas: 1 Isr: 1

~~~     # 查看集群主题数据保存位置

[root@hadoop01 ~]# ls /opt/yanqi/servers/kafka/kafka-logs/ 
tp_re_01-0 tp_re_01-1 tp_re_01-2
[root@hadoop02 ~]# ls /opt/yanqi/servers/kafka/kafka-logs/
tp_re_01-3  tp_re_01-4

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---