|NO.Z.00055|——————————|BigDataEnd|——|Hadoop&kafka.V40|——|kafka.v40|自动再分配|

一、自动再均衡

### --- [kafka高级特性解析]

~~~     [自动再均衡] 
~~~     [broker宕机分区自动分配]
~~~     [broker宕机启动主题分区恢复初始状态]

### --- 自动再均衡

~~~     我们可以在新建主题的时候，手动指定主题各个Leader分区以及Follower分区的分配情况，
~~~     即什么分区副本在哪个broker节点上。
~~~     随着系统的运行，broker的宕机重启，会引发Leader分区和Follower分区的角色转换，
~~~     最后可能Leader大部分都集中在少数几台broker上，由于Leader负责客户端的读写操作，
~~~     此时集中Leader分区的少数几台服务器的网络I/O，CPU，以及内存都会很紧张。
~~~     Leader和Follower的角色转换会引起Leader副本在集群中分布的不均衡，此时我们需要一种手段，
~~~     让Leader的分布重新恢复到一个均衡的状态。

### --- 执行脚本：

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--create --topic tp_demo_03 --replica-assignment "0:1,1:0,0:1"

### --- 上述脚本执行的结果是：

~~~     创建了主题tp_demo_03，有三个分区，每个分区两个副本，
~~~     Leader副本在列表中第一个指定的brokerId上，Follower副本在随后指定的brokerId上。

~~~     # 查看创建的主题

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --list
tp_demo_03

~~~     # 查看主题的详细信息

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--describe --topic tp_demo_03
~~~输出参数
Topic:tp_demo_03    PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: tp_demo_03   Partition: 0    Leader: 0   Replicas: 0,1   Isr: 0,1
    Topic: tp_demo_03   Partition: 1    Leader: 1   Replicas: 1,0   Isr: 1,0
    Topic: tp_demo_03   Partition: 2    Leader: 0   Replicas: 0,1   Isr: 0,1

二、模拟broker宕机，查看分区自动再分配

### --- 然后模拟broker0宕机的情况：

~~~     # 通过jps找到Kafka进程PID
[root@hadoop01 ~]# jps
19576 QuorumPeerMain
34844 Jps
19869 Kafka

~~~     # 直接杀死进程

[root@hadoop01 ~]# kill -9 19869
[root@hadoop01 ~]# jps
19576 QuorumPeerMain
34856 Jps

~~~     # 查看主题分区信息：

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--describe --topic tp_demo_03
~~~输出参数
Topic:tp_demo_03    PartitionCount:3    ReplicationFactor:2    Configs:
    Topic: tp_demo_03    Partition: 0    Leader: 1    Replicas: 0,1    Isr: 1
    Topic: tp_demo_03    Partition: 1    Leader: 1    Replicas: 1,0    Isr: 1
    Topic: tp_demo_03    Partition: 2    Leader: 1    Replicas: 0,1    Isr: 1

~~~     # 重新启动node1上的Kafka

[root@hadoop01 ~]# kafka-server-start.sh -daemon /opt/yanqi/servers/kafka/config/server.properties
[root@hadoop01 ~]# jps
19576 QuorumPeerMain
35612 Kafka
35679 Jps

### --- 查看主题的分区信息：
~~~     # broker恢复了，但是Leader的分配并没有变化，还是处于Leader切换后的分配情况。

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--describe --topic tp_demo_03
~~~输出参数
Topic:tp_demo_03    PartitionCount:3    ReplicationFactor:2    Configs:
    Topic: tp_demo_03    Partition: 0    Leader: 1    Replicas: 0,1    Isr: 1,0
    Topic: tp_demo_03    Partition: 1    Leader: 1    Replicas: 1,0    Isr: 1,0
    Topic: tp_demo_03    Partition: 2    Leader: 1    Replicas: 0,1    Isr: 1,0

三、模拟broker宕机，分区自动分配；broker启动主题分区恢复到初始状态

### --- 设置节点启动后自动分配主题的初始配置

~~~     模拟broker宕机，分区自动分配，broker启动后主题恢复到初始状态
~~~     是否有一种方式，可以让Kafka自动帮我们进行修改？改为初始的副本分配？
~~~     此时，用到了Kafka提供的自动再均衡脚本： kafka-preferred-replica-election.sh

[root@hadoop01 ~]# kafka-preferred-replica-election.sh 
This tool causes leadership for each partition to be transferred back to the 'preferred replica', it can be used to balance leadership among the servers.
Option                                 Description                           
------                                 -----------                           
--path-to-json-file <String: list of   The JSON file with the list of        
  partitions for which preferred         partitions for which preferred      
  replica leader election needs to be    replica leader election should be   
  triggered>                             done, in the following format -     
                                       {"partitions":                        
                                        [{"topic": "foo", "partition": 1},   
                                         {"topic": "foobar", "partition": 2}]
                                       }                                     
                                       Defaults to all existing partitions   
--zookeeper <String: urls>             REQUIRED: The connection string for   
                                         the zookeeper connection in the form
                                         host:port. Multiple URLS can be     
                                         given to allow fail-over.

### --- 先看介绍：

~~~     该工具会让每个分区的Leader副本分配在合适的位置，
~~~     让Leader分区和Follower分区在服务器之间均衡分配。
~~~     如果该脚本仅指定zookeeper地址，则会对集群中所有的主题进行操作，自动再平衡。

### --- 创建preferred-replicas.json，内容如下：

[root@hadoop01 ~]# vim preferred-replicas.json
{
  "partitions": [
    {
      "topic": "tp_demo_03",
      "partition": 0
    },
    {
      "topic": "tp_demo_03",
      "partition": 1
    },
    {
      "topic": "tp_demo_03",
      "partition": 2
    } 
  ] 
}

### --- 执行操作：

[root@hadoop01 ~]# kafka-preferred-replica-election.sh --zookeeper \
localhost:2181/myKafka --path-to-json-file preferred-replicas.json
~~~输出参数
Created preferred replica election path with 
{"version":1,"partitions":[
{"topic":"tp_demo_03","partition":0},
{"topic":"tp_demo_03","partition":1},
{"topic":"tp_demo_03","partition":2}
]}

Successfully started preferred replica election for partitions Set(tp_demo_03-0, tp_demo_03-1, tp_demo_03-2)

### --- 查看操作的结果：

[root@hadoop01 ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka \
--describe --topic tp_demo_03
~~~输出参数 
Topic:tp_demo_03    PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: tp_demo_03   Partition: 0    Leader: 0   Replicas: 0,1   Isr: 1,0
    Topic: tp_demo_03   Partition: 1    Leader: 1   Replicas: 1,0   Isr: 1,0
    Topic: tp_demo_03   Partition: 2    Leader: 0   Replicas: 0,1   Isr: 1,0

~~~     # 恢复到最初的分配情况。
~~~     # 之所以是这样的分配，是因为我们在创建主题的时候：
 --replica-assignment 1 "0:1,1:0,0:1"

~~~     在逗号分割的每个数值对中排在前面的是Leader分区，后面的是副本分区。
~~~     那么所谓的preferred replica，就是排在前面的数字就是Leader副本应该在的brokerId。

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

——W.S.Landor