Kafka迁移与扩容工具用法

１．迁移topic到新增的node上

假如现在一个kafka集群运行三个broker，broker.id依次为101,102,103,后来由于业务数据突然暴增，需要新增三个broker,broker.id依次为104,105,106.目的是要把chatmessage迁移到新增node上。脚本(json格式)如下所示：

 

kafka-reassign-partitions.sh --zookeeper 192.168.1.10:2181 --topics-to-move-json-file  migration-chatmessage-topic.json  --broker-list  "104,105,106"  --generate

 

migration-chatmessage-topic.json文件内容如下：

 

{
    "topics":
        [
            {
                "topic": "chatmessage"
            }
        ],
    "version":1
}

 

生成分配partitions的json脚本：

{
    "version":1,
    "partitions":
        [
            {
                "topic":"chatmessage",
                "partition":10,"replicas":[8]
            },
            {
                "topic":"chatmessage",
                "partition":5,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":3,
                "replicas":[5]
            },
            {
                "topic":"chatmessage", 
                "partition":4, 
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":9,
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":1,
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":11,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":7,
                "replicas":[5]
            },
            {
                "topic":"chatmessage",
                "partition":2,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":0,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":6,
                "replicas":[4]
            },
            {
                "topic":"chatmessage",
                "partition":8,
                "replicas":[4]
            }
        ]
    }
}

重新分配parttions的json脚本如下：migration-topic-chatmessage-topic.json

 

{"version":1,
    "partitions":
        [
            {"topic":"cluster-switch-topic","partition":10,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":5,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":4,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":3,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":9,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":1,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":11,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":7,"replicas":[4]},
            {"topic":"cluster-switch-topic","partition":2,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":0,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":6,"replicas":[5]},
            {"topic":"cluster-switch-topic","partition":8,"replicas":[5]}
        ]
}

 

执行一下脚本

./kafka-reassign-partitions.sh --zookeeper 192.168.1.10:2181  --reassignment-json-file migration-topic-chatmessage-topic.json --execute

 

topic修改(replicats-factor)副本个数

　　假如初始时chatmessage为一个副本，为了提高可用性，需要改为2副本模式。脚本replicas-chatmessage-topic.json文件内容如下:

 

{
        "partitions":
                [
                {
                        "topic": "chatmessage",
                        "partition": 0,
                        "replicas": [101,102,104]
                },
                {
                        "topic": "chatmessage",
                        "partition": 1,
                        "replicas": [102,103,106]
                },
              .......
        "version":1
}

 

编辑好json文件后执行一下命令：

./kafka-reassign-partitions.sh --zookeeper  192.168.1.10:2181 --reassignment-json-file  replicas-chatmessage-topic.json  --execute

3.topic的分区扩容用法

　　先扩容分区数量，脚本如下:

./kafka-topics.sh --zookeeper 192.168.1.10:2181  --alter   --partitions 15   --topic   chatmessage

 

　　把topic为chatmessage的分区数量扩展到１５个。

　　设置topic分区副本：

{
        "partitions":
                [
                {
                        "topic": "chatmessage",
                        "partition": 12,
                        "replicas": [101,102]
                },
                {
                        "topic": "chatmessage",
                        "partition": 13,
                        "replicas": [103,104]
                },
                {
                        "topic": "chatmessage",
                        "partition": 14,
                        "replicas": [105,106]
                }
                ],             
        "version":1
}

 

　　编辑json文件，执行一下脚本：

./bin/kafka-reassign-partitions.sh --zookeeper  192.168.１.10:2181 --reassignment-json-file partitions-extension-chatmessage-topic.json  --execute