Kafka集群数据同步 MirrorMaker详解

1、什么是MirrorMaker？

MirrorMaker是Kafka附带的一个用于在Kafka集群之间制作镜像数据的工具。该工具主要作用是从源集群中消费并生产到目标集群。

一个集群可以启动多个MirrorMaker配置到多个Kafka集群；

2、如果想在Kafka测试集群，同步线的Kafka数据，做测试使用

1、在测试集群配置mirror-consumer.properties文件示例，配置①消费者组和②消费位置以及③线上的Kafka集群地址

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see org.apache.kafka.clients.consumer.ConsumerConfig for more details

# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
bootstrap.servers=kafka0:9092,kafka1:kafka2:9092

# consumer group id
group.id=mirror-consumer-group

# What to do when there is no initial offset in Kafka or if the current
# offset does not exist any more on the server: latest, earliest, none
auto.offset.reset=earliest

　

2、在测试集群配置mirror-producer.properties文件示例，配置线下集群的测试地址和压缩方式

############################# Producer Basics #############################

# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
bootstrap.servers=192.168.1.200:9092,192.168.1.201:9092,192.168.1.203:9092

# specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd
compression.type=none

# name of the partitioner class for partitioning events; default partition spreads data randomly
#partitioner.class=

# the maximum amount of time the client will wait for the response of a request
#request.timeout.ms=

# how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for
#max.block.ms=

# the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together
#linger.ms=

# the maximum size of a request in bytes
#max.request.size=

# the default batch size in bytes when batching multiple records sent to a partition
#batch.size=

# the total bytes of memory the producer can use to buffer records waiting to be sent to the server
#buffer.memory=

　　

3、配置启动脚本start-mirror-maker.sh，和要同步的kafka数据；

/home/pirate/programs/kafka/bin/kafka-mirror-maker.sh --consumer.config /home/pirate/programs/kafka/config/mirror-consumer.properties --producer.config /home/pirate/programs/kafka/config/mirror-producer.properties --whitelist 'dwm_instance|slgjp-udplog-instance' &

　　

参考：Kafka 集群数据备份 MirrorMaker 详解

拓展：kafka 异步双活方案 mirror maker2 深度解析