Way to Setup Kafka Cluster: Apache Kafka(转发)
原文:https://data-flair.training/blogs/kafka-cluster/
1. Objective
Today, in this Kafka article, we will see Kafka Cluster Setup.
This Kafka Cluster tutorial provide us some simple steps to setup Kafka Cluster. In simple words, for high availability of the Kafka service, we need to setup Kafka in cluster mode.
So, in this Kafka Cluster document, we will learn Kafka multi-node cluster setup and Kafka multi-broker cluster setup.
Also, we will see Kafka Zookeeper cluster setup.
So, let’s start Kafka Cluster Setup.
2. Kafka Cluster Setup
In order to gain better reliability and high availability of the Kafka service, we need to setup Kafka in cluster mode. At very first:
- From Apache’s site, download Kafka. Also, extract the zip file.
- Further, make two copies of the extracted folder, and then add the suffix _1, _2, _3 to these folders name. Hence, you will have the folders kafka_2.11-1.1.0_1, kafka_2.11-1.1.0_2, kafka_2.11-1.1.0_3, if our extracted folder name was kafka_2.11-1.1.0.
- Go to the kafka_2.11-1.1.0_1 folder.
Let’s discuss Apache Kafka Connect – A Complete Guide 2018
3. Steps to Setup Kafka Cluster
Now, follow several steps to set up Kafka Cluster:
- Make a folder of name “logs”. In this folder, all the Kafka logs will be stored.
- Then, open the server.properties file, on going to the config directory. Here, we will find the file, which contains Kafka broker configurations.
- Further, set broker.id to 1. Make sure it is the id of the broker in a Kafka Cluster, so for each broker, it must be unique.
- Then, uncomment the listener’s configuration and also set it to PLAINTEXT://localhost:9091. It says, for connection requests, the Kafka broker will be listening on port 9091.
- Moreover, with the logs folder path, set the log.dirs configuration that we created in step 1.
- Also, set the Apache Zookeeper address, in the zookeeper.connect configuration. However, if Zookeeper is running in a Kafka cluster, then ensure to give the address as a comma-separated list, i.e.:localhost:2181, localhost:2182.
Basically, these are some general configurations that we need to be set up for the development environment.
In this way, our first Kafka broker configuration is ready.
Now, follow the same steps with the following changes, for the other two folders or brokers.
Let’s revise role of Zookeeper in Apache Kafka
- Now, change broker.id to 2 and 3, In step 3, respectively.
- And, also change the ports used to 9092 and 9093, respectively, in step 4.Note: It is possible to provide any port number, which is available.
Therefore, for all brokers, our configuration is ready.
Now, run the command ./bin/kafka-server-start.sh config/server.properties, on going to the home directory of each Kafka folder.
- Execute the command (all as one line):
- ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 50 --topic demo
Here with a replication factor of three for each partition, 50 partitions are created. On defining a replication factor of three, there will be one leader and two followers, for a partition. Also, at the time when message or record is sent to the leader, it is copied in followers.
- Execute this command:
- ./bin/kafka-topics.sh --describe --topic Hello-Kafka --zookeeper localhost:2181
It helps us to know that which broker is the leader or follower for which partition.
- Output:
Topic:demoPartitionCount:50ReplicationFactor:3Configs:
Topic: demoPartition: 0Leader: 2Replicas: 2,3,1Isr: 2,3,1
Topic: demoPartition: 1Leader: 3Replicas: 3,1,2Isr: 3,1,2
Topic: demoPartition: 2Leader: 1Replicas: 1,2,3Isr: 1,2,3
Topic: demoPartition: 3Leader: 2Replicas: 2,1,3Isr: 2,1,3
Topic: demoPartition: 4Leader: 3Replicas: 3,2,1Isr: 3,2,1
Topic: demoPartition: 5Leader: 1Replicas: 1,3,2Isr: 1,3,2
Topic: demoPartition: 6Leader: 2Replicas: 2,3,1Isr: 2,3,1
…………………………………………………………
…………………………………………………………
…………………………………………………………
Now, we can see Broker 2 is the leader, for Partition 0 and Broker 3 is the leader, for partition 1. And, here ISR refers to in sync replicas.
Have a look at Apache Kafka Terminologies and Concepts
So, this was all about Kafka Cluster. Hope you like our explanation
4. Conclusion
Hence, in this Kafka Cluster Setup tutorial, we have learned the Kafka Cluster Setup of three brokers.
Moreover, we discussed Kafka multi-node setup & Kafka – Zookeeper setup.
Still, if any doubt regarding Kafka Cluster Setup, ask in the comment tab.
See also – How to Create Kafka Clients
For reference
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 没有源码,如何修改代码逻辑?
· 一个奇形怪状的面试题:Bean中的CHM要不要加volatile?
· [.NET]调用本地 Deepseek 模型
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· .NET Core 托管堆内存泄露/CPU异常的常见思路
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· DeepSeek R1 简明指南:架构、训练、本地部署及硬件要求
· 没有源码,如何修改代码逻辑?
· NetPad:一个.NET开源、跨平台的C#编辑器
· 面试官:你是如何进行SQL调优的?
2019-08-21 zookeeper——官网介绍3
2017-08-21 NLog汇总
2017-08-21 汉字的相关处理
2016-08-21 数据结构