Kafka 1.0.0集群安装
Kafka版本介绍见下图:
环境
主机名 | IP 地址 | 安装路径 | 系统 |
sht-sgmhadoopdn-01 | 172.16.101.58 |
/opt/kafka_2.12-1.0.0 /opt/kafka(软连接) |
CentOS Linux release 7.3.1611 (Core) |
sht-sgmhadoopdn-02 | 172.16.101.59 | ||
sht-sgmhadoopdn-03 | 172.16.101.60 |
软件下载
Kafka下载
http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz
JDK下载
https://download.oracle.com/otn/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz
软件安装
一. 安装jdk
# wget https://download.oracle.com/otn/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz # tar -zxf jdk-8u45-linux-x64.tar.gz -C /usr/
二.设置.bash_profile环境变量
JAVA_HOME=/usr/java/jdk1.8.0_45 KAFKA_HOME=/opt/kafka CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib PATH=$KAFKA_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$HOME/bin export PATH
三. 安装kafka
# wget http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz -P /opt # tar -zxf kafka_2.12-1.0.0.tgz # ln -s kafka_2.12-1.0.0 kafka # mkdir /opt/kafka/data
软件配置
一. zookeeper配置
1. 文件zookeeper.properties,各节点一致
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/opt/kafka/data clientPort=2182 server.1=sht-sgmhadoopdn-01:2889:3889 server.2=sht-sgmhadoopdn-02:2889:3889 server.3=sht-sgmhadoopdn-03:2889:3889
2. 各节点分别创建server-id
sht-sgmhadoopdn-01
# echo 1 > /opt/kafka/data/myid
sht-sgmhadoopdn-02
# echo 2 > /opt/kafka/data/myid
sht-sgmhadoopdn-03
# echo 3 > /opt/kafka/data/myid
3.启动zookeeper
# /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
4.查看zookeeper集群状态
# echo stat | nc sht-sgmhadoopdn-01 2182 | grep Mode Mode: follower # echo stat | nc sht-sgmhadoopdn-02 2182 | grep Mode Mode: leader # echo stat | nc sht-sgmhadoopdn-03 2182 | grep Mode Mode: follower
二. kafka配置
1.各节点配置文件server.properties
sht-sgmhadoopdn-01
broker.id=0 listeners=PLAINTEXT://172.16.101.58:9092 advertised.listeners=PLAINTEXT://172.16.101.58:9092 log.dirs=/opt/kafka/data zookeeper.connect=sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182
sht-sgmhadoopdn-02
broker.id=1 listeners=PLAINTEXT://172.16.101.59:9092 advertised.listeners=PLAINTEXT://172.16.101.59:9092 log.dirs=/opt/kafka/data zookeeper.connect=sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182
sht-sgmhadoopdn-03
broker.id=2 listeners=PLAINTEXT://172.16.101.60:9092 advertised.listeners=PLAINTEXT://172.16.101.60:9092 log.dirs=/opt/kafka/data zookeeper.connect=sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182
2. 启动kafka
# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
3. 查看集群状态
# echo dump | nc sht-sgmhadoopdn-01 2182 | grep broker /brokers/ids/0 /brokers/ids/1 /brokers/ids/2
三 测试
1. 创建topic
# kafka-topics.sh --zookeeper sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182 --create --partitions 3 --replication-factor 3 --topic test-topic Created topic "test-topic". # kafka-topics.sh --zookeeper sht-sgmhadoopdn-01:2182,sht-sgmhadoopdn-02:2182,sht-sgmhadoopdn-03:2182 --describe --topic test-topic Topic:test-topic PartitionCount:3 ReplicationFactor:3 Configs: Topic: test-topic Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: test-topic Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: test-topic Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
2. 生产者生产数据
# kafka-console-producer.sh --broker-list sht-sgmhadoopdn-01:9092,sht-sgmhadoopdn-02:9092,sht-sgmhadoopdn-03:9092 --topic test-topic >Hello >This is my fisrt Kafka !
3. 消费者消费数据
# kafka-console-consumer.sh --bootstrap-server bootstrap-servers=sht-sgmhadoopdn-01:9092,sht-sgmhadoopdn-02:9092,sht-sgmhadoopdn-03:9092 --topic test-topic --from-beginning Hello This is my fisrt Kafka !