kafka

kafka

1.安装

点击kafka_2.11-2.1.0下载
解压
tar -zxvf kafka_2.11-2.1.0.tgz
移动到指定目录
mv kafka_2.11-2.1.0 /usr/local/kafka

创建启动脚本,方便以后操作
启动zookeeper脚本:
vi startZookeeper.sh
输入以下内容:
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper.log 2>&1 &
启动kafka脚本:
vi startKafka.sh
输入以下内容:
nohup bin/kafka-server-start.sh config/server.properties > kafka.log 2>&1 &
赋可执行权限:
chmod 755 start*

启动zookeeper:
./startZookeeper.sh
检查是否启动成功:
ps -ef | grep zookeeper
启动kafka:
./startKafka.sh
检查是否启动成功:
ps -ef | grep kafka

2.常用命令

列出所有topic:
bin/kafka-topics.sh --list --zookeeper localhost:2181

创建topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

运行消费者:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test

运行生产者:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
在生产者中输入数据,对应可以在消费者中看到相同的数据

3.设置集群

到目前为止,我们都是在和单个broker打交道.对kafka来说,单个broker只是一个大小为1的集群,所以除了启动几个broker示例外,没有什么变化.为了体验一下,我们将集群扩展到3个节点(仍然都在本地机器上).

cp config/server.properties config/server-1.properties   
cp config/server.properties config/server-2.properties

vi config/server-1.properties
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1

vi config/server-2.properties
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2

broker.id属性是集群中每个节点的唯一且永久的名称.因为我们在同一台机器上运行这些节点,并且要防止broker在相同的端口上注册或重写彼此的数据,所以我们要覆盖端口和日志目录.

由于之前已经启动了zookeeper和单节点,现在只需要在启动这2个新节点

nohup bin/kafka-server-start.sh config/server-1.properties > kafka-1.log 2>&1 &
nohup bin/kafka-server-start.sh config/server-2.properties > kafka-2.log 2>&1 &

现在创建一个有3个replication factor的新topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

现在有了一个集群,我们怎么知道哪个broker正在做什么呢?可以运行describe topics命令来查看

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

输出:

Topic:my-replicated-topic       PartitionCount:1        ReplicationFactor:3     Configs:
    Topic: my-replicated-topic      Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2

解释:
第一行是所有分区partition的摘要
之后的每一行是每个分区的信息
因为这个topic只有1个partition,所以只有一行

• "leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.

• "replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.

• "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.