Kafka shell ：运维常用命令

一、通过kafka-topics.sh来执行topic相关的命令

1、用法

[root@kafka-node1 kafka]# bin/kafka-topics.sh
Create, delete, describe, or change a topic.
Option                                   Description
------                                   -----------
--alter                                  Alter the number of partitions,
                                           replica assignment, and/or
                                           configuration for the topic.
--config <String: name=value>            A topic configuration override for the
                                           topic being created or altered.The
                                           following is a list of valid
                                           configurations:
                                         	cleanup.policy
                                         	compression.type
                                         	delete.retention.ms
                                         	file.delete.delay.ms
                                         	flush.messages
                                         	flush.ms
                                         	follower.replication.throttled.
                                           replicas
                                         	index.interval.bytes
                                         	leader.replication.throttled.replicas
                                         	max.message.bytes
                                         	message.format.version
                                         	message.timestamp.difference.max.ms
                                         	message.timestamp.type
                                         	min.cleanable.dirty.ratio
                                         	min.compaction.lag.ms
                                         	min.insync.replicas
                                         	preallocate
                                         	retention.bytes
                                         	retention.ms
                                         	segment.bytes
                                         	segment.index.bytes
                                         	segment.jitter.ms
                                         	segment.ms
                                         	unclean.leader.election.enable
                                         See the Kafka documentation for full
                                           details on the topic configs.
--create                                 Create a new topic.
--delete                                 Delete a topic
--delete-config <String: name>           A topic configuration override to be
                                           removed for an existing topic (see
                                           the list of configurations under the
                                           --config option).
--describe                               List details for the given topics.
--disable-rack-aware                     Disable rack aware replica assignment
--force                                  Suppress console prompts
--help                                   Print usage information.
--if-exists                              if set when altering or deleting
                                           topics, the action will only execute
                                           if the topic exists
--if-not-exists                          if set when creating topics, the
                                           action will only execute if the
                                           topic does not already exist
--list                                   List all available topics.
--partitions <Integer: # of partitions>  The number of partitions for the topic
                                           being created or altered (WARNING:
                                           If partitions are increased for a
                                           topic that has a key, the partition
                                           logic or ordering of the messages
                                           will be affected
--replica-assignment <String:            A list of manual partition-to-broker
  broker_id_for_part1_replica1 :           assignments for the topic being
  broker_id_for_part1_replica2 ,           created or altered.
  broker_id_for_part2_replica1 :
  broker_id_for_part2_replica2 , ...>
--replication-factor <Integer:           The replication factor for each
  replication factor>                      partition in the topic being created.
--topic <String: topic>                  The topic to be create, alter or
                                           describe. Can also accept a regular
                                           expression except for --create option
--topics-with-overrides                  if set when describing topics, only
                                           show topics that have overridden
                                           configs
--unavailable-partitions                 if set when describing topics, only
                                           show partitions whose leader is not
                                           available
--under-replicated-partitions            if set when describing topics, only
                                           show under replicated partitions
--zookeeper <String: urls>               REQUIRED: The connection string for
                                           the zookeeper connection in the form
                                           host:port. Multiple URLS can be
                                           given to allow fail-over.

启动zookeeper

使用安装包中的脚本启动单节点Zookeeper实例：

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

启动Kafka服务

使用kafka-server-start.sh启动kafka服务：

bin/kafka-server-start.sh config/server.properties

创建Topic

使用kafka-topics.sh 创建但分区单副本的topic test

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看Topic

bin/kafka-topics.sh --list --zookeeper localhost:2181

产生消息

使用kafka-console-producer.sh 发送消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 

消费消息

使用kafka-console-consumer.sh 接收消息并在终端打印

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

删除Topic

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test

注意事项：当执行删除命令之后。topic不是物理删除，而是一个标记删除的操作

问题1：标记删除之后的主题是否还可以继续生产数据？

不会又影响

注意：当服务器重启就会删除已经标记的topic，这个需要和版本有关系

如果是0.10的话，注意，有如果删除topic的需求，需要在server.partitions文件中加入delete.topic.enable=true

查看Topic的详细信息

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

第一行给出了所有分区的摘要，每个附加行给出了关于一个分区的信息。 由于我们只有一个分区，所以只有一行。

“Leader”: 是负责给定分区的所有读取和写入的节点。 每个节点将成为分区随机选择部分的领导者。

“Replicas”: 是复制此分区日志的节点列表，无论它们是否是领导者，或者即使他们当前处于活动状态。

“Isr”: 是一组“同步”副本。这是复制品列表的子集，当前活着并被引导到领导者。

更改tipic的信息

 更改topic信息、topic名称、zookeeper地址是必须的参数、其他参数还包括了partition个数等

1、修改

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic5 --partitions 40

2、修改增加配置参数

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic4 --config flush.messages=1

3、删除配置参数

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic4 --delete-config flush.messages

二、python调用

1、生产消息

#!/usr/bin/env python
#-*- coding:utf-8 -*-
# Author:ahong luo
from kafka import KafkaProducer
# 创建生产者
producer = KafkaProducer(bootstrap_servers="192.168.118.95")
for _ in range(80000):
    producer.send('test',b'192.168.118.95')
producer.send('test',b' 192.168.118.95')

2、消费消息

#-*- coding:utf-8 -*-
# Author:ahong luo
#https://pypi.org/project/kafka-python/
from kafka import KafkaConsumer

#connect to Kafka server and pass the topic we want to consume
consumer = KafkaConsumer('logs', bootstrap_servers="192.168.118.100:2181")

for msg in consumer:
    print(msg)