b0119 Kafka 使用

说明

版本 kafka_2.133-2.80

安装

单机

b0104 大数据集群-2021伪分布式-环境搭建

分布式

b0105 大数据集群-2021分布式A(手动3节点)-环境搭建

 

使用

### 启动
zkServer.sh start

cd /opt/kafka_2.13-2.8.0/bin
nohup ./kafka-server-start.sh ../config/server.properties &


# 创建topic 发送 接收
kafka-topics.sh --bootstrap-server hc2108:9092 --create --topic testKafka
kafka-topics.sh --bootstrap-server hc2108:9092 --list

kafka-console-producer.sh --bootstrap-server hc2108:9092 --topic testKafka
kafka-console-consumer.sh --bootstrap-server hc2108:9092 --topic testKafka

python

安装依赖包

pip install kafka
pip install kafka-python

consumer

#encoding:utf8
from kafka import KafkaConsumer
import uuid


consumer = KafkaConsumer('testKafka',
                         bootstrap_servers=['192.168.1.10:9092'],
                         group_id=uuid.uuid4().hex,
                         auto_offset_reset="latest"
                         )

for message in consumer:
    # print(message.topic)
    # print(type(message))  # <class 'kafka.consumer.fetcher.ConsumerRecord'>
    # print(message)
    # ConsumerRecord(topic='testKafka', partition=0, offset=16, timestamp=1657944799959, timestamp_type=0, key=None, value=b'hh\x08\x08', headers=[], checksum=None, serialized_key_size=-1, serialized_value_size=4, serialized_header_size=-1)

    print("--------------------------")
    print(message.topic)
    print(message.partition)
    print(message.offset)
    print(message.timestamp)

    print(message.headers)
    # key none
    print(message.key)

    # bytes 字节
    print(message.value)

    # 字符串
    print(message.value.decode("utf-8"))

    print(message.serialized_value_size)

    consumer.commit()


consumer.close()

 

producer

# -*- coding: utf-8 -*-

from kafka import KafkaProducer
import json


producer = KafkaProducer(bootstrap_servers=['192.168.1.10:9092'])
 #此处ip可以是多个['0.0.0.1:9092','0.0.0.2:9092','0.0.0.3:9092' ]
# for i in range(3):
#      msg = "msg %d" % i
#      print(msg)
#      # 必须以二进制格式发送
#      bytes_msg = msg.encode("utf-8")
#      producer.send('testKafka', bytes_msg)

# 发送字典
msg_dict = {
     "name":"lucy",
     "age":28
}

msg_str = json.dumps(msg_dict, ensure_ascii=False)
msg_bytes = msg_str.encode("utf-8")
producer.send('testKafka', msg_bytes)

producer.close()

 

ref  kafka-9-python操作kafka及偏移量的处理 , 皮皮冰燃，2020-12-21 14:30:07 

 

Todo

• 长连接， KafkaConsumer,KafkaProducer 里面有很多时间有效的参数，有默认值，用他们连接kafka，如何保持连接长期有效。