packetbeat发送数据到pulsar

默认情况下，packetbeat不支持将数据发送到pulsar，因此使用了streamnative开发的kop（kafka on pulsar）来解决这个问题，以下是单机版安装步骤。

一、安装pulsar与kop

1. 下载并安装

wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/apache-pulsar-2.9.3-bin.tar.gz
tar -zxf apache-pulsar-2.9.3-bin.tar.gz
cd apache-pulsar-2.9.3/
mkdir connectors
cd connectors
wget https://github.91chi.fun/https://github.com//streamnative/kop/releases/download/v2.9.3.7/pulsar-protocol-handler-kafka-2.9.3.7.nar #下载kop，并放到特定目录下。kop版本要与pulsar版本保持一致。
cd ..

2. 修改pulsar配置

vim conf/standalone.conf （集群版就修改conf/broker.conf）

# 加入以下配置
messagingProtocols=kafka
protocolHandlerDirectory=/home/apache-pulsar-2.9.3/connectors
allowAutoTopicCreationType=partitioned  # 此处默认为non-partitioned，修改为partitioned
# By default, allowAutoTopicCreationType is set to non-partitioned. Since topics are partitioned by default in Kafka,
# it's better to avoid creating non-partitioned topics for Kafka clients unless Kafka clients need to interact
# with existing non-partitioned topics.

kafkaListeners=PLAINTEXT://127.0.0.1:9092
kafkaAdvertisedListeners=PLAINTEXT://127.0.0.1:9092
brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
brokerDeleteInactiveTopicsEnabled=false    #默认为true，此处修改为false

3. 启动pulsar

bin/pulsar standalone

此时，pulsar就可以接收kafka协议的数据。

二、配置packetbeat将数据发送到kafka

vim packetbeat/packetbeat.yml

# -------------------------------- Kafka Output --------------------------------
output.kafka:
#   Boolean flag to enable or disable the output module.
   enabled: true
   hosts: ["localhost:9092"]
   topic: dns_topic
   version: 2.0.0   # 注意版本指定为2.0.0
   codec.json:
     pretty: false

此时，启动packetbeat后，数据就可以发送到pulsar了，可以通过pulsar消费者消费数据了