kafka-安装配置
## 1. 软件下载
下载地址(可以选用国内镜像地址)
[kafka download](http://kafka.apache.org/downloads.html)
[zookeeper download](https://zookeeper.apache.org/releases.html)
[jdk8](https://www.oracle.com/java/technologies/javase-jdk8-downloads.html)
## 2. 软件安装
### 2.1 安装zookeeper
Kafka内带zookeeper，但是建议独立安装zookeeper。这里使用版本为apache-zookeeper-3.6.0-bin.tar.gz。
#### 2.1.1 安装配置-单机
``` shell
# 安装zookeeper
tar -xf apache-zookeeper-3.6.0-bin.tar.gz -C /ups/app/elastic/
ln -s apache-zookeeper-3.6.0-bin zookeeper
cd /ups/app/elastic/zookeeper
mkdir -p /ups/app/elastic/zookeeper/data
# 配置参数文件
cp conf/zoo_sample.cfg conf/zoo.cfg
sed -ri 's|^(dataDir=).*|\1/ups/app/elastic/zookeeper/data|' conf/zoo.cfg
# 启动zookeeper服务
nohup sh ./bin/zkServer.sh start >/dev/null 2>&1 &
# 检查状态
sh ./bin/zkServer.sh status
# 测试连接
./bin/zkCli.sh -server localhost:2181
```
#### 2.1.2 集群环境配置
``` shell
# 1. 在单机配置环境的基础上添加以下选项：
cat >> conf/zoo.cfg << -EOF
# 配置Zookeeper集群信息
# server.[服务器编号]=[服务器地址]:[LF通信端口]:[选举端口]
# 服务器编号：必须与data/myid文件中的id一致
# LF通信端口: 服务器与集群中的leader交换信息的端口，一般选用相同的端口
# 选举端口: 选举新leader时服务器间相互通信的端口，一般选用相同的端口
server.1=192.168.10.100:2888:3888
server.2=192.168.10.101:2888:3888
server.3=192.168.10.102:2888:3888
EOF
# 2. 创建mydi文件（注意：对应关系）
# 192.168.10.100
echo "1" > data/myid
# 192.168.10.101
echo "2" > data/myid
# 192.168.10.102
echo "3" > data/myid
# 依次启动服务
nohup sh ./bin/zkServer.sh start >/dev/null 2>&1 &
# 检查状态
sh ./bin/zkServer.sh status
```
### 2.2 安装(kafka)
此处使用版本为kafka_2.12-2.0.0.tgz
#### 2.2.1 安装配置
##### 2.2.1.1 单机环境
```shell
# 解压软件
tar -xf /ups/soft/Elastic6.8.0/kafka_2.12-2.0.0.tgz -C /ups/app/elastic/
ln -s kafka_2.12-2.0.0 kafka
# 修改配置文件
cd /ups/app/elastic/config
vi server.properties
broker.id=1
listeners=PLAINTEXT://192.168.10.151:9092
advertised.listeners=PLAINTEXT://localhost:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/ups/app/elastic/kafka/data/kafkaDir
num.partitions=5
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
auto.create.topics.enable=true
delete.topic.enable=true
# 在 kafka中启动自带zookeeper服务
# ./bin/zookeeper-server-start.sh config/zookeeper.properties
# 启动kafka服务
nohup ./bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
# 检查服务
ss -tnlp|grep 9092; jps
```
##### 2.2.1.2 集群环境
``` shell
# 在单机的环境下添加或修改配置 config/server.properties
broker.id=1 # 每台服务器的值都不相同
listeners=PLAINTEXT://192.168.10.100:9092
host.name=192.168.10.100
# 设置zookeeper的连接端口
zookeeper.connect=192.168.10.100:2181,192.168.10.101:2181,192.168.10.102:2181
# 依次启动kafka服务
cd /ups/app/elastic/kafka
nohup ./bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
# 检查服务
ss -tnlp|grep 9092; jps
```
### 2.3 配置环境变量
```
cat> /etc/profile.d/kafka.sh <<-EOF
#Kafka
export KAFKA_HOME=/ups/app/elastic/kafka
export PATH=\${PATH}:\${KAFKA_HOME}/bin
EOF
cat> /etc/profile.d/zookeeper.sh <<-EOF
#Kafka
export ZOOKEEPER_HOME=/ups/app/elastic/zookeeper
export PATH=\${PATH}:\${ZOOKEEPER_HOME}/bin
EOF
```
## 3. 验证kafka服务是否正常
### 3.1 topic操作
#### 3.1.1 创建的topic
```shell
cd /ups/app/elastic/kafka
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
```
#### 3.1.2 查看已经创建的topic信息
```
./bin/kafka-topics.sh --list --zookeeper localhost:2181
```
#### 3.1.3 查看topic副本信息
```shell
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
```
#### 3.1.4 删除topic
``` shell
./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
```
### 3.2 生产者发送消息
#### 3.2.1 使用方法
``` shell
[root@appServer kafka]# bin/kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option Description
------ -----------
--batch-size <Integer: size> Number of messages to send in a single
batch if they are not being sent
synchronously. (default: 200)
--broker-list <String: broker-list> REQUIRED: The broker list string in
the form HOST1:PORT1,HOST2:PORT2.
--compression-codec [String: The compression codec: either 'none',
compression-codec] 'gzip', 'snappy', or 'lz4'.If
specified without value, then it
defaults to 'gzip'
--line-reader <String: reader_class> The class name of the class to use for
reading lines from standard in. By
default each line is read as a
separate message. (default: kafka.
tools.
ConsoleProducer$LineMessageReader)
--max-block-ms <Long: max block on The max time that the producer will
send> block for during a send request
(default: 60000)
--max-memory-bytes <Long: total memory The total memory used by the producer
in bytes> to buffer records waiting to be sent
to the server. (default: 33554432)
--max-partition-memory-bytes <Long: The buffer size allocated for a
memory in bytes per partition> partition. When records are received
which are smaller than this size the
producer will attempt to
optimistically group them together
until this size is reached.
(default: 16384)
--message-send-max-retries <Integer> Brokers can fail receiving the message
for multiple reasons, and being
unavailable transiently is just one
of them. This property specifies the
number of retires before the
producer give up and drop this
message. (default: 3)
--metadata-expiry-ms <Long: metadata The period of time in milliseconds
expiration interval> after which we force a refresh of
metadata even if we haven't seen any
leadership changes. (default: 300000)
--producer-property <String: A mechanism to pass user-defined
producer_prop> properties in the form key=value to
the producer.
--producer.config <String: config file> Producer config properties file. Note
that [producer-property] takes
precedence over this config.
--property <String: prop> A mechanism to pass user-defined
properties in the form key=value to
the message reader. This allows
custom configuration for a user-
defined message reader.
--request-required-acks <String: The required acks of the producer
request required acks> requests (default: 1)
--request-timeout-ms <Integer: request The ack timeout of the producer
timeout ms> requests. Value must be non-negative
and non-zero (default: 1500)
--retry-backoff-ms <Integer> Before each retry, the producer
refreshes the metadata of relevant
topics. Since leader election takes
a bit of time, this property
specifies the amount of time that
the producer waits before refreshing
the metadata. (default: 100)
--socket-buffer-size <Integer: size> The size of the tcp RECV size.
(default: 102400)
--sync If set message send requests to the
brokers are synchronously, one at a
time as they arrive.
--timeout <Integer: timeout_ms> If set and the producer is running in
asynchronous mode, this gives the
maximum amount of time a message
will queue awaiting sufficient batch
size. The value is given in ms.
(default: 1000)
--topic <String: topic> REQUIRED: The topic id to produce
messages to.
[root@appServer kafka]#
```
```shell
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
```
### 3.3 消费者消费消息
#### 3.3.1 使用方法
``` shell
[root@appServer kafka]# ./bin/kafka-console-consumer.sh
The console consumer is a tool that reads data from Kafka and outputs it to standard output.
Option Description
------ -----------
--bootstrap-server <String: server to REQUIRED: The server(s) to connect to.
connect to>
--consumer-property <String: A mechanism to pass user-defined
consumer_prop> properties in the form key=value to
the consumer.
--consumer.config <String: config file> Consumer config properties file. Note
that [consumer-property] takes
precedence over this config.
--enable-systest-events Log lifecycle events of the consumer
in addition to logging consumed
messages. (This is specific for
system tests.)
--formatter <String: class> The name of a class to use for
formatting kafka messages for
display. (default: kafka.tools.
DefaultMessageFormatter)
--from-beginning If the consumer does not already have
an established offset to consume
from, start with the earliest
message present in the log rather
than the latest message.
--group <String: consumer group id> The consumer group id of the consumer.
--isolation-level <String> Set to read_committed in order to
filter out transactional messages
which are not committed. Set to
read_uncommittedto read all
messages. (default: read_uncommitted)
--key-deserializer <String:
deserializer for key>
--max-messages <Integer: num_messages> The maximum number of messages to
consume before exiting. If not set,
consumption is continual.
--offset <String: consume offset> The offset id to consume from (a non-
negative number), or 'earliest'
which means from beginning, or
'latest' which means from end
(default: latest)
--partition <Integer: partition> The partition to consume from.
Consumption starts from the end of
the partition unless '--offset' is
specified.
--property <String: prop> The properties to initialize the
message formatter. Default
properties include:
print.timestamp=true|false
print.key=true|false
print.value=true|false
key.separator=<key.separator>
line.separator=<line.separator>
key.deserializer=<key.deserializer>
value.deserializer=<value.
deserializer>
Users can also pass in customized
properties for their formatter; more
specifically, users can pass in
properties keyed with 'key.
deserializer.' and 'value.
deserializer.' prefixes to configure
their deserializers.
--skip-message-on-error If there is an error when processing a
message, skip it instead of halt.
--timeout-ms <Integer: timeout_ms> If specified, exit if no message is
available for consumption for the
specified interval.
--topic <String: topic> The topic id to consume on.
--value-deserializer <String:
deserializer for values>
--whitelist <String: whitelist> Whitelist of topics to include for
consumption.
[root@appServer kafka]#
```
```shell
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test
# 实时消费数据
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
# 旧版本消费数据
bin/kafka-console-consumer.sh --zookeeper localhost:2181/kafka1.0 --topic test [--from-beginning]
```
==注意：==
0.8以前老版本的kafka，消费的进度(offset)是写在zk中的，所以consumer需要知道zk的地址。后来的版本都统一由broker管理，所以就用bootstrap-server
