kafka集群压力测试--基础。

1.生产者测试

kafka-producer-perf-test.bat --num-records 1000000 --topic test --record-size 200 --throughput 100000 --producer-props bootstrap.servers=localhost:9092

--num-records 发送消息的数量

--topic 主题

-- record-size 单条消息的大小字节

--throughput 吞储量阀值 10万就是每秒不超过10万条数据

--producer-props producer的配置，可以写多个配置用逗号隔开

2.消费者测试

kafka-consumer-perf-test.bat --messages 1000000 --threads 1 --zookeeper localhost:2181 --num-fetch-threads 3 --topic test

--messages 消费多少消息

--threads 线程数量

--zookeeper zookeeper的地址

--num-fetch-threads 拉取数据的线程数量即为消费者的数量

测试参数

./kafka-producer-perf-test.sh --num-records 10000000000 --topic test-rep-one --record-size 200 --throughput 1000000000 --producer-props bootstrap.servers=hadoop.23.51fbops.com:9092,hadoop.22.51fbops.com:9092,hadoop.24.51fbops.com:9092

./kafka-consumer-perf-test.sh --messages 100000000 --threads 3 --zookeeper hadoop.22.51fbops.com:2181,hadoop.23.51fbops.com:2181,hadoop.24.51fbops.com:2181 --num-fetch-threads 3 --topic test-rep-one

生产者测试结果 16.39Mb/秒流量，160万/秒消息，无堆积

8274379 records sent, 1654875.8 records/sec (15.78 MB/sec), 0.7 ms avg latency, 14.0 max latency.
8410914 records sent, 1682182.8 records/sec (16.04 MB/sec), 0.7 ms avg latency, 11.0 max latency.
8385566 records sent, 1677113.2 records/sec (15.99 MB/sec), 0.8 ms avg latency, 9.0 max latency.
8293790 records sent, 1658758.0 records/sec (15.82 MB/sec), 0.7 ms avg latency, 11.0 max latency.
7911355 records sent, 1582271.0 records/sec (15.09 MB/sec), 0.8 ms avg latency, 9.0 max latency.
8268134 records sent, 1653626.8 records/sec (15.77 MB/sec), 0.7 ms avg latency, 13.0 max latency.
8198960 records sent, 1639792.0 records/sec (15.64 MB/sec), 0.8 ms avg latency, 15.0 max latency.
8349350 records sent, 1669870.0 records/sec (15.93 MB/sec), 0.7 ms avg latency, 12.0 max latency.
8140649 records sent, 1628129.8 records/sec (15.53 MB/sec), 0.8 ms avg latency, 12.0 max latency.
8592471 records sent, 1718494.2 records/sec (16.39 MB/sec), 0.7 ms avg latency, 9.0 max latency.
8183905 records sent, 1636781.0 records/sec (15.61 MB/sec), 0.8 ms avg latency, 7.0 max latency.

消费者

[root@hadoop.22.51fbops.com bin]# ./kafka-consumer-perf-test.sh --messages 100000 --threads 3 --zookeeper hadoop.22.51fbops.com:2181,hadoop.23.51fbops.com:2181,hadoop.24.51fbops.com:2181 --num-fetch-threads 3 --topic test-rep-one
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2017-12-29 10:52:59:047, 2017-12-29 10:53:10:508, 286.1023, 24.9631, 30000000, 2617572.6376

目前业务需求为9000条/秒。