Kafka 总体调优
5.1 如何提升吞吐量
如何提升吞吐量？
1）提升生产吞吐量
- buffer.memory：发送消息的缓冲区大小，默认值是 32m，可以增加到 64m。
- batch.size：默认是 16k。如果 batch 设置太小，会导致频繁网络请求，吞吐量下降； 如果 batch 太大，会导致一条消息需要等待很久才能被发送出去，增加网络延时。
- linger.ms，这个值默认是 0，意思就是消息必须立即被发送。一般设置一个 5-100 毫秒。如果 linger.ms 设置的太小，会导致频繁网络请求，吞吐量下降；如果 linger.ms 太长， 会导致一条消息需要等待很久才能被发送出去，增加网络延时。
- compression.type：默认是 none，不压缩，但是也可以使用 lz4 压缩，效率还是不 错的，压缩之后可以减小数据量，提升吞吐量，但是会加大 producer 端的 CPU 开销。
2）增加分区
**3）消费者提高吞吐量 **
- 调整 fetch.max.bytes 大小，默认是 50m。
- 调整 max.poll.records 大小，默认是 500 条。
4）增加下游消费者处理能力
5.2 数据精准一次
**1）生产者角度 **
- acks 设置为-1 （acks=-1）。
- 幂等性（enable.idempotence = true） + 事务 。
**2）broker 服务端角度 **
- 分区副本大于等于 2 （--replication-factor 2）。
- ISR 里应答的最小副本数量大于等于 2 （min.insync.replicas = 2）。
3）消费者
- 事务 + 手动提交 offset （enable.auto.commit = false）。
- 消费者输出的目的地必须支持事务（MySQL、Kafka）。
5.3 合理设置分区数
- 创建一个只有 1 个分区的 topic。
- 测试这个 topic 的 producer 吞吐量和 consumer 吞吐量。
- 假设他们的值分别是 Tp 和 Tc，单位可以是 MB/s。
- 然后假设总的目标吞吐量是 Tt，那么分区数 = Tt / min（Tp，Tc）。
例如：producer 吞吐量 = 20m/s；consumer 吞吐量 = 50m/s，期望吞吐量 100m/s；
分区数 = 100 / 20 = 5 分区
分区数一般设置为：3-10 个
分区数不是越多越好，也不是越少越好，需要搭建完集群，进行压测，再灵活调整分区 个数。
5.4 单条日志大于 1m
| 参数名称 | 描述 |
|---|---|
| message.max.bytes | 默认 1m，broker 端接收每个批次消息最大值。 |
| max.request.size | 默认 1m，生产者发往 broker 每个请求消息最大值。针对 topic 级别设置消息体的大小。 |
| replica.fetch.max.bytes | 默认 1m，副本同步数据，每个批次消息最大值。 |
| fetch.max.bytes | 默认 Default: 52428800（50 m）。消费者获取服务器端一批消息最大的字节数。如果服务器端一批次的数据大于该值 （50m）仍然可以拉取回来这批数据，因此，这不是一个绝对 最大值。一批次的大小受 message.max.bytes （broker config） or max.message.bytes （topic config）影响。 |
5.5 服务器挂了
在生产环境中，如果某个 Kafka 节点挂掉。
正常处理办法：
- 先尝试重新启动一下，如果能启动正常，那直接解决。
- 如果重启不行，考虑增加内存、增加 CPU、网络带宽。
- 如果将 kafka 整个节点误删除，如果副本数大于等于 2，可以按照服役新节点的方式重新服役一个新节点，并执行负载均衡。
5.6 集群压力测试
1）Kafka 压测
用 Kafka 官方自带的脚本，对 Kafka 进行压测。
- 生产者压测：kafka-producer-perf-test.sh
- 消费者压测：kafka-consumer-perf-test.sh
2）Kafka Producer 压力测试
（1）创建一个 test topic，设置为 3 个分区 3 个副本
[atguigu@hadoop102 kafka]$ ./bin/kafka-topics.sh --bootstrap-server hadoop102:9092 --create --replication-factor 3 --partitions 3 --topic test
Created topic test.
（2）在/opt/module/kafka/bin 目录下面有这两个文件。我们来测试一下
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=16384 linger.ms=0
参数说明：
- record-size 是一条信息有多大，单位是字节，本次测试设置为 1k。
- num-records 是总共发送多少条信息，本次测试设置为 100 万条。
- throughput 是每秒多少条信息，设成-1，表示不限流，尽可能快的生产数据，可测 出生产者最大吞吐量。本次实验设置为每秒钟 1 万条。
- producer-props 后面可以配置生产者相关参数，batch.size 配置为 16k。
输出结果：
35866 records sent, 7171.8 records/sec (7.00 MB/sec), 955.4 ms avg latency, 1535.0 ms max latency.
35985 records sent, 7178.3 records/sec (7.01 MB/sec), 2148.6 ms avg latency, 2946.0 ms max latency.
31260 records sent, 6249.5 records/sec (6.10 MB/sec), 3874.2 ms avg latency, 4745.0 ms max latency.
28440 records sent, 5686.9 records/sec (5.55 MB/sec), 5275.8 ms avg latency, 5722.0 ms max latency.
36570 records sent, 7312.5 records/sec (7.14 MB/sec), 4605.9 ms avg latency, 5595.0 ms max latency.
32370 records sent, 6470.1 records/sec (6.32 MB/sec), 4307.3 ms avg latency, 5517.0 ms max latency.
31800 records sent, 6358.7 records/sec (6.21 MB/sec), 4807.2 ms avg latency, 5956.0 ms max latency.
30540 records sent, 6108.0 records/sec (5.96 MB/sec), 4988.9 ms avg latency, 6340.0 ms max latency.
31035 records sent, 6207.0 records/sec (6.06 MB/sec), 4990.0 ms avg latency, 6519.0 ms max latency.
38655 records sent, 7718.7 records/sec (7.54 MB/sec), 4226.9 ms avg latency, 6073.0 ms max latency.
37620 records sent, 7524.0 records/sec (7.35 MB/sec), 4030.5 ms avg latency, 5601.0 ms max latency.
26745 records sent, 5347.9 records/sec (5.22 MB/sec), 4790.1 ms avg latency, 7275.0 ms max latency.
29340 records sent, 5868.0 records/sec (5.73 MB/sec), 5366.6 ms avg latency, 8056.0 ms max latency.
31515 records sent, 6303.0 records/sec (6.16 MB/sec), 4896.6 ms avg latency, 7823.0 ms max latency.
30630 records sent, 6122.3 records/sec (5.98 MB/sec), 5141.6 ms avg latency, 8067.0 ms max latency.
35955 records sent, 7191.0 records/sec (7.02 MB/sec), 4426.8 ms avg latency, 8060.0 ms max latency.
39615 records sent, 7923.0 records/sec (7.74 MB/sec), 3934.8 ms avg latency, 7256.0 ms max latency.
26430 records sent, 5283.9 records/sec (5.16 MB/sec), 4725.2 ms avg latency, 8573.0 ms max latency.
38730 records sent, 7733.6 records/sec (7.55 MB/sec), 4578.7 ms avg latency, 8971.0 ms max latency.
33255 records sent, 6647.0 records/sec (6.49 MB/sec), 4245.3 ms avg latency, 8234.0 ms max latency.
29430 records sent, 5876.6 records/sec (5.74 MB/sec), 4824.8 ms avg latency, 8985.0 ms max latency.
37590 records sent, 7513.5 records/sec (7.34 MB/sec), 4701.9 ms avg latency, 9112.0 ms max latency.
36390 records sent, 7278.0 records/sec (7.11 MB/sec), 4150.9 ms avg latency, 7991.0 ms max latency.
29865 records sent, 5959.9 records/sec (5.82 MB/sec), 4618.2 ms avg latency, 8783.0 ms max latency.
34815 records sent, 6960.2 records/sec (6.80 MB/sec), 4819.7 ms avg latency, 9308.0 ms max latency.
37290 records sent, 7458.0 records/sec (7.28 MB/sec), 4054.0 ms avg latency, 8845.0 ms max latency.
37905 records sent, 7578.0 records/sec (7.40 MB/sec), 4141.9 ms avg latency, 7919.0 ms max latency.
38295 records sent, 7651.3 records/sec (7.47 MB/sec), 4101.3 ms avg latency, 8071.0 ms max latency.
38985 records sent, 7792.3 records/sec (7.61 MB/sec), 3989.0 ms avg latency, 7766.0 ms max latency.
1000000 records sent, 6680.517607 records/sec (6.52 MB/sec), 4324.51 ms avg latency, 9308.00 ms max latency, 3933 ms 50th, 8312 ms 95th, 9030 ms 99th, 9278 ms 99.9th.
（3）调整 batch.size 大小
1）batch.size 默认值是 16k。本次实验 batch.size 设置为 32k。
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=32768 linger.ms=0
输出结果：
49832 records sent, 9966.4 records/sec (9.73 MB/sec), 19.1 ms avg latency, 319.0 ms max latency.
49867 records sent, 9967.4 records/sec (9.73 MB/sec), 16.5 ms avg latency, 117.0 ms max latency.
47808 records sent, 9482.0 records/sec (9.26 MB/sec), 27.2 ms avg latency, 299.0 ms max latency.
52613 records sent, 10520.5 records/sec (10.27 MB/sec), 42.2 ms avg latency, 317.0 ms max latency.
50252 records sent, 10044.4 records/sec (9.81 MB/sec), 46.8 ms avg latency, 249.0 ms max latency.
49629 records sent, 9923.8 records/sec (9.69 MB/sec), 116.9 ms avg latency, 517.0 ms max latency.
48879 records sent, 8725.3 records/sec (8.52 MB/sec), 128.4 ms avg latency, 733.0 ms max latency.
55218 records sent, 11039.2 records/sec (10.78 MB/sec), 201.8 ms avg latency, 819.0 ms max latency.
52279 records sent, 10453.7 records/sec (10.21 MB/sec), 48.4 ms avg latency, 282.0 ms max latency.
50112 records sent, 10022.4 records/sec (9.79 MB/sec), 171.9 ms avg latency, 840.0 ms max latency.
46496 records sent, 9251.1 records/sec (9.03 MB/sec), 117.6 ms avg latency, 618.0 ms max latency.
53847 records sent, 10765.1 records/sec (10.51 MB/sec), 119.2 ms avg latency, 508.0 ms max latency.
49986 records sent, 9995.2 records/sec (9.76 MB/sec), 111.9 ms avg latency, 513.0 ms max latency.
48533 records sent, 9706.6 records/sec (9.48 MB/sec), 187.1 ms avg latency, 695.0 ms max latency.
49460 records sent, 9876.2 records/sec (9.64 MB/sec), 79.4 ms avg latency, 440.0 ms max latency.
46449 records sent, 9269.4 records/sec (9.05 MB/sec), 81.5 ms avg latency, 626.0 ms max latency.
55646 records sent, 11129.2 records/sec (10.87 MB/sec), 84.3 ms avg latency, 611.0 ms max latency.
50216 records sent, 10037.2 records/sec (9.80 MB/sec), 72.7 ms avg latency, 378.0 ms max latency.
49826 records sent, 9965.2 records/sec (9.73 MB/sec), 58.8 ms avg latency, 374.0 ms max latency.
1000000 records sent, 9996.801024 records/sec (9.76 MB/sec), 89.35 ms avg latency, 840.00 ms max latency, 31 ms 50th, 356 ms 95th, 608 ms 99th, 798 ms 99.9th.
2）batch.size 默认值是 16k。本次实验 batch.size 设置为 4k。
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=4096 linger.ms=0
输出结果：
13648 records sent, 2729.1 records/sec (2.67 MB/sec), 1973.9 ms avg latency, 3685.0 ms max latency.
13854 records sent, 2770.2 records/sec (2.71 MB/sec), 5472.5 ms avg latency, 7286.0 ms max latency.
12543 records sent, 2508.6 records/sec (2.45 MB/sec), 8512.1 ms avg latency, 9605.0 ms max latency.
11997 records sent, 2399.4 records/sec (2.34 MB/sec), 9764.4 ms avg latency, 10409.0 ms max latency.
13017 records sent, 2603.4 records/sec (2.54 MB/sec), 10053.2 ms avg latency, 10541.0 ms max latency.
12534 records sent, 2506.3 records/sec (2.45 MB/sec), 9607.0 ms avg latency, 10197.0 ms max latency.
11934 records sent, 2386.3 records/sec (2.33 MB/sec), 9881.4 ms avg latency, 10356.0 ms max latency.
12651 records sent, 2530.2 records/sec (2.47 MB/sec), 10046.1 ms avg latency, 10403.0 ms max latency.
11946 records sent, 2388.7 records/sec (2.33 MB/sec), 9857.5 ms avg latency, 10210.0 ms max latency.
11298 records sent, 2259.6 records/sec (2.21 MB/sec), 10368.7 ms avg latency, 10718.0 ms max latency.
12618 records sent, 2502.6 records/sec (2.44 MB/sec), 10533.8 ms avg latency, 10855.0 ms max latency.
12366 records sent, 2473.2 records/sec (2.42 MB/sec), 9974.7 ms avg latency, 10602.0 ms max latency.
12090 records sent, 2418.0 records/sec (2.36 MB/sec), 9951.1 ms avg latency, 10407.0 ms max latency.
11022 records sent, 2203.5 records/sec (2.15 MB/sec), 10195.0 ms avg latency, 10892.0 ms max latency.
9936 records sent, 1986.4 records/sec (1.94 MB/sec), 10972.8 ms avg latency, 11831.0 ms max latency.
7872 records sent, 1574.1 records/sec (1.54 MB/sec), 12381.0 ms avg latency, 13411.0 ms max latency.
9705 records sent, 1941.0 records/sec (1.90 MB/sec), 13646.4 ms avg latency, 14004.0 ms max latency.
10710 records sent, 2141.6 records/sec (2.09 MB/sec), 13450.4 ms avg latency, 13972.0 ms max latency.
9144 records sent, 1828.8 records/sec (1.79 MB/sec), 12383.0 ms avg latency, 12950.0 ms max latency.
11490 records sent, 2297.5 records/sec (2.24 MB/sec), 11951.2 ms avg latency, 12379.0 ms max latency.
9840 records sent, 1962.1 records/sec (1.92 MB/sec), 11573.2 ms avg latency, 12173.0 ms max latency.
8244 records sent, 1648.5 records/sec (1.61 MB/sec), 12331.1 ms avg latency, 13008.0 ms max latency.
10557 records sent, 2111.0 records/sec (2.06 MB/sec), 13079.6 ms avg latency, 13560.0 ms max latency.
10356 records sent, 2071.2 records/sec (2.02 MB/sec), 12871.3 ms avg latency, 13829.0 ms max latency.
11022 records sent, 2204.0 records/sec (2.15 MB/sec), 11618.8 ms avg latency, 12555.0 ms max latency.
11616 records sent, 2323.2 records/sec (2.27 MB/sec), 11198.5 ms avg latency, 11977.0 ms max latency.
10728 records sent, 2145.6 records/sec (2.10 MB/sec), 10738.9 ms avg latency, 11567.0 ms max latency.
10872 records sent, 2174.0 records/sec (2.12 MB/sec), 11279.1 ms avg latency, 11888.0 ms max latency.
11985 records sent, 2396.5 records/sec (2.34 MB/sec), 11268.4 ms avg latency, 11683.0 ms max latency.
11232 records sent, 2246.4 records/sec (2.19 MB/sec), 10639.0 ms avg latency, 11174.0 ms max latency.
11418 records sent, 2283.6 records/sec (2.23 MB/sec), 10611.7 ms avg latency, 11093.0 ms max latency.
11568 records sent, 2313.6 records/sec (2.26 MB/sec), 10698.5 ms avg latency, 11257.0 ms max latency.
10620 records sent, 2124.0 records/sec (2.07 MB/sec), 10789.7 ms avg latency, 11493.0 ms max latency.
10863 records sent, 2172.6 records/sec (2.12 MB/sec), 11348.5 ms avg latency, 11885.0 ms max latency.
11433 records sent, 2285.7 records/sec (2.23 MB/sec), 11334.5 ms avg latency, 11870.0 ms max latency.
11463 records sent, 2292.6 records/sec (2.24 MB/sec), 10891.2 ms avg latency, 11848.0 ms max latency.
11376 records sent, 2270.7 records/sec (2.22 MB/sec), 10780.6 ms avg latency, 11222.0 ms max latency.
11745 records sent, 2349.0 records/sec (2.29 MB/sec), 10642.6 ms avg latency, 11051.0 ms max latency.
10530 records sent, 2105.6 records/sec (2.06 MB/sec), 10717.0 ms avg latency, 11307.0 ms max latency.
10332 records sent, 2047.2 records/sec (2.00 MB/sec), 11490.2 ms avg latency, 11986.0 ms max latency.
11856 records sent, 2371.2 records/sec (2.32 MB/sec), 11501.7 ms avg latency, 11990.0 ms max latency.
11595 records sent, 2319.0 records/sec (2.26 MB/sec), 10742.1 ms avg latency, 11733.0 ms max latency.
11256 records sent, 2251.2 records/sec (2.20 MB/sec), 10580.1 ms avg latency, 11021.0 ms max latency.
11397 records sent, 2279.4 records/sec (2.23 MB/sec), 10754.6 ms avg latency, 11303.0 ms max latency.
10749 records sent, 2149.8 records/sec (2.10 MB/sec), 10970.8 ms avg latency, 11483.0 ms max latency.
10665 records sent, 2133.0 records/sec (2.08 MB/sec), 11414.5 ms avg latency, 11901.0 ms max latency.
9534 records sent, 1906.4 records/sec (1.86 MB/sec), 11640.7 ms avg latency, 12677.0 ms max latency.
10242 records sent, 2048.4 records/sec (2.00 MB/sec), 12257.5 ms avg latency, 13088.0 ms max latency.
11757 records sent, 2350.9 records/sec (2.30 MB/sec), 11875.5 ms avg latency, 12680.0 ms max latency.
12018 records sent, 2403.6 records/sec (2.35 MB/sec), 10719.5 ms avg latency, 12420.0 ms max latency.
11526 records sent, 2304.3 records/sec (2.25 MB/sec), 10247.4 ms avg latency, 11212.0 ms max latency.
10794 records sent, 2158.8 records/sec (2.11 MB/sec), 10822.6 ms avg latency, 11714.0 ms max latency.
12444 records sent, 2487.8 records/sec (2.43 MB/sec), 10839.0 ms avg latency, 11787.0 ms max latency.
12225 records sent, 2444.5 records/sec (2.39 MB/sec), 10203.9 ms avg latency, 11747.0 ms max latency.
12111 records sent, 2422.2 records/sec (2.37 MB/sec), 10019.2 ms avg latency, 11052.0 ms max latency.
12144 records sent, 2428.8 records/sec (2.37 MB/sec), 10146.3 ms avg latency, 11128.0 ms max latency.
11100 records sent, 2219.1 records/sec (2.17 MB/sec), 10335.6 ms avg latency, 11383.0 ms max latency.
11331 records sent, 2265.7 records/sec (2.21 MB/sec), 10842.3 ms avg latency, 12033.0 ms max latency.
11922 records sent, 2383.9 records/sec (2.33 MB/sec), 10784.2 ms avg latency, 11902.0 ms max latency.
12330 records sent, 2465.5 records/sec (2.41 MB/sec), 10241.5 ms avg latency, 11715.0 ms max latency.
12450 records sent, 2489.5 records/sec (2.43 MB/sec), 10039.0 ms avg latency, 11237.0 ms max latency.
11703 records sent, 2340.6 records/sec (2.29 MB/sec), 10062.8 ms avg latency, 11151.0 ms max latency.
10134 records sent, 2026.4 records/sec (1.98 MB/sec), 10533.5 ms avg latency, 11947.0 ms max latency.
9501 records sent, 1900.2 records/sec (1.86 MB/sec), 11645.4 ms avg latency, 13093.0 ms max latency.
10800 records sent, 2134.8 records/sec (2.08 MB/sec), 12113.9 ms avg latency, 13510.0 ms max latency.
10623 records sent, 2102.7 records/sec (2.05 MB/sec), 11806.5 ms avg latency, 13552.0 ms max latency.
9999 records sent, 1999.0 records/sec (1.95 MB/sec), 11637.2 ms avg latency, 13243.0 ms max latency.
10980 records sent, 2195.1 records/sec (2.14 MB/sec), 12049.9 ms avg latency, 13605.0 ms max latency.
9657 records sent, 1930.2 records/sec (1.89 MB/sec), 11840.2 ms avg latency, 13783.0 ms max latency.
9909 records sent, 1981.4 records/sec (1.93 MB/sec), 12306.3 ms avg latency, 14313.0 ms max latency.
11220 records sent, 2244.0 records/sec (2.19 MB/sec), 12205.3 ms avg latency, 13538.0 ms max latency.
10836 records sent, 2167.2 records/sec (2.12 MB/sec), 11568.3 ms avg latency, 13490.0 ms max latency.
11736 records sent, 2347.2 records/sec (2.29 MB/sec), 10902.4 ms avg latency, 12581.0 ms max latency.
11898 records sent, 2357.9 records/sec (2.30 MB/sec), 10688.4 ms avg latency, 12502.0 ms max latency.
11253 records sent, 2249.7 records/sec (2.20 MB/sec), 10429.0 ms avg latency, 11836.0 ms max latency.
11406 records sent, 2280.7 records/sec (2.23 MB/sec), 10803.9 ms avg latency, 12144.0 ms max latency.
12462 records sent, 2491.9 records/sec (2.43 MB/sec), 10626.9 ms avg latency, 12126.0 ms max latency.
12606 records sent, 2521.2 records/sec (2.46 MB/sec), 9991.6 ms avg latency, 12050.0 ms max latency.
12063 records sent, 2412.6 records/sec (2.36 MB/sec), 9916.6 ms avg latency, 11462.0 ms max latency.
12183 records sent, 2436.6 records/sec (2.38 MB/sec), 10066.2 ms avg latency, 11620.0 ms max latency.
11040 records sent, 2208.0 records/sec (2.16 MB/sec), 10130.1 ms avg latency, 12287.0 ms max latency.
12231 records sent, 2445.7 records/sec (2.39 MB/sec), 10598.4 ms avg latency, 12378.0 ms max latency.
12300 records sent, 2458.5 records/sec (2.40 MB/sec), 10328.1 ms avg latency, 11941.0 ms max latency.
12795 records sent, 2558.0 records/sec (2.50 MB/sec), 9953.8 ms avg latency, 11811.0 ms max latency.
11910 records sent, 2382.0 records/sec (2.33 MB/sec), 9944.5 ms avg latency, 11616.0 ms max latency.
12471 records sent, 2493.7 records/sec (2.44 MB/sec), 10045.1 ms avg latency, 11657.0 ms max latency.
11328 records sent, 2265.1 records/sec (2.21 MB/sec), 9951.2 ms avg latency, 11626.0 ms max latency.
11227 records sent, 2245.4 records/sec (2.19 MB/sec), 10693.6 ms avg latency, 11994.0 ms max latency.
1000000 records sent, 2263.216619 records/sec (2.21 MB/sec), 10668.88 ms avg latency, 14313.00 ms max latency, 10877 ms 50th, 13217 ms 95th, 13703 ms 99th, 14065 ms 99.9th.
（4）调整 linger.ms 时间
10000 records sent, 2000.0 records/sec (1.95 MB/sec), 1820.2 ms avg latency, 4004.0 ms max latency.
11256 records sent, 2251.2 records/sec (2.20 MB/sec), 6002.4 ms avg latency, 7927.0 ms max latency.
12033 records sent, 2405.2 records/sec (2.35 MB/sec), 9679.6 ms avg latency, 11373.0 ms max latency.
12531 records sent, 2506.2 records/sec (2.45 MB/sec), 10252.2 ms avg latency, 11268.0 ms max latency.
12528 records sent, 2505.6 records/sec (2.45 MB/sec), 9909.1 ms avg latency, 10469.0 ms max latency.
12132 records sent, 2426.4 records/sec (2.37 MB/sec), 9857.9 ms avg latency, 10598.0 ms max latency.
9933 records sent, 1986.6 records/sec (1.94 MB/sec), 10275.9 ms avg latency, 11773.0 ms max latency.
9657 records sent, 1931.0 records/sec (1.89 MB/sec), 11526.3 ms avg latency, 13158.0 ms max latency.
11364 records sent, 2272.3 records/sec (2.22 MB/sec), 12284.6 ms avg latency, 13667.0 ms max latency.
11286 records sent, 2256.7 records/sec (2.20 MB/sec), 11443.5 ms avg latency, 13691.0 ms max latency.
10779 records sent, 2155.8 records/sec (2.11 MB/sec), 10990.4 ms avg latency, 12891.0 ms max latency.
11544 records sent, 2308.8 records/sec (2.25 MB/sec), 10962.0 ms avg latency, 12397.0 ms max latency.
9081 records sent, 1815.5 records/sec (1.77 MB/sec), 11277.8 ms avg latency, 13220.0 ms max latency.
12006 records sent, 2401.2 records/sec (2.34 MB/sec), 11606.9 ms avg latency, 13345.0 ms max latency.
11205 records sent, 2217.9 records/sec (2.17 MB/sec), 11281.3 ms avg latency, 13616.0 ms max latency.
9642 records sent, 1928.4 records/sec (1.88 MB/sec), 11274.7 ms avg latency, 13387.0 ms max latency.
10761 records sent, 2152.2 records/sec (2.10 MB/sec), 11908.6 ms avg latency, 14013.0 ms max latency.
9906 records sent, 1980.4 records/sec (1.93 MB/sec), 11939.9 ms avg latency, 14443.0 ms max latency.
9951 records sent, 1990.2 records/sec (1.94 MB/sec), 12017.1 ms avg latency, 14671.0 ms max latency.
11211 records sent, 2241.8 records/sec (2.19 MB/sec), 11899.4 ms avg latency, 14401.0 ms max latency.
10902 records sent, 2179.5 records/sec (2.13 MB/sec), 11568.6 ms avg latency, 14362.0 ms max latency.
10722 records sent, 2144.4 records/sec (2.09 MB/sec), 11196.9 ms avg latency, 13731.0 ms max latency.
10851 records sent, 2170.2 records/sec (2.12 MB/sec), 11200.7 ms avg latency, 13821.0 ms max latency.
12339 records sent, 2467.3 records/sec (2.41 MB/sec), 10978.0 ms avg latency, 13755.0 ms max latency.
9378 records sent, 1875.6 records/sec (1.83 MB/sec), 10945.1 ms avg latency, 14214.0 ms max latency.
11505 records sent, 2301.0 records/sec (2.25 MB/sec), 11303.8 ms avg latency, 14221.0 ms max latency.
10536 records sent, 2107.2 records/sec (2.06 MB/sec), 11624.4 ms avg latency, 14849.0 ms max latency.
12510 records sent, 2502.0 records/sec (2.44 MB/sec), 11011.3 ms avg latency, 14857.0 ms max latency.
11658 records sent, 2331.6 records/sec (2.28 MB/sec), 10242.9 ms avg latency, 13370.0 ms max latency.
12111 records sent, 2421.7 records/sec (2.36 MB/sec), 10241.1 ms avg latency, 13099.0 ms max latency.
9561 records sent, 1912.2 records/sec (1.87 MB/sec), 10771.6 ms avg latency, 14110.0 ms max latency.
12189 records sent, 2437.8 records/sec (2.38 MB/sec), 11226.9 ms avg latency, 14363.0 ms max latency.
12003 records sent, 2400.1 records/sec (2.34 MB/sec), 10777.4 ms avg latency, 14208.0 ms max latency.
12498 records sent, 2499.6 records/sec (2.44 MB/sec), 10095.7 ms avg latency, 14115.0 ms max latency.
11637 records sent, 2326.9 records/sec (2.27 MB/sec), 9995.3 ms avg latency, 13389.0 ms max latency.
11979 records sent, 2395.3 records/sec (2.34 MB/sec), 10330.5 ms avg latency, 13610.0 ms max latency.
9657 records sent, 1931.0 records/sec (1.89 MB/sec), 10768.7 ms avg latency, 14604.0 ms max latency.
12210 records sent, 2442.0 records/sec (2.38 MB/sec), 11109.4 ms avg latency, 14787.0 ms max latency.
10569 records sent, 2113.4 records/sec (2.06 MB/sec), 10952.5 ms avg latency, 15308.0 ms max latency.
9495 records sent, 1899.0 records/sec (1.85 MB/sec), 11700.2 ms avg latency, 16108.0 ms max latency.
11841 records sent, 2367.7 records/sec (2.31 MB/sec), 11583.5 ms avg latency, 15605.0 ms max latency.
9426 records sent, 1884.8 records/sec (1.84 MB/sec), 11684.5 ms avg latency, 16292.0 ms max latency.
9351 records sent, 1869.8 records/sec (1.83 MB/sec), 12183.9 ms avg latency, 16142.0 ms max latency.
11343 records sent, 2268.6 records/sec (2.22 MB/sec), 12159.8 ms avg latency, 16183.0 ms max latency.
10725 records sent, 2144.1 records/sec (2.09 MB/sec), 11535.2 ms avg latency, 16386.0 ms max latency.
11055 records sent, 2211.0 records/sec (2.16 MB/sec), 11402.9 ms avg latency, 16058.0 ms max latency.
12117 records sent, 2423.4 records/sec (2.37 MB/sec), 10963.7 ms avg latency, 15120.0 ms max latency.
11943 records sent, 2383.4 records/sec (2.33 MB/sec), 10443.6 ms avg latency, 14973.0 ms max latency.
8892 records sent, 1770.6 records/sec (1.73 MB/sec), 10687.7 ms avg latency, 15575.0 ms max latency.
10986 records sent, 2196.3 records/sec (2.14 MB/sec), 11637.8 ms avg latency, 16285.0 ms max latency.
11757 records sent, 2350.9 records/sec (2.30 MB/sec), 11427.1 ms avg latency, 16750.0 ms max latency.
11880 records sent, 2375.5 records/sec (2.32 MB/sec), 10908.3 ms avg latency, 16808.0 ms max latency.
12258 records sent, 2451.6 records/sec (2.39 MB/sec), 10197.9 ms avg latency, 15499.0 ms max latency.
11769 records sent, 2353.8 records/sec (2.30 MB/sec), 10121.6 ms avg latency, 15260.0 ms max latency.
10440 records sent, 2088.0 records/sec (2.04 MB/sec), 10469.7 ms avg latency, 15898.0 ms max latency.
11967 records sent, 2393.4 records/sec (2.34 MB/sec), 10899.5 ms avg latency, 16095.0 ms max latency.
12306 records sent, 2461.2 records/sec (2.40 MB/sec), 10463.5 ms avg latency, 16186.0 ms max latency.
12204 records sent, 2440.8 records/sec (2.38 MB/sec), 10245.6 ms avg latency, 15932.0 ms max latency.
12579 records sent, 2485.0 records/sec (2.43 MB/sec), 10017.9 ms avg latency, 15379.0 ms max latency.
12354 records sent, 2470.8 records/sec (2.41 MB/sec), 9829.3 ms avg latency, 15139.0 ms max latency.
10371 records sent, 2074.2 records/sec (2.03 MB/sec), 10376.7 ms avg latency, 15931.0 ms max latency.
12027 records sent, 2405.4 records/sec (2.35 MB/sec), 10893.2 ms avg latency, 16053.0 ms max latency.
12066 records sent, 2413.2 records/sec (2.36 MB/sec), 10616.4 ms avg latency, 16046.0 ms max latency.
12393 records sent, 2478.1 records/sec (2.42 MB/sec), 10270.5 ms avg latency, 15939.0 ms max latency.
12414 records sent, 2482.3 records/sec (2.42 MB/sec), 9926.0 ms avg latency, 14963.0 ms max latency.
11829 records sent, 2365.8 records/sec (2.31 MB/sec), 9961.6 ms avg latency, 15103.0 ms max latency.
10695 records sent, 2139.0 records/sec (2.09 MB/sec), 10298.7 ms avg latency, 15851.0 ms max latency.
12045 records sent, 2409.0 records/sec (2.35 MB/sec), 10718.3 ms avg latency, 16178.0 ms max latency.
11790 records sent, 2357.5 records/sec (2.30 MB/sec), 10505.9 ms avg latency, 16436.0 ms max latency.
11991 records sent, 2397.7 records/sec (2.34 MB/sec), 10262.7 ms avg latency, 16491.0 ms max latency.
12201 records sent, 2440.2 records/sec (2.38 MB/sec), 10283.7 ms avg latency, 16306.0 ms max latency.
12624 records sent, 2524.8 records/sec (2.47 MB/sec), 9856.8 ms avg latency, 16035.0 ms max latency.
9522 records sent, 1904.4 records/sec (1.86 MB/sec), 10169.9 ms avg latency, 17061.0 ms max latency.
9819 records sent, 1963.4 records/sec (1.92 MB/sec), 11531.6 ms avg latency, 18043.0 ms max latency.
10845 records sent, 2169.0 records/sec (2.12 MB/sec), 11946.9 ms avg latency, 18684.0 ms max latency.
11388 records sent, 2277.1 records/sec (2.22 MB/sec), 11388.4 ms avg latency, 19132.0 ms max latency.
12039 records sent, 2407.3 records/sec (2.35 MB/sec), 10979.0 ms avg latency, 19104.0 ms max latency.
11493 records sent, 2298.6 records/sec (2.24 MB/sec), 10546.6 ms avg latency, 17845.0 ms max latency.
9639 records sent, 1927.4 records/sec (1.88 MB/sec), 11200.0 ms avg latency, 18139.0 ms max latency.
11334 records sent, 2266.8 records/sec (2.21 MB/sec), 11517.8 ms avg latency, 18324.0 ms max latency.
11634 records sent, 2325.9 records/sec (2.27 MB/sec), 10925.3 ms avg latency, 18284.0 ms max latency.
11088 records sent, 2207.0 records/sec (2.16 MB/sec), 10838.6 ms avg latency, 18467.0 ms max latency.
10920 records sent, 2183.6 records/sec (2.13 MB/sec), 10981.0 ms avg latency, 18081.0 ms max latency.
10782 records sent, 2156.4 records/sec (2.11 MB/sec), 11034.2 ms avg latency, 18361.0 ms max latency.
9312 records sent, 1859.4 records/sec (1.82 MB/sec), 11498.7 ms avg latency, 19558.0 ms max latency.
11136 records sent, 2227.2 records/sec (2.18 MB/sec), 11905.4 ms avg latency, 19797.0 ms max latency.
10995 records sent, 2187.6 records/sec (2.14 MB/sec), 11613.7 ms avg latency, 19646.0 ms max latency.
11745 records sent, 2348.5 records/sec (2.29 MB/sec), 11173.0 ms avg latency, 19267.0 ms max latency.
8079 records sent, 1615.2 records/sec (1.58 MB/sec), 13822.4 ms avg latency, 18832.0 ms max latency.
4413 records sent, 882.4 records/sec (0.86 MB/sec), 16688.3 ms avg latency, 17205.0 ms max latency.
1000000 records sent, 2213.927374 records/sec (2.16 MB/sec), 10817.24 ms avg latency, 19797.00 ms max latency, 10111 ms 50th, 17626 ms 95th, 19089 ms 99th, 19709 ms 99.9th.
（5）调整压缩方式
1）默认的压缩方式是 none。本次实验 compression.type 设置为 snappy。
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=4096 linger.ms=50 compression.type=snappy
10948 records sent, 2189.6 records/sec (2.14 MB/sec), 2251.9 ms avg latency, 3923.0 ms max latency.
12501 records sent, 2499.2 records/sec (2.44 MB/sec), 5722.7 ms avg latency, 7718.0 ms max latency.
11106 records sent, 2215.9 records/sec (2.16 MB/sec), 9361.3 ms avg latency, 10648.0 ms max latency.
11799 records sent, 2359.8 records/sec (2.30 MB/sec), 10511.5 ms avg latency, 11243.0 ms max latency.
11163 records sent, 2232.6 records/sec (2.18 MB/sec), 10760.7 ms avg latency, 11298.0 ms max latency.
8667 records sent, 1733.4 records/sec (1.69 MB/sec), 11089.3 ms avg latency, 12619.0 ms max latency.
10728 records sent, 2145.6 records/sec (2.10 MB/sec), 12451.3 ms avg latency, 13259.0 ms max latency.
11859 records sent, 2371.8 records/sec (2.32 MB/sec), 12042.3 ms avg latency, 13037.0 ms max latency.
12045 records sent, 2408.5 records/sec (2.35 MB/sec), 10517.1 ms avg latency, 12246.0 ms max latency.
9960 records sent, 1992.0 records/sec (1.95 MB/sec), 10783.6 ms avg latency, 11830.0 ms max latency.
11100 records sent, 2219.6 records/sec (2.17 MB/sec), 11159.6 ms avg latency, 11936.0 ms max latency.
8880 records sent, 1776.0 records/sec (1.73 MB/sec), 11881.7 ms avg latency, 12895.0 ms max latency.
11733 records sent, 2346.6 records/sec (2.29 MB/sec), 12007.7 ms avg latency, 12887.0 ms max latency.
12120 records sent, 2424.0 records/sec (2.37 MB/sec), 11316.5 ms avg latency, 12663.0 ms max latency.
11235 records sent, 2246.6 records/sec (2.19 MB/sec), 10449.9 ms avg latency, 11728.0 ms max latency.
10023 records sent, 2004.6 records/sec (1.96 MB/sec), 10879.9 ms avg latency, 12042.0 ms max latency.
9336 records sent, 1865.3 records/sec (1.82 MB/sec), 11861.9 ms avg latency, 12995.0 ms max latency.
8616 records sent, 1721.8 records/sec (1.68 MB/sec), 12851.0 ms avg latency, 14148.0 ms max latency.
10761 records sent, 2151.8 records/sec (2.10 MB/sec), 13254.7 ms avg latency, 14494.0 ms max latency.
9840 records sent, 1968.0 records/sec (1.92 MB/sec), 12574.9 ms avg latency, 13869.0 ms max latency.
9858 records sent, 1971.6 records/sec (1.93 MB/sec), 12155.8 ms avg latency, 13760.0 ms max latency.
11295 records sent, 2259.0 records/sec (2.21 MB/sec), 11988.0 ms avg latency, 13190.0 ms max latency.
10101 records sent, 2019.8 records/sec (1.97 MB/sec), 11613.8 ms avg latency, 12858.0 ms max latency.
8628 records sent, 1724.9 records/sec (1.68 MB/sec), 12047.5 ms avg latency, 13547.0 ms max latency.
11016 records sent, 2202.3 records/sec (2.15 MB/sec), 12683.2 ms avg latency, 13720.0 ms max latency.
10941 records sent, 2188.2 records/sec (2.14 MB/sec), 12131.6 ms avg latency, 13477.0 ms max latency.
10047 records sent, 2009.4 records/sec (1.96 MB/sec), 11505.9 ms avg latency, 12567.0 ms max latency.
12159 records sent, 2430.8 records/sec (2.37 MB/sec), 11272.7 ms avg latency, 12477.0 ms max latency.
12357 records sent, 2470.4 records/sec (2.41 MB/sec), 10579.4 ms avg latency, 12294.0 ms max latency.
9921 records sent, 1984.2 records/sec (1.94 MB/sec), 10479.9 ms avg latency, 12033.0 ms max latency.
10773 records sent, 2154.6 records/sec (2.10 MB/sec), 11282.2 ms avg latency, 12460.0 ms max latency.
9846 records sent, 1969.2 records/sec (1.92 MB/sec), 12004.7 ms avg latency, 13522.0 ms max latency.
10008 records sent, 2001.2 records/sec (1.95 MB/sec), 12047.6 ms avg latency, 13331.0 ms max latency.
10938 records sent, 2187.2 records/sec (2.14 MB/sec), 12003.9 ms avg latency, 13109.0 ms max latency.
11136 records sent, 2227.2 records/sec (2.18 MB/sec), 11452.3 ms avg latency, 13059.0 ms max latency.
9750 records sent, 1949.2 records/sec (1.90 MB/sec), 11273.9 ms avg latency, 12728.0 ms max latency.
11613 records sent, 2306.0 records/sec (2.25 MB/sec), 11629.3 ms avg latency, 12881.0 ms max latency.
9981 records sent, 1995.8 records/sec (1.95 MB/sec), 11681.6 ms avg latency, 13172.0 ms max latency.
10902 records sent, 2178.2 records/sec (2.13 MB/sec), 11595.8 ms avg latency, 13172.0 ms max latency.
10638 records sent, 2126.7 records/sec (2.08 MB/sec), 11690.8 ms avg latency, 13004.0 ms max latency.
10707 records sent, 2141.4 records/sec (2.09 MB/sec), 11394.9 ms avg latency, 12826.0 ms max latency.
9639 records sent, 1927.8 records/sec (1.88 MB/sec), 11634.1 ms avg latency, 13083.0 ms max latency.
9951 records sent, 1990.2 records/sec (1.94 MB/sec), 12174.6 ms avg latency, 13615.0 ms max latency.
10545 records sent, 2109.0 records/sec (2.06 MB/sec), 12370.8 ms avg latency, 13865.0 ms max latency.
10209 records sent, 2041.4 records/sec (1.99 MB/sec), 12019.8 ms avg latency, 13829.0 ms max latency.
11307 records sent, 2260.9 records/sec (2.21 MB/sec), 11591.3 ms avg latency, 13586.0 ms max latency.
10920 records sent, 2183.6 records/sec (2.13 MB/sec), 11269.5 ms avg latency, 12979.0 ms max latency.
9132 records sent, 1826.0 records/sec (1.78 MB/sec), 11647.1 ms avg latency, 13530.0 ms max latency.
11349 records sent, 2269.3 records/sec (2.22 MB/sec), 11955.6 ms avg latency, 13642.0 ms max latency.
11328 records sent, 2265.1 records/sec (2.21 MB/sec), 11520.7 ms avg latency, 13696.0 ms max latency.
11808 records sent, 2361.6 records/sec (2.31 MB/sec), 10807.6 ms avg latency, 13036.0 ms max latency.
12648 records sent, 2529.6 records/sec (2.47 MB/sec), 10312.0 ms avg latency, 12405.0 ms max latency.
12651 records sent, 2529.2 records/sec (2.47 MB/sec), 9784.3 ms avg latency, 11975.0 ms max latency.
10440 records sent, 2088.0 records/sec (2.04 MB/sec), 10199.4 ms avg latency, 12578.0 ms max latency.
11895 records sent, 2378.0 records/sec (2.32 MB/sec), 10765.1 ms avg latency, 13003.0 ms max latency.
12039 records sent, 2407.8 records/sec (2.35 MB/sec), 10561.0 ms avg latency, 13160.0 ms max latency.
12333 records sent, 2465.6 records/sec (2.41 MB/sec), 10168.6 ms avg latency, 12701.0 ms max latency.
11889 records sent, 2377.8 records/sec (2.32 MB/sec), 10134.1 ms avg latency, 12351.0 ms max latency.
12225 records sent, 2444.5 records/sec (2.39 MB/sec), 10197.8 ms avg latency, 12314.0 ms max latency.
10584 records sent, 2116.4 records/sec (2.07 MB/sec), 10554.8 ms avg latency, 12848.0 ms max latency.
11679 records sent, 2335.3 records/sec (2.28 MB/sec), 10910.9 ms avg latency, 13101.0 ms max latency.
11904 records sent, 2380.3 records/sec (2.32 MB/sec), 10793.4 ms avg latency, 13306.0 ms max latency.
11937 records sent, 2386.9 records/sec (2.33 MB/sec), 10291.7 ms avg latency, 12632.0 ms max latency.
11835 records sent, 2366.1 records/sec (2.31 MB/sec), 10217.8 ms avg latency, 12670.0 ms max latency.
12279 records sent, 2455.3 records/sec (2.40 MB/sec), 10237.9 ms avg latency, 12768.0 ms max latency.
10194 records sent, 2038.4 records/sec (1.99 MB/sec), 10552.1 ms avg latency, 13499.0 ms max latency.
11889 records sent, 2377.3 records/sec (2.32 MB/sec), 10816.2 ms avg latency, 13684.0 ms max latency.
11997 records sent, 2399.4 records/sec (2.34 MB/sec), 10740.0 ms avg latency, 13947.0 ms max latency.
12045 records sent, 2408.5 records/sec (2.35 MB/sec), 10256.4 ms avg latency, 13729.0 ms max latency.
12174 records sent, 2434.8 records/sec (2.38 MB/sec), 10225.3 ms avg latency, 13544.0 ms max latency.
12396 records sent, 2479.2 records/sec (2.42 MB/sec), 10041.1 ms avg latency, 13192.0 ms max latency.
10491 records sent, 2097.8 records/sec (2.05 MB/sec), 10364.2 ms avg latency, 13758.0 ms max latency.
11964 records sent, 2392.3 records/sec (2.34 MB/sec), 10733.5 ms avg latency, 13717.0 ms max latency.
11634 records sent, 2326.8 records/sec (2.27 MB/sec), 10655.2 ms avg latency, 13986.0 ms max latency.
12573 records sent, 2514.6 records/sec (2.46 MB/sec), 10346.2 ms avg latency, 13988.0 ms max latency.
12438 records sent, 2487.6 records/sec (2.43 MB/sec), 10001.3 ms avg latency, 13475.0 ms max latency.
12069 records sent, 2413.8 records/sec (2.36 MB/sec), 9879.4 ms avg latency, 13114.0 ms max latency.
10695 records sent, 2139.0 records/sec (2.09 MB/sec), 10442.0 ms avg latency, 13793.0 ms max latency.
11721 records sent, 2344.2 records/sec (2.29 MB/sec), 10781.8 ms avg latency, 14099.0 ms max latency.
11781 records sent, 2355.7 records/sec (2.30 MB/sec), 10619.1 ms avg latency, 14237.0 ms max latency.
12195 records sent, 2438.5 records/sec (2.38 MB/sec), 10392.2 ms avg latency, 14166.0 ms max latency.
12309 records sent, 2461.8 records/sec (2.40 MB/sec), 10112.3 ms avg latency, 13797.0 ms max latency.
10743 records sent, 2148.6 records/sec (2.10 MB/sec), 10503.2 ms avg latency, 14098.0 ms max latency.
10371 records sent, 2073.8 records/sec (2.03 MB/sec), 10946.9 ms avg latency, 15055.0 ms max latency.
12237 records sent, 2447.4 records/sec (2.39 MB/sec), 11008.7 ms avg latency, 15387.0 ms max latency.
11913 records sent, 2382.6 records/sec (2.33 MB/sec), 10411.1 ms avg latency, 15249.0 ms max latency.
12183 records sent, 2436.6 records/sec (2.38 MB/sec), 10282.0 ms avg latency, 14847.0 ms max latency.
12234 records sent, 2446.8 records/sec (2.39 MB/sec), 10119.7 ms avg latency, 14413.0 ms max latency.
11649 records sent, 2329.3 records/sec (2.27 MB/sec), 10112.6 ms avg latency, 14482.0 ms max latency.
6490 records sent, 1297.5 records/sec (1.27 MB/sec), 12166.0 ms avg latency, 14477.0 ms max latency.
1000000 records sent, 2211.195726 records/sec (2.16 MB/sec), 10873.18 ms avg latency, 15387.00 ms max latency, 11012 ms 50th, 13810 ms 95th, 14488 ms 99th, 15293 ms 99.9th.
2）默认的压缩方式是 none。本次实验 compression.type 设置为 zstd。
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=4096 linger.ms=50 compression.type=zstd
输出结果：
11500 records sent, 2300.0 records/sec (2.25 MB/sec), 2423.5 ms avg latency, 3898.0 ms max latency.
18485 records sent, 3697.0 records/sec (3.61 MB/sec), 5268.9 ms avg latency, 7053.0 ms max latency.
20025 records sent, 4005.0 records/sec (3.91 MB/sec), 8471.6 ms avg latency, 10142.0 ms max latency.
19855 records sent, 3970.2 records/sec (3.88 MB/sec), 10537.5 ms avg latency, 11089.0 ms max latency.
19140 records sent, 3828.0 records/sec (3.74 MB/sec), 10420.0 ms avg latency, 11045.0 ms max latency.
18505 records sent, 3701.0 records/sec (3.61 MB/sec), 10619.8 ms avg latency, 11197.0 ms max latency.
16680 records sent, 3336.0 records/sec (3.26 MB/sec), 11188.1 ms avg latency, 11970.0 ms max latency.
16790 records sent, 3355.3 records/sec (3.28 MB/sec), 11694.4 ms avg latency, 12479.0 ms max latency.
18310 records sent, 3662.0 records/sec (3.58 MB/sec), 12090.0 ms avg latency, 12883.0 ms max latency.
18555 records sent, 3711.0 records/sec (3.62 MB/sec), 11638.5 ms avg latency, 12304.0 ms max latency.
17100 records sent, 3420.0 records/sec (3.34 MB/sec), 11138.6 ms avg latency, 11872.0 ms max latency.
17720 records sent, 3544.0 records/sec (3.46 MB/sec), 11541.0 ms avg latency, 12451.0 ms max latency.
18745 records sent, 3746.8 records/sec (3.66 MB/sec), 11557.1 ms avg latency, 12499.0 ms max latency.
15810 records sent, 3160.7 records/sec (3.09 MB/sec), 11345.7 ms avg latency, 12314.0 ms max latency.
14030 records sent, 2805.4 records/sec (2.74 MB/sec), 12535.4 ms avg latency, 13635.0 ms max latency.
16770 records sent, 3352.7 records/sec (3.27 MB/sec), 13181.4 ms avg latency, 14350.0 ms max latency.
10630 records sent, 2125.1 records/sec (2.08 MB/sec), 13982.9 ms avg latency, 16311.0 ms max latency.
10235 records sent, 2046.2 records/sec (2.00 MB/sec), 15188.3 ms avg latency, 17464.0 ms max latency.
14015 records sent, 2801.3 records/sec (2.74 MB/sec), 16713.1 ms avg latency, 18348.0 ms max latency.
10610 records sent, 1959.4 records/sec (1.91 MB/sec), 17691.6 ms avg latency, 20752.0 ms max latency.
11535 records sent, 2307.0 records/sec (2.25 MB/sec), 18000.6 ms avg latency, 21715.0 ms max latency.
15980 records sent, 3194.7 records/sec (3.12 MB/sec), 16647.8 ms avg latency, 20709.0 ms max latency.
13080 records sent, 2599.4 records/sec (2.54 MB/sec), 15385.4 ms avg latency, 18449.0 ms max latency.
9845 records sent, 1968.6 records/sec (1.92 MB/sec), 15322.8 ms avg latency, 18837.0 ms max latency.
10885 records sent, 2177.0 records/sec (2.13 MB/sec), 17289.9 ms avg latency, 20641.0 ms max latency.
14310 records sent, 2862.0 records/sec (2.79 MB/sec), 17315.0 ms avg latency, 20035.0 ms max latency.
11425 records sent, 2236.7 records/sec (2.18 MB/sec), 17462.0 ms avg latency, 21063.0 ms max latency.
18440 records sent, 3688.0 records/sec (3.60 MB/sec), 15359.6 ms avg latency, 21377.0 ms max latency.
8895 records sent, 1779.0 records/sec (1.74 MB/sec), 15366.5 ms avg latency, 18886.0 ms max latency.
19480 records sent, 3895.2 records/sec (3.80 MB/sec), 14607.4 ms avg latency, 18631.0 ms max latency.
18160 records sent, 3631.3 records/sec (3.55 MB/sec), 12985.3 ms avg latency, 16332.0 ms max latency.
13635 records sent, 2532.0 records/sec (2.47 MB/sec), 11547.1 ms avg latency, 15732.0 ms max latency.
11380 records sent, 2267.8 records/sec (2.21 MB/sec), 13726.1 ms avg latency, 16241.0 ms max latency.
11485 records sent, 2297.0 records/sec (2.24 MB/sec), 15609.3 ms avg latency, 18695.0 ms max latency.
19555 records sent, 3909.4 records/sec (3.82 MB/sec), 15754.8 ms avg latency, 18755.0 ms max latency.
19005 records sent, 3800.2 records/sec (3.71 MB/sec), 12930.4 ms avg latency, 18257.0 ms max latency.
14490 records sent, 2898.0 records/sec (2.83 MB/sec), 11521.0 ms avg latency, 14579.0 ms max latency.
8635 records sent, 1707.9 records/sec (1.67 MB/sec), 12192.3 ms avg latency, 16800.0 ms max latency.
14115 records sent, 2821.9 records/sec (2.76 MB/sec), 15453.7 ms avg latency, 18169.0 ms max latency.
3270 records sent, 593.4 records/sec (0.58 MB/sec), 16770.4 ms avg latency, 22056.0 ms max latency.
8860 records sent, 1771.6 records/sec (1.73 MB/sec), 21985.4 ms avg latency, 25516.0 ms max latency.
5100 records sent, 893.5 records/sec (0.87 MB/sec), 23767.9 ms avg latency, 28783.0 ms max latency.
10850 records sent, 2169.6 records/sec (2.12 MB/sec), 26645.7 ms avg latency, 30916.0 ms max latency.
13865 records sent, 2772.4 records/sec (2.71 MB/sec), 24716.9 ms avg latency, 31303.0 ms max latency.
14810 records sent, 2961.4 records/sec (2.89 MB/sec), 20441.3 ms avg latency, 28759.0 ms max latency.
11940 records sent, 2387.5 records/sec (2.33 MB/sec), 16120.2 ms avg latency, 23020.0 ms max latency.
14825 records sent, 2964.4 records/sec (2.89 MB/sec), 15083.3 ms avg latency, 19678.0 ms max latency.
18595 records sent, 3718.3 records/sec (3.63 MB/sec), 14100.0 ms avg latency, 18596.0 ms max latency.
18875 records sent, 3773.5 records/sec (3.69 MB/sec), 12222.4 ms avg latency, 17176.0 ms max latency.
16460 records sent, 3292.0 records/sec (3.21 MB/sec), 11431.9 ms avg latency, 14716.0 ms max latency.
16525 records sent, 3304.3 records/sec (3.23 MB/sec), 11643.2 ms avg latency, 15210.0 ms max latency.
9965 records sent, 1992.2 records/sec (1.95 MB/sec), 12800.1 ms avg latency, 17535.0 ms max latency.
16690 records sent, 3338.0 records/sec (3.26 MB/sec), 14486.6 ms avg latency, 18151.0 ms max latency.
18990 records sent, 3797.2 records/sec (3.71 MB/sec), 13429.0 ms avg latency, 17562.0 ms max latency.
17485 records sent, 3496.3 records/sec (3.41 MB/sec), 12154.8 ms avg latency, 17370.0 ms max latency.
13895 records sent, 2779.0 records/sec (2.71 MB/sec), 11770.3 ms avg latency, 15586.0 ms max latency.
18105 records sent, 3621.0 records/sec (3.54 MB/sec), 12309.1 ms avg latency, 16004.0 ms max latency.
18585 records sent, 3716.3 records/sec (3.63 MB/sec), 12029.0 ms avg latency, 15986.0 ms max latency.
18460 records sent, 3692.0 records/sec (3.61 MB/sec), 11503.8 ms avg latency, 15825.0 ms max latency.
18690 records sent, 3738.0 records/sec (3.65 MB/sec), 10980.4 ms avg latency, 14475.0 ms max latency.
19020 records sent, 3802.5 records/sec (3.71 MB/sec), 10954.2 ms avg latency, 14352.0 ms max latency.
15550 records sent, 3110.0 records/sec (3.04 MB/sec), 11330.4 ms avg latency, 15046.0 ms max latency.
19175 records sent, 3835.0 records/sec (3.75 MB/sec), 11550.4 ms avg latency, 15060.0 ms max latency.
18460 records sent, 3692.0 records/sec (3.61 MB/sec), 11346.3 ms avg latency, 15059.0 ms max latency.
17265 records sent, 3366.2 records/sec (3.29 MB/sec), 11289.1 ms avg latency, 15053.0 ms max latency.
13965 records sent, 2791.9 records/sec (2.73 MB/sec), 13612.8 ms avg latency, 14585.0 ms max latency.
1000000 records sent, 2989.223848 records/sec (2.92 MB/sec), 13285.68 ms avg latency, 31303.00 ms max latency, 12871 ms 50th, 20544 ms 95th, 28703 ms 99th, 31059 ms 99.9th.
3）默认的压缩方式是 none。本次实验 compression.type 设置为 gzip。
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=4096 linger.ms=50 compression.type=gzip
查看结果：
17381 records sent, 3474.8 records/sec (3.39 MB/sec), 1837.5 ms avg latency, 3293.0 ms max latency.
22435 records sent, 4487.0 records/sec (4.38 MB/sec), 4599.7 ms avg latency, 6042.0 ms max latency.
22145 records sent, 4429.0 records/sec (4.33 MB/sec), 7352.7 ms avg latency, 8869.0 ms max latency.
20310 records sent, 4062.0 records/sec (3.97 MB/sec), 9407.7 ms avg latency, 9860.0 ms max latency.
21365 records sent, 4273.0 records/sec (4.17 MB/sec), 9866.9 ms avg latency, 10166.0 ms max latency.
20680 records sent, 4136.0 records/sec (4.04 MB/sec), 9851.5 ms avg latency, 10112.0 ms max latency.
17120 records sent, 3418.5 records/sec (3.34 MB/sec), 10080.1 ms avg latency, 10687.0 ms max latency.
20260 records sent, 4052.0 records/sec (3.96 MB/sec), 10812.2 ms avg latency, 10968.0 ms max latency.
19940 records sent, 3987.2 records/sec (3.89 MB/sec), 10689.6 ms avg latency, 11055.0 ms max latency.
20375 records sent, 4073.4 records/sec (3.98 MB/sec), 10144.2 ms avg latency, 10466.0 ms max latency.
20270 records sent, 4053.2 records/sec (3.96 MB/sec), 10141.6 ms avg latency, 10471.0 ms max latency.
19910 records sent, 3981.2 records/sec (3.89 MB/sec), 10152.4 ms avg latency, 10396.0 ms max latency.
15740 records sent, 3147.4 records/sec (3.07 MB/sec), 10490.1 ms avg latency, 11659.0 ms max latency.
20190 records sent, 4038.0 records/sec (3.94 MB/sec), 11349.3 ms avg latency, 11806.0 ms max latency.
19235 records sent, 3847.0 records/sec (3.76 MB/sec), 11169.5 ms avg latency, 11710.0 ms max latency.
19930 records sent, 3986.0 records/sec (3.89 MB/sec), 10389.8 ms avg latency, 11275.0 ms max latency.
18930 records sent, 3785.2 records/sec (3.70 MB/sec), 10447.1 ms avg latency, 10948.0 ms max latency.
19005 records sent, 3801.0 records/sec (3.71 MB/sec), 10610.1 ms avg latency, 11176.0 ms max latency.
16615 records sent, 3323.0 records/sec (3.25 MB/sec), 11027.6 ms avg latency, 12115.0 ms max latency.
19645 records sent, 3929.0 records/sec (3.84 MB/sec), 11389.6 ms avg latency, 12181.0 ms max latency.
19790 records sent, 3958.0 records/sec (3.87 MB/sec), 11043.7 ms avg latency, 11892.0 ms max latency.
19305 records sent, 3861.0 records/sec (3.77 MB/sec), 10448.0 ms avg latency, 11061.0 ms max latency.
19805 records sent, 3961.0 records/sec (3.87 MB/sec), 10458.4 ms avg latency, 11082.0 ms max latency.
19095 records sent, 3819.0 records/sec (3.73 MB/sec), 10443.9 ms avg latency, 11263.0 ms max latency.
16340 records sent, 3268.0 records/sec (3.19 MB/sec), 10691.2 ms avg latency, 12082.0 ms max latency.
17210 records sent, 3442.0 records/sec (3.36 MB/sec), 11913.1 ms avg latency, 12763.0 ms max latency.
19390 records sent, 3870.3 records/sec (3.78 MB/sec), 11828.7 ms avg latency, 12632.0 ms max latency.
20090 records sent, 4018.0 records/sec (3.92 MB/sec), 10775.1 ms avg latency, 12481.0 ms max latency.
19545 records sent, 3908.2 records/sec (3.82 MB/sec), 10402.3 ms avg latency, 11258.0 ms max latency.
18860 records sent, 3772.0 records/sec (3.68 MB/sec), 10429.6 ms avg latency, 11441.0 ms max latency.
16865 records sent, 3373.0 records/sec (3.29 MB/sec), 10816.8 ms avg latency, 12192.0 ms max latency.
18675 records sent, 3733.5 records/sec (3.65 MB/sec), 11526.1 ms avg latency, 12512.0 ms max latency.
19470 records sent, 3894.0 records/sec (3.80 MB/sec), 11259.7 ms avg latency, 12391.0 ms max latency.
18980 records sent, 3789.2 records/sec (3.70 MB/sec), 10636.5 ms avg latency, 12071.0 ms max latency.
19610 records sent, 3922.0 records/sec (3.83 MB/sec), 10586.6 ms avg latency, 11823.0 ms max latency.
20050 records sent, 4010.0 records/sec (3.92 MB/sec), 10440.3 ms avg latency, 11947.0 ms max latency.
16365 records sent, 3271.7 records/sec (3.20 MB/sec), 10609.9 ms avg latency, 12658.0 ms max latency.
18675 records sent, 3734.3 records/sec (3.65 MB/sec), 11441.6 ms avg latency, 12958.0 ms max latency.
19680 records sent, 3936.0 records/sec (3.84 MB/sec), 11274.5 ms avg latency, 12919.0 ms max latency.
18200 records sent, 3640.0 records/sec (3.55 MB/sec), 10865.8 ms avg latency, 13150.0 ms max latency.
19905 records sent, 3979.4 records/sec (3.89 MB/sec), 10707.9 ms avg latency, 12423.0 ms max latency.
17975 records sent, 3595.0 records/sec (3.51 MB/sec), 10485.7 ms avg latency, 12423.0 ms max latency.
16095 records sent, 3219.0 records/sec (3.14 MB/sec), 11102.8 ms avg latency, 13343.0 ms max latency.
19360 records sent, 3871.2 records/sec (3.78 MB/sec), 11699.2 ms avg latency, 13646.0 ms max latency.
19685 records sent, 3937.0 records/sec (3.84 MB/sec), 11264.6 ms avg latency, 13737.0 ms max latency.
19320 records sent, 3864.0 records/sec (3.77 MB/sec), 10519.4 ms avg latency, 13157.0 ms max latency.
19685 records sent, 3937.0 records/sec (3.84 MB/sec), 10428.0 ms avg latency, 12520.0 ms max latency.
19670 records sent, 3934.0 records/sec (3.84 MB/sec), 10439.9 ms avg latency, 12603.0 ms max latency.
15575 records sent, 3115.0 records/sec (3.04 MB/sec), 10729.4 ms avg latency, 13580.0 ms max latency.
18760 records sent, 3752.0 records/sec (3.66 MB/sec), 11609.0 ms avg latency, 14031.0 ms max latency.
20405 records sent, 4080.2 records/sec (3.98 MB/sec), 11274.6 ms avg latency, 14019.0 ms max latency.
18215 records sent, 3643.0 records/sec (3.56 MB/sec), 10964.4 ms avg latency, 13829.0 ms max latency.
1000000 records sent, 3796.536041 records/sec (3.71 MB/sec), 10388.50 ms avg latency, 14031.00 ms max latency, 10781 ms 50th, 12915 ms 95th, 13784 ms 99th, 13992 ms 99.9th.
4）默认的压缩方式是 none。本次实验 compression.type 设置为 lz4。
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=4096 linger.ms=50 compression.type=lz4
输出结果：
11662 records sent, 2332.4 records/sec (2.28 MB/sec), 2245.5 ms avg latency, 3872.0 ms max latency.
11988 records sent, 2397.1 records/sec (2.34 MB/sec), 5439.1 ms avg latency, 7660.0 ms max latency.
10935 records sent, 2187.0 records/sec (2.14 MB/sec), 9488.4 ms avg latency, 10811.0 ms max latency.
10383 records sent, 2076.2 records/sec (2.03 MB/sec), 11270.1 ms avg latency, 12017.0 ms max latency.
10857 records sent, 2171.4 records/sec (2.12 MB/sec), 11529.3 ms avg latency, 11996.0 ms max latency.
11499 records sent, 2299.8 records/sec (2.25 MB/sec), 11405.1 ms avg latency, 11836.0 ms max latency.
12147 records sent, 2429.4 records/sec (2.37 MB/sec), 10673.2 ms avg latency, 11315.0 ms max latency.
11727 records sent, 2344.5 records/sec (2.29 MB/sec), 10108.1 ms avg latency, 10569.0 ms max latency.
10938 records sent, 2187.6 records/sec (2.14 MB/sec), 10777.2 ms avg latency, 11222.0 ms max latency.
12345 records sent, 2468.5 records/sec (2.41 MB/sec), 10814.2 ms avg latency, 10951.0 ms max latency.
12579 records sent, 2515.3 records/sec (2.46 MB/sec), 10067.6 ms avg latency, 10798.0 ms max latency.
12366 records sent, 2472.7 records/sec (2.41 MB/sec), 9910.9 ms avg latency, 10084.0 ms max latency.
11952 records sent, 2389.9 records/sec (2.33 MB/sec), 9971.1 ms avg latency, 10212.0 ms max latency.
11424 records sent, 2283.9 records/sec (2.23 MB/sec), 10162.9 ms avg latency, 10631.0 ms max latency.
10335 records sent, 2066.6 records/sec (2.02 MB/sec), 11048.7 ms avg latency, 11648.0 ms max latency.
12951 records sent, 2589.7 records/sec (2.53 MB/sec), 10982.2 ms avg latency, 11568.0 ms max latency.
12525 records sent, 2505.0 records/sec (2.45 MB/sec), 9983.3 ms avg latency, 11201.0 ms max latency.
12348 records sent, 2469.6 records/sec (2.41 MB/sec), 9792.4 ms avg latency, 10363.0 ms max latency.
11661 records sent, 2332.2 records/sec (2.28 MB/sec), 9991.5 ms avg latency, 10562.0 ms max latency.
11058 records sent, 2211.6 records/sec (2.16 MB/sec), 10370.0 ms avg latency, 10994.0 ms max latency.
11316 records sent, 2263.2 records/sec (2.21 MB/sec), 11030.3 ms avg latency, 11537.0 ms max latency.
12660 records sent, 2532.0 records/sec (2.47 MB/sec), 10732.6 ms avg latency, 11351.0 ms max latency.
12132 records sent, 2425.9 records/sec (2.37 MB/sec), 10013.4 ms avg latency, 10615.0 ms max latency.
12309 records sent, 2461.8 records/sec (2.40 MB/sec), 10004.7 ms avg latency, 10405.0 ms max latency.
10797 records sent, 2158.5 records/sec (2.11 MB/sec), 10245.9 ms avg latency, 10752.0 ms max latency.
11127 records sent, 2225.0 records/sec (2.17 MB/sec), 10713.4 ms avg latency, 11301.0 ms max latency.
11616 records sent, 2323.2 records/sec (2.27 MB/sec), 11120.3 ms avg latency, 11638.0 ms max latency.
12318 records sent, 2462.1 records/sec (2.40 MB/sec), 10601.4 ms avg latency, 11100.0 ms max latency.
12417 records sent, 2482.9 records/sec (2.42 MB/sec), 10025.5 ms avg latency, 10751.0 ms max latency.
10677 records sent, 2135.4 records/sec (2.09 MB/sec), 10271.7 ms avg latency, 11057.0 ms max latency.
12423 records sent, 2483.6 records/sec (2.43 MB/sec), 10601.8 ms avg latency, 11088.0 ms max latency.
11403 records sent, 2280.6 records/sec (2.23 MB/sec), 10461.1 ms avg latency, 10971.0 ms max latency.
10935 records sent, 2187.0 records/sec (2.14 MB/sec), 10752.5 ms avg latency, 11464.0 ms max latency.
12477 records sent, 2495.4 records/sec (2.44 MB/sec), 10836.1 ms avg latency, 11478.0 ms max latency.
11934 records sent, 2386.8 records/sec (2.33 MB/sec), 10187.7 ms avg latency, 11148.0 ms max latency.
12180 records sent, 2435.5 records/sec (2.38 MB/sec), 10200.5 ms avg latency, 10594.0 ms max latency.
12552 records sent, 2510.4 records/sec (2.45 MB/sec), 10087.5 ms avg latency, 10447.0 ms max latency.
11553 records sent, 2310.6 records/sec (2.26 MB/sec), 9934.3 ms avg latency, 10577.0 ms max latency.
11415 records sent, 2283.0 records/sec (2.23 MB/sec), 10525.8 ms avg latency, 11108.0 ms max latency.
12585 records sent, 2516.5 records/sec (2.46 MB/sec), 10482.4 ms avg latency, 11081.0 ms max latency.
12231 records sent, 2446.2 records/sec (2.39 MB/sec), 9969.4 ms avg latency, 11042.0 ms max latency.
12351 records sent, 2469.7 records/sec (2.41 MB/sec), 9979.8 ms avg latency, 10733.0 ms max latency.
11067 records sent, 2213.4 records/sec (2.16 MB/sec), 10157.7 ms avg latency, 11192.0 ms max latency.
10614 records sent, 2122.8 records/sec (2.07 MB/sec), 10716.5 ms avg latency, 11921.0 ms max latency.
9822 records sent, 1964.4 records/sec (1.92 MB/sec), 11771.5 ms avg latency, 12960.0 ms max latency.
11940 records sent, 2388.0 records/sec (2.33 MB/sec), 11728.0 ms avg latency, 12768.0 ms max latency.
11913 records sent, 2382.6 records/sec (2.33 MB/sec), 10698.8 ms avg latency, 12270.0 ms max latency.
11625 records sent, 2325.0 records/sec (2.27 MB/sec), 10330.2 ms avg latency, 11146.0 ms max latency.
11868 records sent, 2373.6 records/sec (2.32 MB/sec), 10500.5 ms avg latency, 11367.0 ms max latency.
10983 records sent, 2196.2 records/sec (2.14 MB/sec), 10481.0 ms avg latency, 11448.0 ms max latency.
11217 records sent, 2243.0 records/sec (2.19 MB/sec), 11007.2 ms avg latency, 11865.0 ms max latency.
12534 records sent, 2506.3 records/sec (2.45 MB/sec), 10786.5 ms avg latency, 11870.0 ms max latency.
11964 records sent, 2392.8 records/sec (2.34 MB/sec), 10101.4 ms avg latency, 11612.0 ms max latency.
11919 records sent, 2383.8 records/sec (2.33 MB/sec), 10129.9 ms avg latency, 11102.0 ms max latency.
11295 records sent, 2259.0 records/sec (2.21 MB/sec), 10464.7 ms avg latency, 11256.0 ms max latency.
10254 records sent, 2050.8 records/sec (2.00 MB/sec), 10914.5 ms avg latency, 12117.0 ms max latency.
10713 records sent, 2142.2 records/sec (2.09 MB/sec), 11594.9 ms avg latency, 12510.0 ms max latency.
11940 records sent, 2386.6 records/sec (2.33 MB/sec), 11297.4 ms avg latency, 12459.0 ms max latency.
11985 records sent, 2397.0 records/sec (2.34 MB/sec), 10569.0 ms avg latency, 12096.0 ms max latency.
11709 records sent, 2341.8 records/sec (2.29 MB/sec), 10363.6 ms avg latency, 11403.0 ms max latency.
11940 records sent, 2388.0 records/sec (2.33 MB/sec), 10318.3 ms avg latency, 11419.0 ms max latency.
10227 records sent, 2045.0 records/sec (2.00 MB/sec), 10553.3 ms avg latency, 11972.0 ms max latency.
10374 records sent, 2074.8 records/sec (2.03 MB/sec), 11584.8 ms avg latency, 12765.0 ms max latency.
12498 records sent, 2499.6 records/sec (2.44 MB/sec), 11383.7 ms avg latency, 12768.0 ms max latency.
11448 records sent, 2289.1 records/sec (2.24 MB/sec), 10336.6 ms avg latency, 12376.0 ms max latency.
11031 records sent, 2206.2 records/sec (2.15 MB/sec), 10568.2 ms avg latency, 12012.0 ms max latency.
11322 records sent, 2264.4 records/sec (2.21 MB/sec), 10929.4 ms avg latency, 12329.0 ms max latency.
10608 records sent, 2120.3 records/sec (2.07 MB/sec), 11029.0 ms avg latency, 12459.0 ms max latency.
10629 records sent, 2125.8 records/sec (2.08 MB/sec), 11467.8 ms avg latency, 12756.0 ms max latency.
11415 records sent, 2282.5 records/sec (2.23 MB/sec), 11424.1 ms avg latency, 12572.0 ms max latency.
10659 records sent, 2130.5 records/sec (2.08 MB/sec), 11178.6 ms avg latency, 12514.0 ms max latency.
10329 records sent, 2065.0 records/sec (2.02 MB/sec), 11330.5 ms avg latency, 12534.0 ms max latency.
10089 records sent, 2017.0 records/sec (1.97 MB/sec), 11601.9 ms avg latency, 12978.0 ms max latency.
9888 records sent, 1976.8 records/sec (1.93 MB/sec), 12098.9 ms avg latency, 13609.0 ms max latency.
11349 records sent, 2269.3 records/sec (2.22 MB/sec), 12248.9 ms avg latency, 13680.0 ms max latency.
10272 records sent, 2054.4 records/sec (2.01 MB/sec), 11390.4 ms avg latency, 12762.0 ms max latency.
8874 records sent, 1774.4 records/sec (1.73 MB/sec), 12060.4 ms avg latency, 13399.0 ms max latency.
11070 records sent, 2213.6 records/sec (2.16 MB/sec), 12334.8 ms avg latency, 13699.0 ms max latency.
11016 records sent, 2202.8 records/sec (2.15 MB/sec), 11849.9 ms avg latency, 13833.0 ms max latency.
10941 records sent, 2187.8 records/sec (2.14 MB/sec), 11006.1 ms avg latency, 12417.0 ms max latency.
9660 records sent, 1932.0 records/sec (1.89 MB/sec), 11620.1 ms avg latency, 13239.0 ms max latency.
10596 records sent, 2118.4 records/sec (2.07 MB/sec), 11911.1 ms avg latency, 13223.0 ms max latency.
11940 records sent, 2388.0 records/sec (2.33 MB/sec), 11615.8 ms avg latency, 13384.0 ms max latency.
10758 records sent, 2151.6 records/sec (2.10 MB/sec), 10829.3 ms avg latency, 12205.0 ms max latency.
9966 records sent, 1993.2 records/sec (1.95 MB/sec), 11163.0 ms avg latency, 12657.0 ms max latency.
9333 records sent, 1852.9 records/sec (1.81 MB/sec), 12075.6 ms avg latency, 13872.0 ms max latency.
9315 records sent, 1860.8 records/sec (1.82 MB/sec), 12996.5 ms avg latency, 14661.0 ms max latency.
9912 records sent, 1982.0 records/sec (1.94 MB/sec), 13130.8 ms avg latency, 14636.0 ms max latency.
1000000 records sent, 2263.380540 records/sec (2.21 MB/sec), 10651.82 ms avg latency, 14661.00 ms max latency, 10606 ms 50th, 13012 ms 95th, 13876 ms 99th, 14604 ms 99.9th.
（6）调整缓存大小
[atguigu@hadoop105 kafka]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=hadoop102:9092,hadoop103:9092,hadoop104:9092 batch.size=4096 linger.ms=50 buffer.memory=67108864
输出结果：
13726 records sent, 2744.1 records/sec (2.68 MB/sec), 1845.8 ms avg latency, 3655.0 ms max latency.
14046 records sent, 2807.5 records/sec (2.74 MB/sec), 5500.9 ms avg latency, 7262.0 ms max latency.
13149 records sent, 2629.8 records/sec (2.57 MB/sec), 9109.2 ms avg latency, 10945.0 ms max latency.
13122 records sent, 2624.4 records/sec (2.56 MB/sec), 12674.1 ms avg latency, 14622.0 ms max latency.
11073 records sent, 2214.6 records/sec (2.16 MB/sec), 16615.1 ms avg latency, 18510.0 ms max latency.
12432 records sent, 2486.4 records/sec (2.43 MB/sec), 19348.5 ms avg latency, 20063.0 ms max latency.
10794 records sent, 2158.4 records/sec (2.11 MB/sec), 20061.6 ms avg latency, 20865.0 ms max latency.
10248 records sent, 2049.6 records/sec (2.00 MB/sec), 21404.3 ms avg latency, 22046.0 ms max latency.
10755 records sent, 2150.6 records/sec (2.10 MB/sec), 22108.8 ms avg latency, 22557.0 ms max latency.
10167 records sent, 2033.0 records/sec (1.99 MB/sec), 22484.5 ms avg latency, 23215.0 ms max latency.
11094 records sent, 2218.8 records/sec (2.17 MB/sec), 23273.7 ms avg latency, 23744.0 ms max latency.
11016 records sent, 2202.8 records/sec (2.15 MB/sec), 23229.3 ms avg latency, 23773.0 ms max latency.
10650 records sent, 2129.6 records/sec (2.08 MB/sec), 22709.0 ms avg latency, 23376.0 ms max latency.
7872 records sent, 1574.1 records/sec (1.54 MB/sec), 23579.5 ms avg latency, 24532.0 ms max latency.
11535 records sent, 2302.9 records/sec (2.25 MB/sec), 23794.7 ms avg latency, 24498.0 ms max latency.
11436 records sent, 2287.2 records/sec (2.23 MB/sec), 23480.9 ms avg latency, 23916.0 ms max latency.
11259 records sent, 2251.8 records/sec (2.20 MB/sec), 23413.1 ms avg latency, 23747.0 ms max latency.
11394 records sent, 2278.3 records/sec (2.22 MB/sec), 23155.0 ms avg latency, 23857.0 ms max latency.
10230 records sent, 2046.0 records/sec (2.00 MB/sec), 21911.0 ms avg latency, 23124.0 ms max latency.
9378 records sent, 1875.2 records/sec (1.83 MB/sec), 22665.2 ms avg latency, 23691.0 ms max latency.
11472 records sent, 2294.4 records/sec (2.24 MB/sec), 23070.0 ms avg latency, 23929.0 ms max latency.
11385 records sent, 2277.0 records/sec (2.22 MB/sec), 22816.5 ms avg latency, 23810.0 ms max latency.
12213 records sent, 2442.1 records/sec (2.38 MB/sec), 22551.3 ms avg latency, 23730.0 ms max latency.
8235 records sent, 1647.0 records/sec (1.61 MB/sec), 22830.8 ms avg latency, 25038.0 ms max latency.
10101 records sent, 2020.2 records/sec (1.97 MB/sec), 23001.7 ms avg latency, 24938.0 ms max latency.
9267 records sent, 1853.0 records/sec (1.81 MB/sec), 23686.2 ms avg latency, 25363.0 ms max latency.
10593 records sent, 2118.6 records/sec (2.07 MB/sec), 24356.1 ms avg latency, 25812.0 ms max latency.
10395 records sent, 2078.6 records/sec (2.03 MB/sec), 24948.4 ms avg latency, 26349.0 ms max latency.
9477 records sent, 1895.4 records/sec (1.85 MB/sec), 24976.4 ms avg latency, 27345.0 ms max latency.
10446 records sent, 2089.2 records/sec (2.04 MB/sec), 24476.0 ms avg latency, 26125.0 ms max latency.
9798 records sent, 1959.2 records/sec (1.91 MB/sec), 24146.3 ms avg latency, 25984.0 ms max latency.
10116 records sent, 2022.8 records/sec (1.98 MB/sec), 24489.6 ms avg latency, 26083.0 ms max latency.
11904 records sent, 2380.8 records/sec (2.33 MB/sec), 23841.7 ms avg latency, 25769.0 ms max latency.
12024 records sent, 2381.5 records/sec (2.33 MB/sec), 22898.2 ms avg latency, 25282.0 ms max latency.
11874 records sent, 2374.8 records/sec (2.32 MB/sec), 22171.2 ms avg latency, 24184.0 ms max latency.
11859 records sent, 2371.3 records/sec (2.32 MB/sec), 21213.3 ms avg latency, 23580.0 ms max latency.
10512 records sent, 2102.0 records/sec (2.05 MB/sec), 20811.5 ms avg latency, 23016.0 ms max latency.
9687 records sent, 1937.4 records/sec (1.89 MB/sec), 21979.6 ms avg latency, 24179.0 ms max latency.
11883 records sent, 2376.6 records/sec (2.32 MB/sec), 22237.9 ms avg latency, 24157.0 ms max latency.
12054 records sent, 2410.3 records/sec (2.35 MB/sec), 22143.4 ms avg latency, 24031.0 ms max latency.
11409 records sent, 2280.9 records/sec (2.23 MB/sec), 21976.1 ms avg latency, 24298.0 ms max latency.
10470 records sent, 2093.6 records/sec (2.04 MB/sec), 21705.7 ms avg latency, 24489.0 ms max latency.
9144 records sent, 1828.8 records/sec (1.79 MB/sec), 21790.5 ms avg latency, 24874.0 ms max latency.
9411 records sent, 1881.8 records/sec (1.84 MB/sec), 23173.7 ms avg latency, 25942.0 ms max latency.
11958 records sent, 2391.1 records/sec (2.34 MB/sec), 23500.0 ms avg latency, 26003.0 ms max latency.
11187 records sent, 2237.4 records/sec (2.18 MB/sec), 23507.3 ms avg latency, 26357.0 ms max latency.
11580 records sent, 2315.1 records/sec (2.26 MB/sec), 23028.5 ms avg latency, 26227.0 ms max latency.
10476 records sent, 2095.2 records/sec (2.05 MB/sec), 22561.3 ms avg latency, 26044.0 ms max latency.
8637 records sent, 1726.4 records/sec (1.69 MB/sec), 22505.4 ms avg latency, 25969.0 ms max latency.
10356 records sent, 2070.4 records/sec (2.02 MB/sec), 23484.1 ms avg latency, 26538.0 ms max latency.
10275 records sent, 2054.6 records/sec (2.01 MB/sec), 23917.9 ms avg latency, 27193.0 ms max latency.
11445 records sent, 2288.1 records/sec (2.23 MB/sec), 24173.0 ms avg latency, 27195.0 ms max latency.
11487 records sent, 2297.4 records/sec (2.24 MB/sec), 23654.3 ms avg latency, 27039.0 ms max latency.
11049 records sent, 2209.4 records/sec (2.16 MB/sec), 22814.6 ms avg latency, 26174.0 ms max latency.
10020 records sent, 2003.6 records/sec (1.96 MB/sec), 22132.0 ms avg latency, 25275.0 ms max latency.
10752 records sent, 2150.4 records/sec (2.10 MB/sec), 22790.2 ms avg latency, 25647.0 ms max latency.
12048 records sent, 2409.1 records/sec (2.35 MB/sec), 22436.7 ms avg latency, 24974.0 ms max latency.
10332 records sent, 2066.0 records/sec (2.02 MB/sec), 22476.5 ms avg latency, 24893.0 ms max latency.
10053 records sent, 2010.2 records/sec (1.96 MB/sec), 23146.1 ms avg latency, 25343.0 ms max latency.
9882 records sent, 1976.4 records/sec (1.93 MB/sec), 23194.1 ms avg latency, 25801.0 ms max latency.
9717 records sent, 1943.4 records/sec (1.90 MB/sec), 23276.0 ms avg latency, 25658.0 ms max latency.
7713 records sent, 1542.6 records/sec (1.51 MB/sec), 24735.9 ms avg latency, 27510.0 ms max latency.
10575 records sent, 2115.0 records/sec (2.07 MB/sec), 25545.7 ms avg latency, 28258.0 ms max latency.
7902 records sent, 1580.4 records/sec (1.54 MB/sec), 26185.5 ms avg latency, 29432.0 ms max latency.
10380 records sent, 2075.6 records/sec (2.03 MB/sec), 26354.2 ms avg latency, 29291.0 ms max latency.
9927 records sent, 1985.4 records/sec (1.94 MB/sec), 26230.4 ms avg latency, 29454.0 ms max latency.
9216 records sent, 1843.2 records/sec (1.80 MB/sec), 26263.3 ms avg latency, 29894.0 ms max latency.
8904 records sent, 1780.4 records/sec (1.74 MB/sec), 26126.5 ms avg latency, 29876.0 ms max latency.
11133 records sent, 2205.4 records/sec (2.15 MB/sec), 25630.7 ms avg latency, 28871.0 ms max latency.
9477 records sent, 1895.4 records/sec (1.85 MB/sec), 24998.1 ms avg latency, 28189.0 ms max latency.
11589 records sent, 2317.8 records/sec (2.26 MB/sec), 24346.9 ms avg latency, 28116.0 ms max latency.
9813 records sent, 1962.2 records/sec (1.92 MB/sec), 24143.1 ms avg latency, 28008.0 ms max latency.
9198 records sent, 1839.2 records/sec (1.80 MB/sec), 24085.7 ms avg latency, 28016.0 ms max latency.
8406 records sent, 1680.9 records/sec (1.64 MB/sec), 24717.4 ms avg latency, 28685.0 ms max latency.
10308 records sent, 2061.6 records/sec (2.01 MB/sec), 25210.3 ms avg latency, 28878.0 ms max latency.
9897 records sent, 1978.2 records/sec (1.93 MB/sec), 25056.0 ms avg latency, 29171.0 ms max latency.
10482 records sent, 2096.4 records/sec (2.05 MB/sec), 25239.9 ms avg latency, 29952.0 ms max latency.
10098 records sent, 2019.2 records/sec (1.97 MB/sec), 25000.2 ms avg latency, 29607.0 ms max latency.
9090 records sent, 1818.0 records/sec (1.78 MB/sec), 24939.3 ms avg latency, 29766.0 ms max latency.
8583 records sent, 1716.3 records/sec (1.68 MB/sec), 25247.0 ms avg latency, 29566.0 ms max latency.
10236 records sent, 2033.4 records/sec (1.99 MB/sec), 25462.3 ms avg latency, 30078.0 ms max latency.
10275 records sent, 2055.0 records/sec (2.01 MB/sec), 25238.1 ms avg latency, 30131.0 ms max latency.
10164 records sent, 2032.8 records/sec (1.99 MB/sec), 25370.6 ms avg latency, 30241.0 ms max latency.
8520 records sent, 1695.2 records/sec (1.66 MB/sec), 25812.5 ms avg latency, 31267.0 ms max latency.
8580 records sent, 1714.6 records/sec (1.67 MB/sec), 25836.2 ms avg latency, 31657.0 ms max latency.
8922 records sent, 1784.4 records/sec (1.74 MB/sec), 26423.2 ms avg latency, 32017.0 ms max latency.
9696 records sent, 1939.2 records/sec (1.89 MB/sec), 26471.1 ms avg latency, 31918.0 ms max latency.
10530 records sent, 2105.6 records/sec (2.06 MB/sec), 26276.4 ms avg latency, 31964.0 ms max latency.
11823 records sent, 2364.6 records/sec (2.31 MB/sec), 25278.3 ms avg latency, 31825.0 ms max latency.
11100 records sent, 2220.0 records/sec (2.17 MB/sec), 24127.5 ms avg latency, 31136.0 ms max latency.
10254 records sent, 2049.6 records/sec (2.00 MB/sec), 23024.7 ms avg latency, 29940.0 ms max latency.
9549 records sent, 1909.0 records/sec (1.86 MB/sec), 23313.4 ms avg latency, 29168.0 ms max latency.
9756 records sent, 1950.8 records/sec (1.91 MB/sec), 23680.6 ms avg latency, 29321.0 ms max latency.
11841 records sent, 2368.2 records/sec (2.31 MB/sec), 23541.8 ms avg latency, 29345.0 ms max latency.
9780 records sent, 1956.0 records/sec (1.91 MB/sec), 25696.0 ms avg latency, 28761.0 ms max latency.
1000000 records sent, 2085.997326 records/sec (2.04 MB/sec), 22712.75 ms avg latency, 32017.00 ms max latency, 23490 ms 50th, 29417 ms 95th, 31445 ms 99th, 31900 ms 99.9th.
3）Kafka Consumer 压力测试
（1）修改/opt/module/kafka/config/consumer.properties 文件中的一次拉取条数为 500
max.poll.records=500
（2）消费 100 万条日志进行压测
[atguigu@hadoop105 kafka]$ ./bin/kafka-consumer-perf-test.sh --bootstrap-server hadoop102:9092,hadoop103:9092,hadoop104:9092 --topic test --messages 1000000 --consumer.config config/consumer.properties
参数说明：
- --bootstrap-server 指定 Kafka 集群地址
- --topic 指定 topic 的名称
- --messages 总共要消费的消息个数。本次实验 100 万条。
输出结果：
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-01-20 09:58:26:171, 2022-01-20 09:58:33:321, 977.0166, 136.6457, 1000465, 139925.1748, 415, 6735, 145.0656, 148547.1418
（3）一次拉取条数为 2000
1）修改/opt/module/kafka/config/consumer.properties 文件中的一次拉取条数为 2000
max.poll.records=2000
2）再次执行
[atguigu@hadoop105 kafka]$ ./bin/kafka-consumer-perf-test.sh --bootstrap-server hadoop102:9092,hadoop103:9092,hadoop104:9092 --topic test --messages 1000000 --consumer.config config/consumer.properties
输出结果：
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-01-20 10:18:06:268, 2022-01-20 10:18:12:863, 977.5146, 148.2206, 1000975, 151777.8620, 358, 6237, 156.7283, 160489.8188
（4）调整 fetch.max.bytes 大小为 100m
1）修改/opt/module/kafka/config/consumer.properties 文件中的拉取一批数据大小 100m
fetch.max.bytes=104857600
2）再次执行
[atguigu@hadoop105 kafka]$ ./bin/kafka-consumer-perf-test.sh --bootstrap-server hadoop102:9092,hadoop103:9092,hadoop104:9092 --topic test --messages 1000000 --consumer.config config/consumer.properties
输出结果：
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-01-20 10:26:13:203, 2022-01-20 10:26:19:662, 977.5146, 151.3415, 1000975, 154973.6801, 362, 6097, 160.3272, 164175.0041
