kafka性能测试1.0.0

kafka提供工具kafka-producer-perf-test.sh用以压测，

参数	说明
messages	生产者发送总的消息数量
message-size	每条消息大小
batch-size	每次批量发送消息的数量
topics	生产者发送的topic
threads	生产者使用几个线程同时发送
broker-list	安装kafka服务的机器ip:port列表
producer-num-retries	一个消息失败发送重试次数
request-timeout-ms	一个消息请求发送超时时间

./kafka-producer-perf-test.sh --topic log.business --num-records 1000000 --record-size 500 --throughput 1000000 --threads 100 --batch-size 4096 --producer-props bootstrap.servers=*****:9092 --sync

 

后来发现，这些参数并不适用，其中message-size，batch-size，threads，sync等参数在1.0.0中不适用，不得已根据脚本提示重新找答案

运行命令：

[udap@10 bin]$ ./kafka-producer-perf-test.sh 
usage: producer-performance [-h] --topic TOPIC --num-records NUM-RECORDS [--payload-delimiter PAYLOAD-DELIMITER] --throughput THROUGHPUT [--producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]]
                            [--producer.config CONFIG-FILE] [--print-metrics] [--transactional-id TRANSACTIONAL-ID] [--transaction-duration-ms TRANSACTION-DURATION] (--record-size RECORD-SIZE |
                            --payload-file PAYLOAD-FILE)

This tool is used to verify the producer performance.

optional arguments:
  -h, --help             show this help message and exit
  --topic TOPIC          produce messages to this topic
  --num-records NUM-RECORDS
                         number of messages to produce
  --payload-delimiter PAYLOAD-DELIMITER
                         provides delimiter to be used when --payload-file is provided. Defaults to new line. Note that this parameter will be ignored if --payload-file is not provided. (default: \n)
  --throughput THROUGHPUT
                         throttle maximum message throughput to *approximately* THROUGHPUT messages/sec
  --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]
                         kafka producer related configuration properties like bootstrap.servers,client.id etc. These configs take precedence over those passed via --producer.config.
  --producer.config CONFIG-FILE
                         producer config properties file.
  --print-metrics        print out metrics at the end of the test. (default: false)
  --transactional-id TRANSACTIONAL-ID
                         The transactionalId to use if transaction-duration-ms is > 0.  Useful  when testing the performance of concurrent transactions. (default: performance-producer-default-transactional-
                         id)
  --transaction-duration-ms TRANSACTION-DURATION
                         The max age of each transaction. The commitTransaction will be called after this this time has elapsed. Transactions are only enabled if this value is positive. (default: 0)

  either --record-size or --payload-file must be specified but not both.

  --record-size RECORD-SIZE
                         message size in bytes. Note that you must provide exactly one of --record-size or --payload-file.
  --payload-file PAYLOAD-FILE
                         file to read the message payloads from. This works only for UTF-8 encoded  text  files.  Payloads  will  be  read from this file and a payload will be randomly selected when sending
                         messages. Note that you must provide exactly one of --record-size or --payload-file.

 

根据提示得到压测命令如下： 

./kafka-producer-perf-test.sh --topic log.business --throughput 100000 --num-records 1000000 --record-size 200 --producer-props bootstrap.servers=******:9092 ack=0

 

其中参数其实脚本都有解释：

--topic topic名称，

--num-records 总共需要发送的消息数，

--record-size 每个记录的字节数，

--throughput 每秒钟发送的记录数，

--producer-props bootstrap.servers=localhost:9092 发送端的配置信息

这里面比之前版本少了好多参数，提示说明producer-props中有一些配置提供，上官网查看：http://kafka.apache.org/documentation/ 找到目录：Producer Configs 发现其中有一些有用的配置，比如acks，batch.size，ssl等，不过这些配置都有默认值，如果值不一样则可以根据提示设置参数，表格比较长，就不摘抄过来了，根据路径过去看。

by the way

kafka性能与每次写入的record size有关，每条记录的大小会严重影响性能，当我记录字节数设置为200时，单点压测性能接近50W requests per second