Mongo压测介绍

参考

https://github.com/brianfrankcooper/YCSB/

摘抄

https://blog.51cto.com/navyaijm/2421973

说明

使用YCSB对mongodb进行压测，YCSB是雅虎做的一个工具，好久没有更新了，但是目前(2021年5月14日)还能用,并且也没有别的好使的工具。

下载

下载release版，不要用源码，源码编译会下载很多已经过时的jar包，maven仓库里并没有，非常麻烦，并且可能无法成功

下载安装java和maven3

略

下载YCSB

Download the latest release of YCSB:

curl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz
tar xfvz ycsb-0.17.0.tar.gz
cd ycsb-0.17.0

下载可能依赖的jar包（可能不需要，编译使用的时候用到了，懒得再试了，写这吧）

wget http://www.allanbank.com/repo/com/allanbank/mongodb-async-driver/2.0.1/mongodb-async-driver-2.0.1.jar

mvn install:install-file -Dfile=/tmp/mongodb-async-driver-2.0.1.jar  -DgroupId=com.allanbank -DartifactId=mongodb-async-driver -Dversion=2.0.1 -Dpackaging=jar

使用

每个压测项的binding目录都有一个README.MD说明，比如mongodb的：/ycsb-0.17.0/mongodb-binding/README.MD，但是都是英文的，也可以参考一下国内博客https://www.yht7.com/news/21478

官方入门命令：

cd ycsb-0.17.0

#异步模式
./bin/ycsb load mongodb-async -s -P workloads/workloada -p mongodb.url=mongodb://localhost:27017/ycsb_test?w=0

#同步模式
./bin/ycsb load mongodb -s -P workloads/workloada -p mongodb.url=mongodb://localhost:27017/ycsb_test?w=0

这里的load执行的是插入命令，把load改成run就是读和修改操作

workloads/workloada里就是对压测细节的描述，可以直接复制一个编辑为自己想要的

copy workloada 1w
vi 1w

mongodb.url=mongodb://192.168.9.77:20000
mongodb.writeConcern=normal
table=ycsb_test
recordcount=10000
operationcount=10000
workload=site.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

其它属性说明

fieldcount: 每条记录字段个数 (default: 10)
fieldlength: 每个字段长度 (default: 100)
readallfields: 是否读取所有字段true或者读取一个字段false (default: true)
readproportion: 读取作业比例 (default: 0.95)
updateproportion: 更新作业比例 (default: 0.05)
insertproportion: 插入作业比例 (default: 0)
scanproportion: 扫描作业比例 (default: 0)
readmodifywriteproportion: 读取一条记录修改它并写回的比例 (default: 0)
requestdistribution: 请求的分布规则 uniform, zipfian or latest (default: uniform)
maxscanlength: 扫描作业最大记录数 (default: 1000)
scanlengthdistribution: 在1和最大扫描记录数的之间的分布规则 (default: uniform)
insertorder: 记录被插入的规则ordered或者hashed (default: hashed)
operationcount: 执行的操作数.
maxexecutiontime: 执行操作的最长时间，当然如果没有超过这个时间以运行时间为主。
table: 测试表的名称 (default: usertable)
recordcount: 加载到数据库的纪录条数 (default: 0)

使用(1w条比较少，实际应该调大点）：

1、造数据，测试写入性能，输出结果到文件

 ./bin/ycsb load mongodb -s -P workloads/1w -threads 100 > loadresult
 
  ./bin/ycsb load mongodb -s -P workloads/1w -threads 100 > loadresult

2、读或者读写混合

通过调整读写的比例，可以只读或者读写混合

 ./bin/ycsb run  mongodb -s -P workloads/1w -threads 100 > runresult

./bin/ycsb load mongodb -threads 100 -P workloads/1w

指标观察

1、服务器指标，主要观察CPU、内存、磁盘IO的利用率和延时，可以通过top、iostat工具查看实时情况
2、MongoDB可以通过mongostat 工具查看实时情况

mongostat的输出说明

inserts：每秒插入次数
query：每秒查询次数
update：每秒更新次数
delete：每秒删除次数
getmore：每秒执行getmore次数
command：每秒的命令数，比以上插入、查找、更新、删除的综合还多，还统计了别的命令
dirty：WiredTiger存储引擎中dirty 数据占缓存百分比
used:WiredTiger存储引擎中引擎使用缓存占百分比
flushs：每秒执行fsync将数据写入硬盘的次数。
vsize：虚拟内存使用量，单位MB
res：物理内存使用量，单位MB
qrw:客户端等待读的长度,队列中的长度
arw:客户端等待写的队列长度
netIn 和 netOut：网络流量，单位是字节 byte
conn：当前连接数
time：时间戳

ycsb指令参数说明

[root@fangchenmi_test3 ycsb-0.17.0]# ./bin/ycsb --help
usage: ./bin/ycsb command database [options]

Commands:
    load           Execute the load phase
    run            Execute the transaction phase
    shell          Interactive mode

Databases:
    accumulo       https://github.com/brianfrankcooper/YCSB/tree/master/accumulo
    accumulo1.6    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.6
    accumulo1.7    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.7
    accumulo1.8    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.8
    aerospike      https://github.com/brianfrankcooper/YCSB/tree/master/aerospike
    arangodb       https://github.com/brianfrankcooper/YCSB/tree/master/arangodb
    arangodb3      https://github.com/brianfrankcooper/YCSB/tree/master/arangodb3
    asynchbase     https://github.com/brianfrankcooper/YCSB/tree/master/asynchbase
    azurecosmos    https://github.com/brianfrankcooper/YCSB/tree/master/azurecosmos
    azuretablestorage https://github.com/brianfrankcooper/YCSB/tree/master/azuretablestorage
    basic          https://github.com/brianfrankcooper/YCSB/tree/master/basic
    basicts        https://github.com/brianfrankcooper/YCSB/tree/master/basicts
    cassandra-cql  https://github.com/brianfrankcooper/YCSB/tree/master/cassandra
    cassandra2-cql https://github.com/brianfrankcooper/YCSB/tree/master/cassandra2
    cloudspanner   https://github.com/brianfrankcooper/YCSB/tree/master/cloudspanner
    couchbase      https://github.com/brianfrankcooper/YCSB/tree/master/couchbase
    couchbase2     https://github.com/brianfrankcooper/YCSB/tree/master/couchbase2
    crail          https://github.com/brianfrankcooper/YCSB/tree/master/crail
    dynamodb       https://github.com/brianfrankcooper/YCSB/tree/master/dynamodb
    elasticsearch  https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch
    elasticsearch5 https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch5
    elasticsearch5-rest https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch5
    foundationdb   https://github.com/brianfrankcooper/YCSB/tree/master/foundationdb
    geode          https://github.com/brianfrankcooper/YCSB/tree/master/geode
    googlebigtable https://github.com/brianfrankcooper/YCSB/tree/master/googlebigtable
    googledatastore https://github.com/brianfrankcooper/YCSB/tree/master/googledatastore
    griddb         https://github.com/brianfrankcooper/YCSB/tree/master/griddb
    hbase098       https://github.com/brianfrankcooper/YCSB/tree/master/hbase098
    hbase10        https://github.com/brianfrankcooper/YCSB/tree/master/hbase10
    hbase12        https://github.com/brianfrankcooper/YCSB/tree/master/hbase12
    hbase14        https://github.com/brianfrankcooper/YCSB/tree/master/hbase14
    hbase20        https://github.com/brianfrankcooper/YCSB/tree/master/hbase20
    hypertable     https://github.com/brianfrankcooper/YCSB/tree/master/hypertable
    ignite         https://github.com/brianfrankcooper/YCSB/tree/master/ignite
    ignite-sql     https://github.com/brianfrankcooper/YCSB/tree/master/ignite
    infinispan     https://github.com/brianfrankcooper/YCSB/tree/master/infinispan
    infinispan-cs  https://github.com/brianfrankcooper/YCSB/tree/master/infinispan
    jdbc           https://github.com/brianfrankcooper/YCSB/tree/master/jdbc
    kudu           https://github.com/brianfrankcooper/YCSB/tree/master/kudu
    maprdb         https://github.com/brianfrankcooper/YCSB/tree/master/maprdb
    maprjsondb     https://github.com/brianfrankcooper/YCSB/tree/master/maprjsondb
    memcached      https://github.com/brianfrankcooper/YCSB/tree/master/memcached
    mongodb        https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
    mongodb-async  https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
    nosqldb        https://github.com/brianfrankcooper/YCSB/tree/master/nosqldb
    orientdb       https://github.com/brianfrankcooper/YCSB/tree/master/orientdb
    postgrenosql   https://github.com/brianfrankcooper/YCSB/tree/master/postgrenosql
    rados          https://github.com/brianfrankcooper/YCSB/tree/master/rados
    redis          https://github.com/brianfrankcooper/YCSB/tree/master/redis
    rest           https://github.com/brianfrankcooper/YCSB/tree/master/rest
    riak           https://github.com/brianfrankcooper/YCSB/tree/master/riak
    rocksdb        https://github.com/brianfrankcooper/YCSB/tree/master/rocksdb
    s3             https://github.com/brianfrankcooper/YCSB/tree/master/s3
    solr           https://github.com/brianfrankcooper/YCSB/tree/master/solr
    solr6          https://github.com/brianfrankcooper/YCSB/tree/master/solr6
    tablestore     https://github.com/brianfrankcooper/YCSB/tree/master/tablestore
    tarantool      https://github.com/brianfrankcooper/YCSB/tree/master/tarantool

Options:
    -P file        Specify workload file
    -cp path       Additional Java classpath entries
    -jvm-args args Additional arguments to the JVM
    -p key=value   Override workload property
    -s             Print status to stderr
    -target n      Target ops/sec (default: unthrottled)
    -threads n     Number of client threads (default: 1)

Workload Files:
    There are various predefined workloads under workloads/ directory.
    See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties
    for the list of workload properties.

positional arguments:
  {load,run,shell}      Command to run.
  {accumulo,accumulo1.6,accumulo1.7,accumulo1.8,aerospike,arangodb,arangodb3,asynchbase,azurecosmos,azuretablestorage,basic,basicts,cassandra-cql,cassandra2-cql,cloudspanner,couchbase,couchbase2,crail,dynamodb,elasticsearch,elasticsearch5,elasticsearch5-rest,foundationdb,geode,googlebigtable,googledatastore,griddb,hbase098,hbase10,hbase12,hbase14,hbase20,hypertable,ignite,ignite-sql,infinispan,infinispan-cs,jdbc,kudu,maprdb,maprjsondb,memcached,mongodb,mongodb-async,nosqldb,orientdb,postgrenosql,rados,redis,rest,riak,rocksdb,s3,solr,solr6,tablestore,tarantool}
                        Database to test.

optional arguments:
  -h, --help            show this help message and exit
  -cp CLASSPATH         Additional classpath entries, e.g. '-cp
                        /tmp/hbase-1.0.1.1/conf'. Will be prepended to the
                        YCSB classpath.
  -jvm-args JVM_ARGS    Additional arguments to pass to 'java', e.g. '-Xmx4g'