spdk(二)----编译spdk和fio测试

SPDK使用了DPDK中一些通用的功能和机制，编译spdk需要依赖dpdk，所以这里首先编译dpdk

1. 下载安装dpdk

git https://github.com/DPDK/dpdk

yum install perl

由于我的内核版本是3.10.0

yum install http://ftp.scientificlinux.org/linux/scientific/7.4/x86_64/updates/security/kernel-devel-3.10.0-957.1.3.el7.x86_64.rpm

(其他内核版本的kernel-devel包可以到这里下载：https://kojipkgs.fedoraproject.org/packages/kernel/4.8.0/1.fc26/x86_64/)

make config T=x86_64-native-linuxapp-gcc
yum install numactl-devel

make -j4 && make install

dpdk默认安装到/usr/local，包括.a库文件和头文件

2. 编译 spdk

yum install https://www.nasm.us/pub/nasm/stable/linux/nasm-2.14.02-0.fc27.x86_64.rpm

(可能需要安装的一些包：yum install librbd-devel.x86_64, yum install numactl-devel， yum install CUnit-devel.x86_64)
对于编译spdk依赖包的安装还可以参考：./scripts/pkgdep.sh （安装依赖包）

./configure --enable-debug --with-rbd --with-dpdk=/usr/local/share/dpdk/x86_64-native-linux-gcc/ --with-virtio --without-isal --disable-tests

make  -j4

以上就将spdk编译好了。
在spdk/app/vhost目录下可以看到一个名为vhost的可执行文件，它就是SPDK在虚拟化场景下为虚拟机模拟程序qemu提供的存储转发服务，借此为虚拟机用户带来高性能的虚拟磁盘。

fio测试：https://www.jianshu.com/p/eeaf81ffb7b5

测试结果：

ioengine=spdk

[root@devel ~]# LD_PRELOAD=/root/spdk-19.10.1/examples/nvme/fio_plugin/fio_plugin /root/fio/fio /root/spdk-19.10.1/examples/nvme/fio_plugin/example_config.fio
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.3
Starting 1 thread
Starting SPDK v19.10.1 / DPDK 19.08.0 initialization...
[ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=
▽
0x200000000000 --match-allocations --file-prefix=spdk_pid430343 ]
nvme_qpair.c: 119:nvme_admin_qpair_print_command: *NOTICE*: GET LOG PAGE (02) sqid:0 cid:95 nsid:ffffffff cdw10:007f00c0 cdw11:00000000
nvme_qpair.c: 307:spdk_nvme_qpair_print_completion: *NOTICE*: INVALID LOG PAGE (01/09) sqid:0 cid:95 cdw0:0 sqhd:0010 p:1 m:0 dnr:0
nvme_ctrlr.c: 548:nvme_ctrlr_set_intel_support_log_pages: *WARNING*: Intel log pages not supported on Intel drive!
Jobs: 1 (f=1): [r(1)][100.0%][r=1145MiB/s,w=0KiB/s][r=293k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=430351: Sun Mar  1 22:22:36 2020
   read: IOPS=293k, BW=1144MiB/s (1200MB/s)(223GiB/200001msec)
    slat (nsec): min=137, max=35285, avg=175.85, stdev=209.20
    clat (usec): min=217, max=1499, avg=436.59, stdev=85.17
     lat (usec): min=217, max=1499, avg=436.77, stdev=85.17
    clat percentiles (usec):
     |  1.00th=[  285],  5.00th=[  314], 10.00th=[  330], 20.00th=[  359],
     | 30.00th=[  383], 40.00th=[  408], 50.00th=[  433], 60.00th=[  457],
     | 70.00th=[  482], 80.00th=[  510], 90.00th=[  553], 95.00th=[  578],
     | 99.00th=[  644], 99.50th=[  676], 99.90th=[  799], 99.95th=[  857],
     | 99.99th=[  988]
   bw (  MiB/s): min= 1111, max= 1161, per=99.99%, avg=1143.87, stdev= 6.64, samples=399
   iops        : min=284652, max=297304, avg=292829.47, stdev=1699.63, samples=399
  lat (usec)   : 250=0.01%, 500=76.54%, 750=23.28%, 1000=0.16%
  lat (msec)   : 2=0.01%
  cpu          : usr=100.00%, sys=0.01%, ctx=363, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=58574243,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=1144MiB/s (1200MB/s), 1144MiB/s-1144MiB/s (1200MB/s-1200MB/s), io=223GiB (240GB), run=200001-200001msec

ioengine:libaio

[root@devel ~]# /root/fio/fio /root/spdk-19.10.1/examples/nvme/fio_plugin/libaio_config.fio
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.3
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=1123MiB/s,w=0KiB/s][r=288k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=6147: Sun Mar  1 22:37:20 2020
   read: IOPS=287k, BW=1123MiB/s (1177MB/s)(219GiB/200001msec)
    slat (nsec): min=1150, max=4269.3k, avg=1987.69, stdev=1945.76
    clat (usec): min=243, max=4787, avg=443.02, stdev=88.18
     lat (usec): min=245, max=4788, avg=445.04, stdev=88.08
    clat percentiles (usec):
     |  1.00th=[  297],  5.00th=[  318], 10.00th=[  334], 20.00th=[  363],
     | 30.00th=[  388], 40.00th=[  412], 50.00th=[  437], 60.00th=[  461],
     | 70.00th=[  490], 80.00th=[  519], 90.00th=[  562], 95.00th=[  586],
     | 99.00th=[  652], 99.50th=[  685], 99.90th=[  807], 99.95th=[  889],
     | 99.99th=[ 1450]
   bw (  MiB/s): min= 1039, max= 1142, per=100.00%, avg=1122.76, stdev= 9.58, samples=400
   iops        : min=266192, max=292588, avg=287426.43, stdev=2451.76, samples=400
  lat (usec)   : 250=0.01%, 500=74.43%, 750=25.39%, 1000=0.15%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%
  cpu          : usr=17.93%, sys=61.07%, ctx=1379637, majf=0, minf=129
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwt: total=57484278,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=1123MiB/s (1177MB/s), 1123MiB/s-1123MiB/s (1177MB/s-1177MB/s), io=219GiB (235GB), run=200001-200001msec

Disk stats (read/write):
  nvme0n1: ios=57446275/0, merge=0/0, ticks=22587108/0, in_queue=22594529, util=100.00%

截图：