[ovs][libvirt][virtio][qemu] vhost user client 排障

 

ovs-dpdk + qemu 使用vhostuserclient通信，连路不通，libvirt报错如下：

 83 2018-05-04 13:17:37.265+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.9 (CentOS BuildSystem <http://bugs.centos.org>, 2018-03-07-13:51:24, x86-01.bsys.centos.org), qemu version: 2.9.0, hostna
 84 LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /export/Datapath/tong/qemu_home/bin/qemu-system-x86_64 -name guest=centos7.0,debug-threads=on -S -object secret,id=master
 85 2018-05-04T13:17:37.295614Z qemu-system-x86_64: -chardev socket,id=charnet1,path=/tmp/nlb_vm0.sock,server: QEMU waiting for connection on: disconnected:unix:/tmp/nlb_vm0.sock,server
 86 2018-05-04T13:17:37.808975Z qemu-system-x86_64: -chardev socket,id=charnet2,path=/tmp/nlb_vm1.sock,server: QEMU waiting for connection on: disconnected:unix:/tmp/nlb_vm1.sock,server
 87 2018-05-04T13:17:38.809417Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
 88 2018-05-04T13:17:45.706248Z qemu-system-x86_64: Failed initializing vhost-user memory map, consider using -object memory-backend-file share=on
 89 2018-05-04T13:17:45.706284Z qemu-system-x86_64: vhost_set_mem_table failed: Interrupted system call (4)
 90 2018-05-04T13:17:45.706421Z qemu-system-x86_64: unable to start vhost net: 4: falling back on userspace virtio
 91 2018-05-04T13:17:45.708417Z qemu-system-x86_64: Failed initializing vhost-user memory map, consider using -object memory-backend-file share=on
 92 2018-05-04T13:17:45.708436Z qemu-system-x86_64: vhost_set_mem_table failed: Resource temporarily unavailable (11)
 93 2018-05-04T13:17:45.708558Z qemu-system-x86_64: unable to start vhost net: 11: falling back on userspace virtio
 94 2018-05-04T13:33:01.734457Z qemu-system-x86_64: terminating on signal 15 from pid 2367 (/usr/sbin/libvirtd)

 

在libvirt的xml里增加一行：

     20   <cpu mode='host-passthrough' check='none'>
     21     <topology sockets='1' cores='4' threads='2'/>
     22     <numa>
     23       <cell id='0' cpus='0-7' memory='16777216' unit='KiB' memAccess='shared'/>
     24     </numa>
     25   </cpu>

 

对应的qemu命令增加了：

 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/6-centos7.0,share=yes,size=17179869184 -

 

排障成功。

 

参考：

http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/

In addition, QEMU must allocate the VM’s memory on hugetlbfs. vhost-user ports access a virtio-net device’s 
virtual rings and packet buffers mapping the VM’s physical memory on hugetlbfs. To enable vhost-user ports 
to map the VM’s memory into their process address space, pass the following parameters to QEMU:

-object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on
-numa node,memdev=mem -mem-prealloc

 

https://libvirt.org/formatdomain.html#elementsNUMATuning