ubuntu1804 搭建dpdk 测试环境
1、安装编译dpdk 所必须的包；
apt install make
apt install gcc
root@noc-NF5270M4:/home/noc# apt install libnuma-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
libnuma-dev
0 upgraded, 1 newly installed, 0 to remove and 251 not upgraded.
Need to get 32.3 kB of archives.
After this operation, 166 kB of additional disk space will be used.
Get:1 https://mirrors.ustc.edu.cn/ubuntu bionic-updates/main amd64 libnuma-dev a md64 2.0.11-2.1ubuntu0.1 [32.3 kB]
Fetched 32.3 kB in 0s (88.5 kB/s)
Selecting previously unselected package libnuma-dev:amd64.
(Reading database ... 133625 files and directories currently installed.)
Preparing to unpack .../libnuma-dev_2.0.11-2.1ubuntu0.1_amd64.deb ...
Unpacking libnuma-dev:amd64 (2.0.11-2.1ubuntu0.1) ...
Setting up libnuma-dev:amd64 (2.0.11-2.1ubuntu0.1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
2、对大页内存进行设置，直接在cmdline 里进行设置；
vim /etc/default/grub
GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=64"
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ sudo grub-mkconfig -o /boot/grub/grub.cfg
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.3.0-28-generic
Found initrd image: /boot/initrd.img-5.3.0-28-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
设置完成后重新启动机器；
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ sudo reboot
查看设置的是否生效；
noc@noc-NF5270M4:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.3.0-28-generic root=/dev/mapper/ubuntu--vg-root ro default_hugepagesz=1G hugepagesz=1G hugepages=64 quiet splash vt.handoff=1
查看大页内存是否正常的进行了挂载；
noc@noc-NF5270M4:~$ mount | grep huge
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=1024M)
3、查看是否加载了uio模块
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ lsmod | grep uio
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ lsmod | grep igb
igb 221184 0
i2c_algo_bit 16384 2 igb,ast
dca 16384 3 igb,ioatdma,ixgbe
加载uio 模块，加载编译好的igb_uio.ko 内核文件；
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ sudo modprobe uio
[sudo] password for noc:
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ sudo insmod kmod/igb_uio.ko
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ lsmod | grep igb
igb_uio 20480 0
uio 20480 1 igb_uio
igb 221184 0
i2c_algo_bit 16384 2 igb,ast
dca 16384 3 igb,ioatdma,ixgbe
4、尝试启动testpmd 管理工具；
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ ./app/testpmd -- i
EAL: Detected 16 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: Cannot obtain physical addresses: No such file or directory. Only vfio will function.
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
PANIC in main():
Cannot init EAL
5: [./app/testpmd(_start+0x2a) [0x55621750f01a]]
4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fa5d6728b97]]
3: [./app/testpmd(main+0xc40) [0x556217504c10]]
2: [./app/testpmd(__rte_panic+0xc5) [0x5562174f5992]]
1: [./app/testpmd(rte_dump_stack+0x2e) [0x55621765f01e]]
Aborted (core dumped)
启动失败，查看错误为未找到合适的物理设备，所以导致失败；
5、绑定物理设备到uio 模块上使用；
5、1 查找需要的设备ID信息；
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ lspci | grep Eth
02:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
84:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
84:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
5、2查找设备的厂商ID与产品ID；
noc@noc-NF5270M4:~/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc$ lspci -vmn
Device: 84:00.0
Class: 0200
Vendor: 8086
Device: 10fb
SVendor: 8086
SDevice: 000c
PhySlot: 0-8
Rev: 01
NUMANode: 1
Device: 84:00.1
Class: 0200
Vendor: 8086
Device: 10fb
SVendor: 8086
SDevice: 000c
PhySlot: 0-8
Rev: 01
NUMANode: 1
5、3 将设备绑定到uio 模块里进行加载；
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# echo "8086 10fb" > /sys/bus/pci/drivers/igb_uio/new_id
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# echo 0000:84:00.0 > /sys/bus/pci/drivers/ixgbe/unbind
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# echo 0000:84:00.1 > /sys/bus/pci/drivers/ixgbe/unbind
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# echo 0000:84:00.1 > /sys/bus/pci/drivers/igb_uio/bind
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# echo 0000:84:00.0 > /sys/bus/pci/drivers/igb_uio/bind
查看是否绑定进来；
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# ll /sys/bus/pci/drivers/igb_uio/
total 0
drwxr-xr-x 2 root root 0 7月 14 13:26 ./
drwxr-xr-x 32 root root 0 7月 14 13:23 ../
lrwxrwxrwx 1 root root 0 7月 14 13:38 0000:84:00.0 -> ../../../../devices/pci0000:80/0000:80:02.2/0000:84:00.0/
lrwxrwxrwx 1 root root 0 7月 14 13:38 0000:84:00.1 -> ../../../../devices/pci0000:80/0000:80:02.2/0000:84:00.1/
--w------- 1 root root 4096 7月 14 13:36 bind
lrwxrwxrwx 1 root root 0 7月 14 13:34 module -> ../../../../module/igb_uio/
--w------- 1 root root 4096 7月 14 13:34 new_id
--w------- 1 root root 4096 7月 14 13:34 remove_id
--w------- 1 root root 4096 7月 14 13:26 uevent
--w------- 1 root root 4096 7月 14 13:34 unbind
6、再次用交互式启动，查看启动情况；
root@noc-NF5270M4:/home/noc/dpdk-stable-18.11.8/x86_64-native-linuxapp-gcc# ./app/testpmd -- -i
EAL: Detected 16 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:84:00.0 on NUMA socket 1
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:84:00.1 on NUMA socket 1
EAL: probe driver: 8086:10fb net_ixgbe
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=267456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 1)
Port 0: 68:91:D0:61:57:2E
Configuring Port 1 (socket 1)
Port 1: 68:91:D0:61:57:2F
Checking link statuses...
Done
testpmd>testpmd> show port summary all
Number of available ports: 2
Port MAC Address Name Driver Status Link
0 68:91:D0:61:57:2E 0000:84:00.0 net_ixgbe down 0Mbps
1 68:91:D0:61:57:2F 0000:84:00.1 net_ixgbe down 0Mbps
上面能够正常看到绑定进来的网卡信息，down 的状态可能是因为网络问题再定位即可；
