[dpdk] dpdk编译成动态库使用 -- PCI port自动发现与pmd动态加载
1. 修改配置文件 .conf, 设置如下变量的值.
[root@D129 x86_64-native-linuxapp-gcc]# cat dpdk/x86_64-native-linuxapp-gcc/.config |grep SHARE
CONFIG_RTE_BUILD_SHARED_LIB=y
2. 这个时候, 再编译的 dpdk app就会自动链接dpdk的动态库. 如下:
[root@D129 app]# ldd testpmd linux-vdso.so.1 => (0x00007ffed89fd000) librte_kni.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_kni.so.2 (0x00007fe9c983f000) librte_pipeline.so.3 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pipeline.so.3 (0x00007fe9c962b000) librte_table.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_table.so.2 (0x00007fe9c93ed000) librte_port.so.3 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_port.so.3 (0x00007fe9c9191000) librte_pdump.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pdump.so.1 (0x00007fe9c8d77000) librte_distributor.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_distributor.so.1 (0x00007fe9c8b74000) librte_reorder.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_reorder.so.1 (0x00007fe9c8965000) librte_ip_frag.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_ip_frag.so.1 (0x00007fe9c8724000) librte_meter.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_meter.so.1 (0x00007fe9c8521000) librte_sched.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_sched.so.1 (0x00007fe9c830d000) librte_lpm.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_lpm.so.2 (0x00007fe9c80ff000) librte_acl.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_acl.so.2 (0x00007fe9c7ee5000) librte_jobstats.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_jobstats.so.1 (0x00007fe9c7ce2000) librte_power.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_power.so.1 (0x00007fe9c7acd000) librte_timer.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_timer.so.1 (0x00007fe9c78c3000) librte_hash.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_hash.so.2 (0x00007fe9c76af000) librte_vhost.so.3 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_vhost.so.3 (0x00007fe9c7477000) librte_kvargs.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_kvargs.so.1 (0x00007fe9c7274000) librte_mbuf.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_mbuf.so.2 (0x00007fe9c7071000) libethdev.so.4 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/libethdev.so.4 (0x00007fe9c6dd6000) librte_cryptodev.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_cryptodev.so.1 (0x00007fe9c6bc1000) librte_mempool.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_mempool.so.2 (0x00007fe9c69b9000) librte_ring.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_ring.so.1 (0x00007fe9c67b5000) librte_eal.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_eal.so.2 (0x00007fe9c6546000) librte_cmdline.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_cmdline.so.2 (0x00007fe9c633b000) librte_cfgfile.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_cfgfile.so.2 (0x00007fe9c6137000) librte_pmd_bond.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pmd_bond.so.1 (0x00007fe9c5efa000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe9c5cd9000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fe9c5ad4000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe9c58b8000) libc.so.6 => /lib64/libc.so.6 (0x00007fe9c54f5000) /lib64/ld-linux-x86-64.so.2 (0x00007fe9c9a51000) libm.so.6 => /lib64/libm.so.6 (0x00007fe9c51f2000) librt.so.1 => /lib64/librt.so.1 (0x00007fe9c4fea000) [root@D129 app]#
3. 但是与static的时候对比, 你会发现有如下的问题:
用static链接的时候, rte_init的时候,会扫描所有的PCI设备,找到所有可用的port, 如下:
[root@D129 app]# ./testpmd EAL: Detected 16 lcore(s) EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:00:03.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd EAL: PCI device 0000:00:04.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2304, socket=0 RING: Cannot reserve memory EAL: Error - exiting with code: 1 Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory [root@D129 app]#
在使用shared so库的时候, 会发现,dpdk app扫描不到任何 PCI 设备了. 如下:
[root@D129 app]# ./testpmd EAL: Detected 16 lcore(s) EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: No probed ethernet devices USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2304, socket=0 RING: Cannot reserve memory EAL: Error - exiting with code: 1 Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
这个时候,我们ldd app,会发现, 它并没有ld到pmd的so,
[root@D129 app]# ldd testpmd |grep pmd librte_pmd_bond.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pmd_bond.so.1 (0x00007fa25689a000) [root@D129 app]#
猜测, dpdk在shared的情况下, pmd的库,是需要动态加载的. 并找到如下参数:
[root@D129 app]# ./testpmd -h |grep -A1 LIB.so -d LIB.so|DIR Add a driver or driver directory (can be used multiple times)
使用 -d 参数制定 virtio的so, (我的虚拟机是virtio的网络设备), 果然生效了
[root@D129 app]# ./testpmd -d ../lib/librte_pmd_virtio.so EAL: Detected 16 lcore(s) EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:00:03.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd EAL: PCI device 0000:00:04.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2304, socket=0 RING: Cannot reserve memory EAL: Error - exiting with code: 1 Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory [root@D129 app]#
4. 问题分析
参考 dpdk 的源码 ../dpdk/lib/librte_eal/common/eal_common_options.c::eal_plugins_init()
在读取 -d 参数的同时, dpdk在 rte_init函数中还会读取 RTE_EAL_PMD_PATH 目录下的文件, 所有读到的文件会使用 dlopen() 打开. 详细逻辑参见代码.
变量RTE_EAL_PMD_PATH的值由CONFIG_RTE_EAL_PMD_PATH在编译的时候决定.
5. 参考redhat的rpm spec, 最终的解决方案是这样的.
5.1 生成一个编译与运行时都存在的目录: /lib64/tong-dpdk-pmds/,
5.2 将所以pmd软链接到这个目录下
[root@D129 app]# ll /lib64/tong-dpdk-pmds/ total 0 lrwxrwxrwx 1 root root 28 Jul 25 13:37 librte_pmd_af_packet.so.1 -> ../librte_pmd_af_packet.so.1 lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_bnxt.so.1 -> ../librte_pmd_bnxt.so.1 lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_bond.so.1 -> ../librte_pmd_bond.so.1 lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_cxgbe.so.1 -> ../librte_pmd_cxgbe.so.1 lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_e1000.so.1 -> ../librte_pmd_e1000.so.1 lrwxrwxrwx 1 root root 22 Jul 25 13:37 librte_pmd_ena.so.1 -> ../librte_pmd_ena.so.1 lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_enic.so.1 -> ../librte_pmd_enic.so.1 lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_fm10k.so.1 -> ../librte_pmd_fm10k.so.1 lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_i40e.so.1 -> ../librte_pmd_i40e.so.1 lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_ixgbe.so.1 -> ../librte_pmd_ixgbe.so.1 lrwxrwxrwx 1 root root 30 Jul 25 13:37 librte_pmd_null_crypto.so.1 -> ../librte_pmd_null_crypto.so.1 lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_null.so.1 -> ../librte_pmd_null.so.1 lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_ring.so.2 -> ../librte_pmd_ring.so.2 lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_vhost.so.1 -> ../librte_pmd_vhost.so.1 lrwxrwxrwx 1 root root 25 Jul 25 13:37 librte_pmd_virtio.so.1 -> ../librte_pmd_virtio.so.1 lrwxrwxrwx 1 root root 30 Jul 25 13:37 librte_pmd_vmxnet3_uio.so.1 -> ../librte_pmd_vmxnet3_uio.so.1
5.2 在编译前, 设置变量CONFIG_RTE_EAL_PMD_PATH
[root@D129 app]# cat ../.config |grep PATH CONFIG_RTE_EAL_PMD_PATH="/lib64/tong-dpdk-pmds/"
以上, dpdk app便可以使用 shared lib 正常运行了.
6. 另一个方案: 使用 -d, 需要神马加神马, 比如:
[root@D129 app]# ./testpmd -d ../lib/librte_pmd_virtio.so -d ../lib/librte_pmd_ixgbe.so
完.
