硬件RDMA的驱动配置和测试
author：headsen chen
date： 2019-01-18 10:22:20
notice：created by headsen chen himself and not allowed to copy, or you will count law question!
版本环境：centos6.8 ，64位，内核：2.6.32
1，配置网卡：
在新卡装上机器，接收光纤，两根线都有接，而且是反接的方式接，接通后，光纤灯会亮
2，安装软件RDMA的方式安装，编译内核和用户态，重启进入新内核4.7
3，安装驱动：
正常的kernel安装方法（2.6的内核）
# /mnt/mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64/install
这里必须采用这种，因为是新内核4.7
# tar fx mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64.tgz # cd mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64 # ./install --add-kernel-support --skip-repo Logs dir: /tmp/mlnx-en.28728.logs General log file: /tmp/mlnx-en.28728.logs/general.log Verifying KMP rpms compatibility with target kernel... This program will install the mlnx-en package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with mlnx-en, do not reinstall them. Do you want to continue?[y/N]:y rpm --nosignature -e --allmatches --nodeps rdma rdma Starting mlnx-en-4.4-2.0.7.0 installation ... Installing mlnx-en-utils 4.4 RPM Preparing... ################################################## mlnx-en-utils ################################################## Installing kmod-mlnx-en 4.4 RPM Preparing... ################################################## kmod-mlnx-en ################################################## Installing mlnx-en-sources 4.4 RPM Preparing... ################################################## mlnx-en-sources ################################################## Installing mlnx-en-doc 4.4 RPM Preparing... ################################################## mlnx-en-doc ################################################## Installing user level RPMs: Preparing... ################################################## ofed-scripts ################################################## Preparing... ################################################## mstflint ################################################## Device (83:00.0): 83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] Link Width: x8 PCI Link Speed: 8GT/s Device (83:00.1): 83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] Link Width: x8 PCI Link Speed: 8GT/s Installation finished successfully. Preparing... ########################################### [100%] 1:mlnx-fw-updater ########################################### [100%] Updated /usr/share/hwdata/pci.ids Attempting to perform Firmware update... Querying Mellanox devices firmware ... Device #1: ---------- Device Type: ConnectX4LX Part Number: MCX4121A-XCA_Ax Description: ConnectX-4 Lx EN network interface card; 10GbE dual-port SFP28; PCIe3.0 x8; ROHS R6 PSID: MT_2420110004 PCI Device Name: 83:00.0 Base MAC: ec0d9ad2fd68 Versions: Current Available FW 14.20.1010 14.23.1020 PXE 3.5.0210 3.5.0504 UEFI N/A 14.16.0017 Status: Update required --------- Found 1 device(s) requiring firmware update... Device #1: Updating FW ... Done Restart needed for updates to take effect. Log File: /tmp/mlnx-en.28728.logs/fw_update.log Configuring /etc/security/limits.conf. To load the new driver, run: /etc/init.d/mlnx-en.d restart
4，重启服务：
/etc/init.d/mlnx-en.d restart
5，安装MLNX_OFED_LINUX-4.4
这里不用像软件RDMA 那样的启动rxe_cfg了。
yum -y install libmml tcl tk libmnl tar fx MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64.tgz cd MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64 ./mlnxofedinstall --add-kernel-support --skip-repo /etc/init.d/openibd restart # 这个命令最好在管理卡上执行，xshell上执行有可能导致网卡掉IP， /etc/init.d/network restart chkconfig openibd on ibv_devices 出现一下结果代表成功： # ibv_devices device node GUID ------ ---------------- mlx5_1 ec0d9a0300d2fc99 mlx5_0 ec0d9a0300d2fc98 如果这一步不成功（有时候rxe_cfg不启动也可以）： # rxe_cfg start (并绑定eth4网卡) # ibv_devices device node GUID ------ ---------------- rxe0 ee0d9afffed2fd68
# ibv_devinfo rxe0 hca_id: rxe0 transport: InfiniBand (0) fw_ver: 0.0.0 node_guid: ee0d:9aff:fed2:fd68 sys_image_guid: 0000:0000:0000:0000 vendor_id: 0x0000 vendor_part_id: 0 hw_ver: 0x0 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
6，利用rping 命令测试：
生成server端：
[root@bj01-prd-hadoop499.vivo.lan:/root] # rping -s -a 10.20.15.23 -v -C 10
生成client端：
client端的安装和服务端一样，生成命令是：
# rping -c -a 10.20.15.23 -v -C 10
此时就会出现一下界面，证明安装成功：
# rping -s -a 10.20.15.23 -v -C 10 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10
# rping -c -a 10.20.15.23 -v -C 10 ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT...
------------------------------------------------------
利用udaddy来测试，出现一下结果代表成功：
服务端： [root@bj01-prd-hadoop499.vivo.lan:/mnt/MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64] # udaddy udaddy: starting server receiving data transfers sending replies data transfers complete test complete return status 0
客户端： [root@bj01-prd-hadoop500.vivo.lan:/root] # udaddy -s 10.20.15.23 udaddy: starting client udaddy: connecting initiating data transfers receiving data transfers data transfers complete test complete return status 0
