InfiniBand 网卡测试
简介
因为要做基于RDMA的分布式系统,所以买了2块二手InfiniBand做开发,这一篇博客记录一下infiniband网卡的测试
- 网卡型号:Mellanox ConnectX-2 MHQH29B Dual Port 4x QDR PCIe 2.0 x8
- 机器环境:ubuntu 14.04, ubuntu 12.04
2块网卡可以不通过交换机直连,(使用OpenSM)
安装步骤
- 安装网卡,并连线(没有驱动的时候,网卡上的灯不亮)
- 下载和安装mellanox驱动:http://www.mellanox.com/page/software_overview_ib
- 安装时建议加--force选项
- 安装完成后系统会检测网卡的pcie配置,比如会提示当前一个网卡插在x4插槽
- 重启机器(网卡连通的端口的灯会亮)
- 给每个网卡配置静态ip,例如:
auto ib1
iface ib1 inet static
address 10.0.0.1
netmask 255.255.255.0
测试步骤
网卡信息查看
- ibnodes命令,会发现端口连接的信息
mlx@m04:~$ ibnodes
Ca : 0x0002c903000ae254 ports 2 "up75 HCA-1"
Ca : 0x0002c903000ec606 ports 2 "m04 HCA-1"
- ifconfig会发现ib端口
ib0 Link encap:UNSPEC HWaddr A0-00-02-20-FE-80-00-00-00-00-00-00-00-00-00-00
UP BROADCAST MULTICAST MTU:4092 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
ib1 Link encap:UNSPEC HWaddr A0-00-03-00-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c903:e:c608/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:54575 errors:0 dropped:0 overruns:0 frame:0
TX packets:67623 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:3174514 (3.1 MB) TX bytes:891903946 (891.9 MB)
- ibstatus可以查看网卡状态,如下所示,可以发现port 2协商的速度为4X QDR
mlx@m04:~$ ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:000e:c607
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:0002:c903:000e:c608
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
2台机器无需交换机连通
使用opensm(需root权限)
mlx@m04:~$ sudo opensm
[sudo] password for mlx:
-------------------------------------------------
OpenSM 4.7.0.MLNX20160523.25f7c7a
Command Line Arguments:
Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 4.7.0.MLNX20160523.25f7c7a
Using default GUID 0x2c903000ec608
Entering DISCOVERING state
Entering MASTER state
此时可以互ping:
mlx@m04:~$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.294 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.155 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.151 ms
64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=0.155 ms
^C
--- 10.0.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.151/0.188/0.294/0.063 ms
速度测试
-
一台机器开启opensm(需root权限),使用ib_send_bw
-
把一台机器作为server:
mlx@m04:~$ ib_send_bw -a -c UD -d mlx4_0 -i 2
************************************
* Waiting for client to connect... *
************************************
- 把另外一台机器作为client:由于up75的网卡插在PCIe 2.0 x4端口,所以速度仅达到了x4的上限,没有达到40Gb/s
mlx@up75:~$ ib_send_bw -a -c UD -d mlx4_0 -i 2 10.0.0.1
Max msg size in UD is MTU 4096
Changing to this MTU
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : UD Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x02 QPN 0x0238 PSN 0xf162c2
remote address: LID 0x01 QPN 0x021a PSN 0xbc213c
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
2 1000 5.72 5.20 2.727911
4 1000 11.49 11.34 2.972020
8 1000 22.99 22.61 2.963387
16 1000 45.98 45.31 2.969666
32 1000 91.70 90.55 2.967229
64 1000 183.14 180.77 2.961664
128 1000 366.79 361.35 2.960143
256 1000 727.44 718.16 2.941597
512 1000 1088.50 1044.70 2.139549
1024 1000 1264.96 1263.29 1.293610
2048 1000 1407.22 1406.43 0.720094
4096 1000 1492.93 1492.75 0.382143
延迟测试
-
一台机器开启opensm(需root权限),使用ib_send_lat
-
把一台机器作为server:
mlx@m04:~$ ib_send_lat -a -c UD -d mlx4_0 -i 2
************************************
* Waiting for client to connect... *
************************************
- 把另外一台机器作为client:
mlx@up75:~$ ib_send_lat -a -c UD -d mlx4_0 -i 2 10.0.0.1
Max msg size in UD is MTU 4096
Changing to this MTU
---------------------------------------------------------------------------------------
Send Latency Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : UD Using SRQ : OFF
TX depth : 1
Mtu : 4096[B]
Link type : IB
Max inline data : 188[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x02 QPN 0x0239 PSN 0x29d370
remote address: LID 0x01 QPN 0x021b PSN 0xbc98c4
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 1.25 14.72 1.34
4 1000 1.24 88.94 1.27
8 1000 1.20 77.49 1.22
16 1000 1.21 66.69 1.23
32 1000 1.23 61.58 1.25
64 1000 1.27 12.92 1.30
128 1000 1.42 6.98 1.44
256 1000 1.94 173.62 1.97
512 1000 2.22 41.65 2.25
1024 1000 2.79 37.47 2.81
2048 1000 3.91 18.85 3.94
4096 1000 6.16 38.06 6.20
---------------------------------------------------------------------------------------
参考文献:
Copyleft © 2013-2015 xysmlx
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 开发中对象命名的一点思考
· .NET Core内存结构体系(Windows环境)底层原理浅谈
· C# 深度学习:对抗生成网络(GAN)训练头像生成模型
· .NET 适配 HarmonyOS 进展
· .NET 进程 stackoverflow异常后,还可以接收 TCP 连接请求吗?
· 本地部署 DeepSeek:小白也能轻松搞定!
· 基于DeepSeek R1 满血版大模型的个人知识库,回答都源自对你专属文件的深度学习。
· 在缓慢中沉淀,在挑战中重生!2024个人总结!
· 大人,时代变了! 赶快把自有业务的本地AI“模型”训练起来!
· Tinyfox 简易教程-1:Hello World!
2014-07-27 HDU 4879 ZCC loves march (2014多校2-1008,数据结构,STL,模拟题)