网络性能优化GSO/GIO研究
性能检测工具安装
# curl -O http://downloads.es.net/pub/iperf/iperf-3.0.6.tar.gz
# tar axf iperf-3.0.6.tar.gz
# cd iperf-3.0.6
# ./bootstrap.sh
# ./configure
# make -j $(nproc)
# make install
基本命令
### 网卡参数查询
# ethtool -k <dev>
### 网卡参数配置
# ethtool -K <dev> <tso/gso/gro/ufo> <on/off>
### 启动监听服务
# iperf3 -s
### 发送测试数据
# iperf3 -c <srv> -l <size>
### 网卡抓包
# tcpdump -i <dev> <udp/tcp> -v
测试
测试环境网络拓扑
br-ex用于ssh连接，br-mgmt上网卡用于测试
不启用GSO/GIO
网卡配置
Host A
| 选项 | 状态 |
|---|---|
| tcp-segmentation-offload | off |
| generic-segmentation-offload | off |
| generic-receive-offload | off |
| udp-fragmentation-offload | off |
Host B
| 选项 | 状态 |
|---|---|
| tcp-segmentation-offload | off |
| generic-segmentation-offload | off |
| generic-receive-offload | off |
| udp-fragmentation-offload | off |
测试步骤
### Host B
# tcpdump -i em1 tcp -v
# iperf3 -s
### Host A
# tcpdump -i eno1 tcp -v
# iperf3 -c 92.0.0.240 -l 2000
### 测试部分结果如下Host A & Host B相同
14:11:12.099562 IP (tos 0x0, ttl 64, id 35338, offset 0, flags [DF], proto TCP (6), length 1500)
host-92-0-0-240.as43234.net.47050 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], cksum 0xbfaf (incorrect -> 0x686c), seq 33370646:33372094, ack 1, win 229, options [nop,nop,TS val 13486141 ecr 13531163], length 1448
14:11:12.099563 IP (tos 0x0, ttl 64, id 35339, offset 0, flags [DF], proto TCP (6), length 1500)
host-92-0-0-240.as43234.net.47050 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], cksum 0xbfaf (incorrect -> 0x2b13), seq 33372094:33373542, ack 1, win 229, options [nop,nop,TS val 13486141 ecr 13531163], length 1448
14:11:12.099804 IP (tos 0x0, ttl 64, id 63470, offset 0, flags [DF], proto TCP (6), length 52)
host-92-0-0-241.as43234.net.targus-getdata1 > host-92-0-0-240.as43234.net.47050: Flags [.], cksum 0xadf6 (correct), ack 33359062, win 1720, options [nop,nop,TS val 13531164 ecr 13486140], length 0
### 通过测试结果可以看出，2000字节的数据包被拆分成两个包
启用TSO/GSO/GIO
网卡配置
Host A
| 选项 | 状态 |
|---|---|
| tcp-segmentation-offload | on |
| generic-segmentation-offload | on |
| generic-receive-offload | on |
| udp-fragmentation-offload | off |
Host B
| 选项 | 状态 |
|---|---|
| tcp-segmentation-offload | on |
| generic-segmentation-offload | on |
| generic-receive-offload | on |
| udp-fragmentation-offload | off |
测试步骤
### Host B
# tcpdump -i em1 tcp -v
# iperf3 -c 92.0.0.240 -l 5000
### Host A
# tcpdump -i eno1 tcp -v
# iperf3 -s
### 测试部分结果如下
### Host B（发送端seq 41190502:41213670）
host-92-0-0-241.as43234.net.43976 > host-92-0-0-240.as43234.net.targus-getdata1: Flags [.], cksum 0x1488 (incorrect -> 0x9555), seq 41190502:41213670, ack 1, win 229, options [nop,nop,TS val 15504851 ecr 15459784], length 23168
15:12:09.859303 IP (tos 0x0, ttl 64, id 62667, offset 0, flags [DF], proto TCP (6), length 52)
host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [.], cksum 0x3bc8 (correct), ack 40724246, win 16852, options [nop,nop,TS val 15459784 ecr 15504810], length 0
15:12:09.859553 IP (tos 0x0, ttl 64, id 62668, offset 0, flags [DF], proto TCP (6), length 52)
host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [.], cksum 0x3078 (correct), ack 40727142, win 16852, options [nop,nop,TS val 15459784 ecr 15504810], length 0
### Host A
14:44:05.781627 IP (tos 0x0, ttl 64, id 28209, offset 0, flags [DF], proto TCP (6), length 1500)
host-92-0-0-241.as43234.net.43976 > host-92-0-0-240.as43234.net.targus-getdata1: Flags [.], cksum 0xc904 (correct), seq 41190502:41191950, ack 1, win 229, options [nop,nop,TS val 15504851 ecr 15459784], length 1448
14:44:05.781631 IP (tos 0x0, ttl 64, id 29535, offset 0, flags [DF], proto TCP (6), length 40)
host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [R], cksum 0x80af (correct), seq 1314350815, win 0, length 0
14:44:05.781872 IP (tos 0x0, ttl 64, id 28210, offset 0, flags [DF], proto TCP (6), length 2948)
host-92-0-0-241.as43234.net.43976 > host-92-0-0-240.as43234.net.targus-getdata1: Flags [.], cksum 0xc557 (incorrect -> 0x9be3), seq 41191950:41194846, ack 1, win 229, options [nop,nop,TS val 15504851 ecr 15459784], length 2896
14:44:05.781876 IP (tos 0x0, ttl 64, id 29536, offset 0, flags [DF], proto TCP (6), length 40)
host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [R], cksum 0x80af (correct), seq 1314350815, win 0, length 0
### 可以看出接收端和发送端的length都超过了mtu 1500，说明GSO和GRO都生效了
启用GSO/GIO
网卡配置
Host A
| 选项 | 状态 |
|---|---|
| tcp-segmentation-offload | off |
| generic-segmentation-offload | on |
| generic-receive-offload | on |
| udp-fragmentation-offload | off |
Host B
| 选项 | 状态 |
|---|---|
| tcp-segmentation-offload | on |
| generic-segmentation-offload | on |
| generic-receive-offload | on |
| udp-fragmentation-offload | off |
测试步骤
### Host B
# tcpdump -i em1 tcp
# iperf3 -s
### Host A
# tcpdump -i eno1 tcp
# iperf3 -c 92.0.0.240 -l 5000
### 测试部分结果如下
15:32:41.857417 IP host-92-0-0-240.as43234.net.51572 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], seq 59386070:59387518, ack 1, win 229, options [nop,nop,TS val 16691752 ecr 16736846], length 1448
15:32:41.857540 IP host-92-0-0-240.as43234.net.51572 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], seq 59387518:59388966, ack 1, win 229, options [nop,nop,TS val 16691752 ecr 16736846], length 1448
15:32:41.857542 IP host-92-0-0-241.as43234.net.targus-getdata1 > host-92-0-0-240.as43234.net.51572: Flags [.], ack 59388966, win 1786, options [nop,nop,TS val 16736849 ecr 16691752], length 0
### 但是iperf显示的cwnd（拥塞窗口）比全部关闭是要大，但是小于全部开启
### Host B
# tcpdump -i em1 udp
# iperf3 -s
### Host A
# tcpdump -i eno1 udp
# iperf3 -c 92.0.0.240 -u -l 5000
