三层包测试
发包方式:192.168.20.205
发包工具:pktgen
发包网卡:i350
收包设备:1922.168.20.185
CPU:Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
收包方式:网卡ixgeb驱动,单队列,promisc计数
收包网卡: 82599EB
传输链路:千兆RJ45网线直连
发包方法:
Pktgen:/> set 0 size 500
Pktgen:/> set 0 count 1000000
Pktgen:/> set 0 proto tcp
Pktgen:/> disable 0 vlan
Pktgen:/> start 0
Pktgen:/> clear 0 stats
收包设置:
驱动设置
- 重载驱动,清空计数,同时可以调整队列数。
- 设置混杂模式
- 禁用PAUSE帧
irq设置
绑定IRQ
echo "1,13" > /proc/irq/eth0/smp_affinity_list
CMD
modprobe -r ixgbe
#modprobe ixgbe MQ=1,0
modprobe ixgbe
ifconfig eth0 promisc
ethtool --pause eth0 autoneg off rx off tx off
irq_bind.sh 1
测试结果:
包长(byte) |
pps |
Mbps |
seconds |
socket数 |
收包physics core数 |
队列数 |
发包数 |
收包数 |
500 |
230000 |
960 |
|
1 |
1 |
24 |
10000000 |
10000000 |
400 |
284000 |
950 |
|
1 |
1 |
24 |
10000000 |
10000000 |
300 |
367000 |
940 |
|
1 |
1 |
24 |
10000000 |
10000000 |
200 |
542000 |
952 |
|
1 |
1 |
24 |
10000000 |
10000000 |
100 |
1039000 |
997 |
9.62 |
1 |
1 |
24 |
10000000 |
10000000 |
64 |
1483000 |
996 |
6.74 |
1 |
1 |
24 |
10000000 |
9907134 |
64 |
1483000 |
996 |
67.4 |
1 |
1 |
24 |
100000000 |
99147516 |
64 |
1483000 |
996 |
67.4 |
1 |
1 |
1 |
100000000 |
99671467 |
64 |
1483000 |
996 |
67.4 |
1 |
2 |
24 |
100000000 |
99252877 |
64 |
1483000 |
996 |
67.4 |
1 |
4 |
24 |
100000000 |
99084322 |
64 |
1483000 |
996 |
67.4 |
2 |
8 |
24 |
100000000 |
99177716 |
64 |
1483000 |
996 |
67.4 |
2 |
24 |
24 |
100000000 |
99110338 |
设置 Ring 为4096, 之前是512
ethtool -G eth0 rx 4096
包长(byte) |
pps |
Mbps |
seconds |
socket数 |
收包physics core数 |
队列数 |
发包数 |
收包数 |
64 |
1483000 |
996 |
67.4 |
2 |
8 |
24 |
100000000 |
99177716 |
64 |
1483000 |
996 |
67.4 |
2 |
24 |
24 |
100000000 |
98947571 |
查看丢包数的另一种方法
[root@T185 tong]# ethtool -S eth0 |grep rx_missed_errors
rx_missed_errors: 1052429
设置 dev_weight, 默认值64
echo 512 > /proc/sys/net/core/dev_weight
包长(byte)|pps|Mbps|seconds|socket数|收包physics core数|队列数|dev_weight|发包数|收包数
:---😐:---😐:---😐:---😐:---😐:---😐:---😐:---😐:---:
64|1483000|996|67.4|2|24|24|4096|100000000|98992914
参考: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-network-common-queue-issues.html#s-network-commonque-nichwbuf
换用dpdk
dpdk-app收包测试结果
./dpdk-app -l 1,2,3 ../../conf/dpdk-app.conf
包长(byte) |
pps |
Mbps |
发包数 |
收包数 |
500 |
230000 |
960 |
10000000 |
10000000 |
400 |
284000 |
950 |
10000000 |
10000000 |
300 |
367000 |
940 |
10000000 |
10000000 |
200 |
541000 |
952 |
10000000 |
10000000 |
100 |
1039000 |
997 |
10000000 |
10000000 |
64 |
1483000 |
996 |
10000000 |
10000000 |
64 |
1483000 |
996 |
100000000 |
100000000 |
64 |
1483000 |
996 |
1000000000 |
1000000000 |
IP/UDP 分片包测试
DNS包 测试
真实包测试
包名 |
pps |
Mbps |
队列/线程数 |
发包数 |
收包数 |
TSC/pkt |
onescap-4.4g.pcap |
150000 |
940 |
1 |
3800000 |
601200 |
87712 |
onescap-4.4g.pcap |
150000 |
940 |
1 |
3800000 |
631847 |
82141 |
onescap-4.4g.pcap |
150000 |
940 |
1 |
3800000 |
615956 |
84308 |
onescap-4.4g.pcap |
150000 |
940 |
1 |
3800000 |
689366 |
75871 |
onescap-4.4g.pcap |
150000 |
940 |
1 |
3800000 |
950270 |
53670 |
onescap-4.4g.pcap |
150000 |
940 |
2 |
3800000 |
803675 |
AVG(105741,102386)=104063 |
onescap-4.4g.pcap |
150000 |
940 |
2 |
3800000 |
727040 |
AVG(111395,117661)=114528 |
onescap-4.4g.pcap |
150000 |
940 |
2 |
3800000 |
782799 |
AVG(108618,102867)=105742 |
onescap-4.4g.pcap |
150000 |
940 |
2 |
3800000 |
730686 |
AVG(113122,113510)=113316 |
onescap-4.4g.pcap |
150000 |
940 |
2 |
3800000 |
717762 |
AVG(113489,116942)=115215 |
onescap-4.4g.pcap |
150000 |
940 |
4 |
3800000 |
753101 |
AVG(153469,125007,167986,167374)=153459 |
onescap-4.4g.pcap |
150000 |
940 |
4 |
3800000 |
897236 |
AVG(108565,170719,122691,121693)=130917 |
onescap-4.4g.pcap |
150000 |
940 |
4 |
3800000 |
765500 |
AVG(142303,156840,122374,193267)=153696 |
结论
单包处理的TSC随着线程数增加而增加,说明线程间有锁。以至性能不能随着并行数的增加线性增长。从测试结果看,并行处理几乎没有为性能增长带来任何好处。
自构建http测试
包名 |
pps |
Mbps |
队列/线程数 |
会话数 |
发包数 |
收包数 |
TSC/pkt |
http_get.pcap |
none |
line |
1 |
1 |
10 |
10 |
159457 |
http_get.pcap |
none |
line |
1 |
1 |
10 |
10 |
12384 |
http_get.pcap |
none |
line |
1 |
1 |
10 |
10 |
160807 |
http_get.pcap |
none |
line |
1 |
1 |
10 |
10 |
12730 |
http_get.pcap |
none |
line |
1 |
1 |
10 |
10 |
12074 |