模型推理batch inference速度无明显提升、耗时线性增长问题排查

现象描述

当模型在推理阶段使用batch inference时，推理速度并无明显提升，相比单帧多次推理收益不大。如笔者在Xavier上测试某模型结果

batch size	推理时间ms	折算耗时 ms/img
1	11.23	11.23
2	20.39	10.20
4	38.73	9.68
8	74.11	9.26
32	287.30	8.98

类似情况在网上也很多见，如yolov5作者的测试结果【1】

按理来说，多张图放一个batch喂给模型，模型矩阵运算可以并行操作，推理的速度可以有batch size倍的提升，但实际观察到的现象确实提升不大，尤其是在一些算力较弱的设备上。

原因分析:

在网上搜索一番，大概定位原因，这里参考GitHub tensorrt 的 issues 1046解答【2】:

简单来说，问题在于gpu计算性能有瓶颈。如果单张图的计算量已经快占满计算核心（达到性能瓶颈），再增加batch size也无法多张图并行计算，尤其是在网络中间的一些层channel数特别大时，瞬时矩阵乘法运算量非常大，cuda核用满了就需要排队慢慢计算。

Generally, GPU computation is more efficient when the batch size is larger. This is because when you have a lot of ops, you can fully utilize the GPUs and hide some inefficiency or overhead between ops. However, if there are already a lot of ops at BS=1 and even BS=1 is able to fully utilize the GPUs, you may not see any increase in efficiency anymore.

For example, is your input size BSx3x1600x1000? This is a super large image which is expected to fully utilize even the largest GPU we have (like A100), so I don't think increasing BS gives benefit on GPU efficiency.

In terms of N/V/K, in your case the "N" is already 1600x1000 at BS=1, so N=1600x1000 vs N=2x1600x1000 do not make too much difference in turns of GPU efficiency, compared to N=1 vs N=2.

另外一个现象就是gpu性能越高，batch inference效果提升越明显。如笔者在xavier上测试单帧推理时，GPU利用率就接近60%，所以当batch size增加时基本无增益，而yolov5作者在A100（性能天花板更高）测试时，加速效果更明显。其实当batch size非常大时，相当于在让GPU持续工作直到计算完成，减少了等待时间，所以性能越高可以并行计算的量也就越大，加速越明显。