大数据量导出文件优化
33s,24s/4.88m;2表示测试2次,耗时分别为33秒、24秒,生成文件大小4.88M
数据量 导出文件格式 |
6w | 10w | 20w |
csv | 33s,24s/4.88m;2 | 59s,8.14m | 250s/16.3m;3 |
dbf | 37s,38s/40.1m;2 | 64s/66.8m | 239s,240s,240s/133m;3 |
xlsx | 40s/2.86m;2 | 63s/4.75m | 246s,248s,248s/9.52m;3 |
10w数据测试更多细节
3.2(getData)18 | 3.2(getData)18 | 3.2(getData)34 | |||||
3.3(ObjToMap)11 | 3.3(ObjToMap&Mapping)16 | 3.3(ObjToMap&Mapping)26 | |||||
3.4(ComplexPropData)0 | 3.4(ComplexPropData)0 | 3.4(ComplexPropData)0 | |||||
3.5(Mapping)4 | 3.5(DataIntercept)0 | 3.5(DataIntercept)0 | |||||
3.6(DataIntercept)0 | 2.2 35 | 2.2 60 | |||||
2.2 34 | 2.3(addtitletosheet) 0 | 2.3(addtitletosheet) 0 | |||||
2.3(addtitletosheet) 0 | 2.4(addgridtosheet) 4 | 2.4(addgridtosheet) 9 | |||||
2.4(addgridtosheet) 3 | 2.5(serverpro) 40 | 2.5(serverpro) 69 | |||||
2.5(serverpro) 38 | 2.6(write) 1 | 2.6(write) 2 | |||||
2.6(write) 1 | 1.2(server) 42 | 1.2(server) 72 | |||||
1.2(server) 39 | 1.3 0 | 1.3 0 | |||||
1.3 0 |
结论:10w数据做3次for循环和1次for循环耗时差别不大,所以慢的根本原因不在于此
更多细节:
begin:1464679204339 | begin:1464680191678 | |||
totalCount:100000 | totalCount:100000 | |||
getDatas1:1464679207778 | getDatas1:1464680194934 | |||
usetime:3s | usetime:3s | |||
sortDatas :1464679207778 | sortDatas :1464680194934 | |||
usetime:0s | usetime:0s | |||
0-299999 | 0-299999 | |||
getPagingData :1464679207778 | getPagingData :1464680194934 | |||
usetime:0s | usetime:0s | |||
3.1.2 3 | 3.1.2 3 | |||
3.2(getData)31 | 3.2(getData)3 | |||
3.3(ObjToMap&Mapping)14 | 3.3(ObjToMap&Mapping)38 | |||
3.4(ComplexPropData)0 | 3.4(ComplexPropData)0 | |||
3.5(DataIntercept)0 | 3.5(DataIntercept)0 | |||
2.2 46 | 2.2 41 | |||
2.3(addtitletosheet) 0 | 2.3(addtitletosheet) 0 | |||
2.4(addgridtosheet) 7 | 2.4(addgridtosheet) 5 | |||
2.5(serverpro) 53 | 2.5(serverpro) 47 | |||
2.6(write) 2 | 2.6(write) 2 | |||
1.2(server) 56 | 1.2(server) 49 | |||
1.3 0 | 1.3 0 |
3.2和3.3变化很大,
代码不同:第一次测试,3.2步骤包含代理对象的集合强制类型转换代码collection = ((EntityList) collection).getTarget();
而第二次测试,3.3步骤移除了上述代码慢的代码为:targetMap=BeanToMapUtil.convertBean(obj);即将代理对象转换为map,第一次测试同样包含了该代码。
综上:大数据量时处理代理对象是很耗时的。
修改获取数据方式,直接获取原始对象数据:
begin:1464681604310 | begin:1464681631666 | ||||
totalCount:100000 | totalCount:100000 | ||||
getDatas1:1464681607939 | getDatas1:1464681635058 | ||||
usetime:3s | usetime:3s | ||||
sortDatas :1464681607939 | sortDatas :1464681635058 | ||||
usetime:0s | usetime:0s | ||||
0-299999 | 0-299999 | ||||
getPagingData :1464681607939 | getPagingData :1464681635058 | ||||
usetime:0s | usetime:0s | ||||
3.2(getData)3 | 3.2(getData)3 | ||||
3.3(ObjToMap&Mapping)6 | 3.3(ObjToMap&Mapping)7 | ||||
3.4(ComplexPropData)0 | 3.4(ComplexPropData)0 | ||||
3.5(DataIntercept)0 | 3.5(DataIntercept)0 | ||||
2.2 10 | 2.2 11 | ||||
2.3(addtitletosheet) 0 | 2.3(addtitletosheet) 0 | ||||
2.4(addgridtosheet) 5 | 2.4(addgridtosheet) 5 | ||||
2.5(serverpro) 16 | 2.5(serverpro) 17 | ||||
2.6(write) 2 | 2.6(write) 2 | ||||
1.2(server) 18 | 1.2(server) 19 | ||||
1.3 0 | 1.3 0 |
耗时减少近2/3。
结论:大数据量情况下,代理严重影响性能。
岁月留痕