数据压缩实验(转别人的,待验证)

需要自己验证下最新的版本效率

本测试:

python 和c#都实现了 gzip,zlib,lz4,lzo,snappy算法的压缩解压缩

Lz4lzopythonc#版不能互相解压,gzipzlibsnappy的可以。

 

 

python

压缩率:

 

先看下压缩后的文件大小:

 

 

压缩前的文件大小

 

总结:gzip,zlib 压缩率差不多分别为49.56%,49.57% lzo次之为68.27%,接着是snappy为72.45%,lz4压缩率最大为79.47%。

是否解压成功:

 

总结:解压后文件都回复了原来大小,无损压缩:

 

 

压缩和解压所用时间:

 

各个算法压缩解压缩所用的时间:

 

总结:可以看出来 lz4所用时间最少,snappy次之,接着是lzo,zlib和gzip差不多但是zlib要少一点。

压缩所用系统资源:

 

每0.01秒查看进程利用率 来记录每个算法压缩所需系统资源,第三列为cpu利用率,

下面为记录:

 

root 14303 0.0 5.6 228852 106932 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 14.0 5.6 229464 107544 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 25.0 5.6 231256 108604 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 36.0 5.7 233304 109688 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 47.0 5.8 233304 110776 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 59.0 5.8 237400 111876 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 69.0 6.0 237400 114872 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 80.0 6.0 237400 114872 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 91.0 6.0 237400 115288 pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 102 6.1 245592 116428 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 56.5 6.2 245592 118972 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 62.0 6.2 245592 118972 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 67.5 6.3 245592 121020 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 73.0 6.3 245592 121020 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 78.0 6.4 245592 123068 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 84.0 6.4 245592 123220 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 89.5 6.5 261976 124292 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 95.0 6.6 261976 127168 pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 100 6.6 261976 127168 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 70.6 6.7 261976 129216 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 74.0 6.7 261976 129216 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 77.6 6.8 261976 131264 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 81.3 6.8 261976 131264 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 85.0 6.9 261976 133312 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 88.6 6.9 261976 133312 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 92.3 7.0 261976 135360 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 96.0 7.0 261976 135360 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 99.6 7.2 261976 137408 pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 77.5 7.2 261976 137408 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 80.0 7.3 261976 139456 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 82.7 7.3 261976 139456 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 85.5 7.3 261976 140012 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 88.2 7.3 294744 141160 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 91.0 7.5 294744 143556 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 93.7 7.5 294744 143556 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 96.5 7.6 294744 145604 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 99.2 7.7 294744 147652 pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 81.6 7.7 294744 147652 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 83.8 7.8 294744 149700 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 86.0 7.8 294744 149700 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 88.2 7.9 294744 151748 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 90.2 7.9 294744 151748 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 92.6 8.0 294744 153796 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 94.6 8.0 294744 153796 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 96.8 8.1 294744 155844 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 99.0 8.1 294744 155844 pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 84.3 8.2 294744 157892 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 86.1 8.2 294744 157892 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 88.0 5.6 228948 107144 pts/1 D+ 11:46 0:05 python2.7 test_gzip

root 14303 88.6 0.7 136460 13356 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 90.5 2.1 165460 41232 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 92.5 2.9 181988 56928 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 94.3 4.5 215436 86280 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 96.0 7.0 309784 133808 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 97.8 6.6 256616 127460 pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 85.7 10.5 351628 200592 pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 87.2 11.2 351628 214928 pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 89.1 11.4 358828 219124 pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 90.7 7.0 263816 134660 pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 91.4 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.4 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.5 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.5 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.7 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 80.3 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 80.5 7.0 263816 134664 pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14649 0.0 3.2 224568 62344 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 12.0 5.7 327036 109456 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 23.0 5.7 327036 109456 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 34.0 5.8 327036 111504 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 44.0 5.8 327036 111504 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 56.0 5.9 327036 113552 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 67.0 6.0 327036 115600 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 77.0 6.0 327036 115600 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 89.0 6.1 327036 117648 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 99.0 6.1 327036 117648 pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 55.0 6.2 327036 119696 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 60.5 6.3 327036 121744 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 66.0 6.3 327036 121744 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 71.5 6.4 327036 123792 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 77.5 6.4 327036 123792 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 83.0 6.5 327036 125840 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 88.5 6.5 327036 125840 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 94.0 6.7 327036 127888 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 99.5 6.7 327036 127888 pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 70.0 6.8 327036 129936 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 73.6 6.9 327036 131984 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 77.3 6.9 327036 131984 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 81.0 7.0 327036 134032 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 84.6 7.0 327036 134032 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 88.3 7.1 327036 136080 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 92.0 7.1 327036 136080 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 95.6 7.2 327036 138128 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 99.3 7.2 327036 138128 pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 77.0 7.3 327036 140176 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 79.7 7.4 327036 142224 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 82.5 7.4 327036 142224 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 85.2 7.5 327036 144272 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 88.0 7.5 327036 144272 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 90.7 7.6 327036 146320 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 93.5 7.6 327036 146320 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 96.2 7.7 327036 148368 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 99.0 7.8 327036 150416 pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 81.4 7.8 327036 150416 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 83.6 7.9 327036 152464 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 85.8 7.9 327036 152464 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 88.0 8.0 327036 154512 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 90.0 8.0 327036 154512 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 92.2 8.2 327036 156560 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 94.4 8.2 327036 156560 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 96.6 9.9 377588 189092 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 98.0 2.9 275272 56732 pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 83.3 3.7 205828 72268 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 85.3 5.0 238596 96856 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 85.5 5.0 238596 96856 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 89.0 6.1 238596 117336 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 90.6 7.2 304132 137820 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 92.6 8.2 275272 157604 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 93.8 8.2 275276 157608 pts/1 D+ 11:46 0:05 python2.7 test_zlib

root 14649 94.0 4.7 224592 90032 pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14963 6.0 6.2 325084 120076 pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 17.0 7.7 325084 148748 pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 28.0 9.5 325084 181516 pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 39.0 2.0 201488 39692 pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 51.0 9.8 303704 187968 pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 56.0 9.8 303704 187972 pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 59.0 9.8 303704 187972 pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 59.0 9.8 303704 187972 pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 30.5 9.8 303704 187972 pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 32.0 9.8 303704 187972 pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 15100 10.0 5.9 333336 112888 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 21.0 6.3 333336 121080 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 32.0 6.8 333336 131320 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 43.0 7.3 333336 139512 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 54.0 7.7 333336 147704 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 32.5 8.1 333336 155896 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 38.0 8.6 333336 164088 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 43.5 9.0 333336 172280 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 49.0 9.2 294384 176496 pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 50.0 9.2 294384 176500 pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15100 55.0 5.1 294384 98548 pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 60.5 6.9 294384 133364 pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 66.0 8.7 294384 166132 pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 71.5 9.2 294384 176528 pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 50.0 9.2 294388 176532 pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15100 50.3 9.2 294388 176532 pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15100 50.3 9.2 294388 176532 pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15325 7.0 6.1 351200 116460 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 19.0 7.0 351200 134892 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 30.0 8.1 351200 155372 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 41.0 9.1 351200 173804 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 26.0 9.5 305904 181284 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 36.0 9.5 305908 181308 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 38.5 9.5 305908 181312 pts/1 D+ 11:46 0:00 python2.7 test_snap

root 15325 39.5 9.5 305908 181312 pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 40.5 9.5 305908 181312 pts/1 D+ 11:46 0:00 python2.7 test_snap

 

总结:可以看出 snappy和lz4所用资源差不多,最少,lzo次之,zlib和gzip最多。

 

C#

 

压缩率:

 

各算法压缩后的文件大小:

 

 

压缩前的文件大小

 

总结:可以看出来gzip,zlib 压缩率都为49.57,lzo压缩率68.27,snappy为73.93,lz4压缩率为74.31。

 

是否解压成功:

 

总结:解压后文件都恢复了原来大小,无损压缩:

 

 

压缩和解压所用时间:

 

各个算法压缩解压缩所用的时间:

 

 

 

 

 

 

 

 

 

总结: snappy所用时间最少为0.5秒,lz4次之为0.59,接着是lzo为0.96秒,gzip和zlib远远超过前三个分别为5.71秒和7.15秒。

 

压缩所用系统资源:

 

 

 

 

 

gzip:第一个进程为gzip的。

 

zlib:第一个进程为zlib的。

 

 

 

lz4:第一个进程为lz4的。

 

lzo:第一个进程为lzo的。

 

 

 

snappy:第二个进程为snappy的。

 

总结:可以看出来 gzip和zlib所用资源差不多远远大于lz4,lzo和snappy。

 

各压缩算法跨语言的解压缩:

 

lzo

python压缩的在c#不能解压:

 

一直卡在这界面,解压不了。

 

 

c#压缩的在python中也不能解压。

 

lz4

python压缩的在c#中不能解压

 

c#压缩的在python中也不能解压:

 

 

gzip:

python 压缩的在c#中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

 

 

c#压缩的在python解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

 

 

 

zlib:

python压缩的在c#中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

c#压缩的在python中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

 

 

snappy

python压缩的在c#中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

 

c#压缩的在python中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

 

 

 

 

总结:

 

gzip和zlib压缩率最低,相差不多,同时所消耗资源和时间也是最多的,但他俩都能在python和c#中实现并能互相解压缩。lzo,lz4,snappy三个压缩率比gzip,zlib高,但是所消耗的资源远小于gzip和zlib。lzo和lz4不能跨语言解压缩而snappy可以。

snappy为所找能跨语言压缩解压缩,同时消耗资源较小的压缩算法。

 

 

posted @ 2017-04-14 11:17  骑着驴的王子  阅读(410)  评论(0编辑  收藏  举报