linux下多线程压缩命令pigz

官网地址:
 http://www.zlib.net/pigz/

一.命令安装

yum install zlib-devel  gcc gcc-c++   -y
wget http://www.zlib.net/pigz/pigz-2.4.tar.gz
tar xf pigz-2.4.tar.gz
cd  pigz-2.4
[root@VM_82_178_centos pigz-2.4]# make 
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas   -c -o pigz.o pigz.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas   -c -o yarn.o yarn.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas   -c -o try.o try.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/deflate.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/blocksplitter.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/tree.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/lz77.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/cache.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/hash.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/util.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/squeeze.c
gcc -O3 -Wall -Wextra -Wno-unknown-pragmas -c zopfli/src/zopfli/katajainen.c
gcc  -o pigz pigz.o yarn.o try.o deflate.o blocksplitter.o tree.o lz77.o cache.o hash.o util.o squeeze.o katajainen.o -lm -lpthread -lz
ln -f pigz unpigz
[root@VM_82_178_centos pigz-2.4]# echo $?
0
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.

安装成功

二.参数帮助说明

[root@VM_82_178_centos pigz-2.4]# find / -name "pigz"
/root/pigz-2.4/pigz

[root@VM_82_178_centos pigz-2.4]# /root/pigz-2.4/pigz --help
Usage: pigz [options] [files ...]
  will compress files in place, adding the suffix '.gz'. If no files are
  specified, stdin will be compressed to stdout. pigz does what gzip does,
  but spreads the work over multiple processors and cores when compressing.

Options:
  -0 to -9, -11        Compression level (level 11, zopfli, is much slower)
  --fast, --best       Compression levels 1 and 9 respectively
  -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
  -c, --stdout         Write all processed output to stdout (won't delete)
  -d, --decompress     Decompress the compressed input
  -f, --force          Force overwrite, compress .gz, links, and to terminal
  -F  --first          Do iterations first, before block split for -11
  -h, --help           Display a help screen and quit
  -i, --independent    Compress blocks independently for damage recovery
  -I, --iterations n   Number of iterations for -11 optimization
  -J, --maxsplits n    Maximum number of split blocks for -11
  -k, --keep           Do not delete original file after processing
  -K, --zip            Compress to PKWare zip (.zip) single entry format
  -l, --list           List the contents of the compressed input
  -L, --license        Display the pigz license and quit
  -m, --no-time        Do not store or restore mod time
  -M, --time           Store or restore mod time
  -n, --no-name        Do not store or restore file name or mod time
  -N, --name           Store or restore file name and mod time
  -O  --oneblock       Do not split into smaller blocks for -11
  -p, --processes n    Allow up to n compression threads (default is the
                       number of online processors, or 8 if unknown)
  -q, --quiet          Print no messages, even on error
  -r, --recursive      Process the contents of all subdirectories
  -R, --rsyncable      Input-determined block locations for rsync
  -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
  -t, --test           Test the integrity of the compressed input
  -v, --verbose        Provide more verbose output
  -V  --version        Show the version of pigz
  -Y  --synchronous    Force output file write to permanent storage
  -z, --zlib           Compress to zlib (.zz) instead of gzip format
  --                   All arguments after "--" are treated as files
[root@VM_82_178_centos pigz-2.4]# 
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.

三.实例演示

说明此命令主要是结合tar命令一起使用的

3.1物理服务器一测试

在4核,16G内存的物理服务器上打包测试
tar 结合pigz压缩使用时间46s:

[root@localhost ~]# time tar cvf - /usr/local/mysql-5.7.24-linux-glibc2.12-x86_64 |/root/pigz-2.4/pigz -p 4 >mysql.tar.gz
tar: 从成员名中删除开头的“/”
real	0m46.406s
user	2m50.791s
sys	0m7.436s
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.

**
tar 压缩使用时间2分37秒:**

[root@localhost ~]# time tar zcf mysql1.tar.gz  /usr/local/mysql-5.7.24-linux-glibc2.12-x86_64 
tar: 从成员名中删除开头的“/”

real	2m37.492s
user	2m36.530s
sys	0m7.730s
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.

压缩出来的结果:

[root@localhost ~]# du -sh mysql1.tar.gz  mysql.tar.gz
617M	mysql1.tar.gz
616M	mysql.tar.gz
  • 1.
  • 2.
  • 3.

3.2物理服务器二测试

在2颗cpu,单颗核心12,内存48G物理机器上测试

[root@localhost pigz-2.4]# cd /data/iplog/
[root@localhost iplog]# du -sh *
3.4G	2018-12-25
3.2G	2018-12-26
3.0G	2018-12-27
2.8G	2018-12-28
2.0G	2018-12-29
2.0G	2018-12-30
445M	2018-12-31
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.

tar命令结合pigz命令指定24核心cpu,进行压缩工作

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -p 24 > /tmp/2018-12-25.iplog.tar.gz
real	0m3.753s
user	1m31.944s
sys	0m3.950s
  • 1.
  • 2.
  • 3.
  • 4.

使用的时间不到4秒
压缩同样的文件,tar命令结合pigz命令指定6核心cpu,进行压缩工作

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -p 6 > /tmp/12-25.iplog.tar.gz

real	0m13.462s
user	1m22.719s
sys	0m3.552s
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.

使用的时间不到14秒

使用单独tar命令进行压缩,压缩的时间为1分16秒

[root@localhost iplog]# time tar zcf  /tmp/iplog.tar.gz 2018-12-25

real	1m15.811s
user	1m15.124s
sys	0m3.962s
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.

压缩出来的结果:

[root@localhost iplog]# du -sh /tmp/12-25.iplog.tar.gz /tmp/2018-12-25.iplog.tar.gz /tmp/iplog.tar.gz 
436M	/tmp/12-25.iplog.tar.gz
436M	/tmp/2018-12-25.iplog.tar.gz
435M	/tmp/iplog.tar.gz
  • 1.
  • 2.
  • 3.
  • 4.

pigz命令使用多cpu线程亚缩效率还是非常的高的

指定压缩级别的参数:

Options:
  -0 to -9, -11        Compression level (level 11, zopfli, is much slower)
  • 1.
  • 2.

测试:

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -4 -p 24 > /tmp/01.iplog.tar.gz

real	0m2.504s
user	1m1.557s
sys	0m3.775s
[root@localhost iplog]# du -sh /tmp/01.iplog.tar.gz
477M	/tmp/01.iplog.tar.gz
[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -8 -p 24 > /tmp/02.iplog.tar.gz
real	0m7.569s
user	3m3.478s
sys	0m3.924s
[root@localhost iplog]# du -sh /tmp/02.iplog.tar.gz
415M	/tmp/02.iplog.tar.gz

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -9 -p 24 > /tmp/03.iplog.tar.gz
real	0m11.060s
user	4m27.083s
sys	0m4.148s

[root@localhost iplog]# du -sh /tmp/03.iplog.tar.gz
411M	03.iplog.tar.gz
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.

下面的压缩最耗时,而且也非常消耗CPU,同时压缩比能在tar zcf压缩后的基础上提高16%。所以不建议在有负载的生产机器上采用指定压缩级别的这种方式进行压缩。

[root@localhost iplog]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -11 -p 24 > /tmp/04.iplog.tar.gz
real	18m52.560s
user	445m34.734s
sys	7m22.034s
[root@localhost iplog]# du -sh /tmp/04.iplog.tar.gz
367M	/tmp/04.iplog.tar.gz
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.

比较压缩结果:

[root@localhost ~]# du -sh /tmp/*.iplog.tar.gz 
477M	/tmp/01.iplog.tar.gz
415M	/tmp/02.iplog.tar.gz
411M	/tmp/03.iplog.tar.gz
367M	/tmp/04.iplog.tar.gz
436M	/tmp/12-25.iplog.tar.gz
436M	/tmp/2018-12-25.iplog.tar.gz
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.

上述的压缩文件都采用tar xf解压

[root@localhost tmp]# tar xf 04.iplog.tar.gz
[root@localhost tmp]# du -sh *
3.4G	2018-12-25

[root@localhost tmp]# time tar cf - 2018-12-25  |/opt/pigz-2.4/pigz -9 -K -p 24 > /tmp/iplog.tgz
[root@localhost tmp]# /opt/pigz-2.4/pigz -d iplog.tgz
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.

解压出来的文件是tar包,所以在采用tar xf 来解压


[root@localhost tmp]# /opt/pigz-2.4/pigz -d iplog.tgz
[root@localhost tmp]# ll iplog.tar
-rw-r--r-- 1 root root 3619717120 12月 31 09:33 iplog.tar
[root@localhost tmp]# du -sh iplog.tar
3.4G	iplog.tar
[root@localhost tmp]# tar xf iplog.tar
[root@localhost tmp]# du -sh *
3.4G	2018-12-25
3.4G	iplog.tar
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.

到此简单演示完毕

posted @ 2018-12-31 10:16  勤奋的蓝猫  阅读(24)  评论(0编辑  收藏  举报  来源