linux 系统中wget实现并行下载

（一）

1、测试

[root@centos79 test2]# ls
source.txt
[root@centos79 test2]# cat source.txt        ## 首先准备了3个下载资源
https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip
https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip
https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz
[root@centos79 test2]# cat source.txt | while read file; do wget $file & done  ## 使用&符号放置后台运行
[root@centos79 test2]# --2021-11-16 22:41:42--  https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip
--2021-11-16 22:41:42--  https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip
--2021-11-16 22:41:42--  https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... Resolving github.com (github.com)... Resolving github.com (github.com)... 52.217.193.224
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.193.224|:443... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8917076 (8.5M) [application/zip]
Saving to: ‘plink_linux_x86_64_20210606.zip’

31% [===========================>                                                                ] 2,793,696    318KB/s  eta 33s    connected.

 

2、测试后台进程  （进程都在）

[root@centos79 test2]# ps -aux | grep wget
root      23060  0.2  0.2 152064  4764 pts/0    S    22:41   0:00 wget https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip
root      23061  0.0  0.2 151932  4100 pts/0    S    22:41   0:00 wget https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip
root      23062  0.0  0.2 151932  4096 pts/0    S    22:41   0:00 wget https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz
root      23064  0.0  0.0 112808   960 pts/1    S+   22:41   0:00 grep --color=auto wget

 

（二）

1、

[root@centos79 test2]# cat source.txt
https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip
https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip
https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz
[root@centos79 test2]# cat source.txt | while read file; do wget $file -b; done   ## 加-b参数， fork到后台
Continuing in background, pid 23116.
Output will be written to ‘wget-log’.
Continuing in background, pid 23118.
Output will be written to ‘wget-log.1’.
Continuing in background, pid 23120.
Output will be written to ‘wget-log.2’.
[root@centos79 test2]# 

 

2、查看进程 （进程都在）

[root@centos79 test2]# ps -aux | grep wget
root      23116  0.2  0.2 152068  4136 ?        Ss   22:45   0:00 wget https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip -b
root      23118 24.9  0.2 152068  4244 ?        Rs   22:45   0:02 wget https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip -b
root      23120  0.0  0.1 151936  3456 ?        Ss   22:45   0:00 wget https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz -b
root      23122  0.0  0.0 112808   960 pts/1    S+   22:45   0:00 grep --color=auto wget

 

（三）

1、

[root@centos79 test2]# ls
source.txt
[root@centos79 test2]# cat source.txt
https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip
https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip
https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz
[root@centos79 test2]# cat source.txt | xargs -n 1 -P 2 wget -q

 

2、查看进程

[root@centos79 test2]# ps -aux | grep wget
root      23231  0.0  0.0 108352   616 pts/0    S+   22:51   0:00 xargs -n 1 -P 2 wget -q
root      23232  0.0  0.2 152064  4768 pts/0    S+   22:51   0:00 wget -q https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip
root      23233  0.0  0.2 151932  4100 pts/0    S+   22:51   0:00 wget -q https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip
root      23243  0.0  0.0 112808   960 pts/1    S+   22:51   0:00 grep --color=auto wget