linux 系统中wget实现并行下载
(一)
1、测试
[root@centos79 test2]# ls source.txt [root@centos79 test2]# cat source.txt ## 首先准备了3个下载资源 https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz [root@centos79 test2]# cat source.txt | while read file; do wget $file & done ## 使用&符号放置后台运行 [root@centos79 test2]# --2021-11-16 22:41:42-- https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip --2021-11-16 22:41:42-- https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip --2021-11-16 22:41:42-- https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz Resolving s3.amazonaws.com (s3.amazonaws.com)... Resolving github.com (github.com)... Resolving github.com (github.com)... 52.217.193.224 Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.193.224|:443... 20.205.243.166 Connecting to github.com (github.com)|20.205.243.166|:443... 20.205.243.166 Connecting to github.com (github.com)|20.205.243.166|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 8917076 (8.5M) [application/zip] Saving to: ‘plink_linux_x86_64_20210606.zip’ 31% [===========================> ] 2,793,696 318KB/s eta 33s connected.
2、测试后台进程 (进程都在)
[root@centos79 test2]# ps -aux | grep wget root 23060 0.2 0.2 152064 4764 pts/0 S 22:41 0:00 wget https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip root 23061 0.0 0.2 151932 4100 pts/0 S 22:41 0:00 wget https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip root 23062 0.0 0.2 151932 4096 pts/0 S 22:41 0:00 wget https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz root 23064 0.0 0.0 112808 960 pts/1 S+ 22:41 0:00 grep --color=auto wget
(二)
1、
[root@centos79 test2]# cat source.txt https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz [root@centos79 test2]# cat source.txt | while read file; do wget $file -b; done ## 加-b参数, fork到后台 Continuing in background, pid 23116. Output will be written to ‘wget-log’. Continuing in background, pid 23118. Output will be written to ‘wget-log.1’. Continuing in background, pid 23120. Output will be written to ‘wget-log.2’. [root@centos79 test2]#
2、查看进程 (进程都在)
[root@centos79 test2]# ps -aux | grep wget root 23116 0.2 0.2 152068 4136 ? Ss 22:45 0:00 wget https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip -b root 23118 24.9 0.2 152068 4244 ? Rs 22:45 0:02 wget https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip -b root 23120 0.0 0.1 151936 3456 ? Ss 22:45 0:00 wget https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz -b root 23122 0.0 0.0 112808 960 pts/1 S+ 22:45 0:00 grep --color=auto wget
(三)
1、
[root@centos79 test2]# ls source.txt [root@centos79 test2]# cat source.txt https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip https://github.com/vcftools/vcftools/releases/download/v0.1.16/vcftools-0.1.16.tar.gz [root@centos79 test2]# cat source.txt | xargs -n 1 -P 2 wget -q
2、查看进程
[root@centos79 test2]# ps -aux | grep wget root 23231 0.0 0.0 108352 616 pts/0 S+ 22:51 0:00 xargs -n 1 -P 2 wget -q root 23232 0.0 0.2 152064 4768 pts/0 S+ 22:51 0:00 wget -q https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_20210606.zip root 23233 0.0 0.2 151932 4100 pts/0 S+ 22:51 0:00 wget -q https://github.com/broadinstitute/gatk/releases/download/4.2.3.0/gatk-4.2.3.0.zip root 23243 0.0 0.0 112808 960 pts/1 S+ 22:51 0:00 grep --color=auto wget