xgqfrms™, xgqfrms® : xgqfrms's offical website of cnblogs! xgqfrms™, xgqfrms® : xgqfrms's offical website of GitHub!

Linux bash shell script batch download files All In One

合集 - Linux Shell(70)
1.Linux shell command ln All In One2.Linux shell command cut All In One3.Linux shell standard input bugs All In One4.Linux shell command base64 All In One5.How to use the shell command to get the version of Linux Distributions All In One6.Linux shell command make & Makefile All In One7.Linux shell command strings All In One8.sudo & su & Rust All In One9.Linux shell command make All In One10.Linux shell script shebang env All In One11.Linux shell script switch...case All In One12.Linux shell command ls sort by date All In One13.Linux shell script read file line by line All In One14.Linux shell regular expression All In One15.How to fix IP filter regular expressions written using grep command in Linux shell script All In One16.Linux shell script programming All In One17.Linux shell script redirection All In One18.How to use Linux shell command filter the IP address All In One19.how to create one interactive mode command line configuration tool with shell language on Linux All In One20.Linux shell script get date and time All In One21.Linux shell command screen All In One22.Linux shell `#!` interpreter All In One23.Linux shell set command All In One24.Linux shell script auto generate batch files All In One25.Linux bash script HereDoc All In One26.Linux shell system environment variables All In One27.Linux shell command curl All In One28.Linux shell command chroot All In One29.Linux shell command chmod All In One30.Linux shell number variables add All In One31.Linux bash shell "${1}" All In One32.Linux xattr shell command All In One33.Multiple Ways to Change Terminal Shell in Linux All In One34.Linux shell stdin & stdout & stderr All In One35.Linux shell man command All In One36.Linux shell command line editor All In One37.Linux shell command uname All In One38.Linux bash shell script block comment All In One
39.Linux bash shell script batch download files All In One
40.Linux shell script bug and error All In One41.Linux bash write long string with multi-lines comments All In One42.Linux set command All In One43.Linux echo command All In One44.Linux echo 换行 All In One45.Linux Bash shell 脚本定时器 All In One46.Linux System Variables All In One47.linux bash shell auto read write template48.Linux bash script read args All In One49.Linux Bash Script conditions syntax All In One50.Linux Bash Script loop syntax All In One51.Linux shell script All In One52.如何通过 Linux terminal 查看一个文件的 meta 信息 All In One53.Linux shell uname -a All In One54.Linux shell command create file methods All In One55. how to write string to file in linux bash All In One56.linux bash shell & lsof & grep & ps57.Linux Bash Shell All In One58.Linux shell command copy file All In One59.Linux shell command & zip & tar All In One60.Linux shell command show project directory tree All In One61.Linux shell commands man chmod All In One62.Linux & bash & shell & PID & PPID63.Linux bash shell All In One64.Linux shell command symbolic link & soft link All In One65.Linux bash shell All In One66.Linux shell (.sh 文件) 基础命令 常用命令 汇总67.How to use a shell script to check whether a command had been installed in the Linux server All In One68.How to print a string with a variable by using the echo command in the shell script All In One69.Linux install vim errors All In One70.Raspberry Pi 设置开机登录后自启动脚本 All In One
收起

Linux bash shell script batch download files All In One

Linux bash shell script 批量下载文件 All In One

solution

pdf crawler / pdf 爬虫

#!/bin/bash

# 下载目录
downdir="/Users/xgqfrms-mbp/Documents/swift-ui/Memorize/000-xyz/pdfs/"

# $1 是传递给 shell 的第一个参数
# read line 按行读取文件
cat $1 | while read line
do
  # shell 变量需要使用双引号包裹， 或 echo $line
  echo "$line"
  cd $downdir
  str=$line
  # 按行分割，每行一个, 正则表达式：字符串转数组
  array=(${str//;/ })
  echo "$array"
  url=${array[0]}
  # tr 删除换行字符 ✅
  filename=$(echo ${array[1]} | tr -d '\r')
  # filename=$(echo "l" + ${index} + ".pdf" | tr -d '\r')
  # filename=$(echo "l${index}.pdf" | tr -d '\r')
  # cURL 执行下载, -o 输出文件
  curl $url -o $filename
done

# exit 0


# mkdir pdfs
$ bash ./auto-download-pdfs.sh cs193p.txt
# OR, 可执行脚本
$ chmod +x ./auto-download-pdfs.sh
$ ./auto-download-pdfs.sh cs193p.txt

cs193p.txt

https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l1.pdf;l1.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l2.pdf;l2.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l3.pdf;l3.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l4.pdf;l4.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l5.pdf;l5.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l6.pdf;l6.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l7.pdf;l7.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l8.pdf;l8.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l9.pdf;l9.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l10.pdf;l10.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l11.pdf;l11.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l12.pdf;l12.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l13.pdf;l12.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l14.pdf;l14.pdf

demos

CS193p PDFs, 2020 Spring L1 ~ L14

https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l1.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l1.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l2.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l3.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l4.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l5.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l6.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l7.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l8.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l9.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l10.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l11.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l12.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l13.pdf
https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l14.pdf

Linux tr command

tr 转义或删除字符

$ man tr > man-tr.md

$ cat man-tr.md
TR(1)                                             User Commands                                            TR(1)

NAME
       tr - translate or delete characters

SYNOPSIS
       tr [OPTION]... SET1 [SET2]

DESCRIPTION
       Translate, squeeze, and/or delete characters from standard input, writing to standard output.

       -c, -C, --complement
              use the complement of SET1

       -d, --delete
              delete characters in SET1, do not translate

       -s, --squeeze-repeats
              replace  each  sequence  of  a repeated character that is listed in the last specified SET, with a
              single occurrence of that character

       -t, --truncate-set1
              first truncate SET1 to length of SET2

       --help display this help and exit

       --version
              output version information and exit

       SETs are specified as strings of characters.  Most represent themselves.  Interpreted sequences are:

       \NNN   character with octal value NNN (1 to 3 octal digits)

       \\     backslash

       \a     audible BEL

       \b     backspace

       \f     form feed

       \n     new line

       \r     return

       \t     horizontal tab

       \v     vertical tab

       CHAR1-CHAR2
              all characters from CHAR1 to CHAR2 in ascending order

       [CHAR*]
              in SET2, copies of CHAR until length of SET1

       [CHAR*REPEAT]
              REPEAT copies of CHAR, REPEAT octal if starting with 0

       [:alnum:]
              all letters and digits

       [:alpha:]
              all letters

       [:blank:]
              all horizontal whitespace

       [:cntrl:]
              all control characters

       [:digit:]
              all digits

       [:graph:]
              all printable characters, not including space

       [:lower:]
              all lower case letters

       [:print:]
              all printable characters, including space

       [:punct:]
              all punctuation characters

       [:space:]
              all horizontal or vertical whitespace

       [:upper:]
              all upper case letters

       [:xdigit:]
              all hexadecimal digits

       [=CHAR=]
              all characters which are equivalent to CHAR

       Translation occurs if -d is not given and both SET1 and SET2 appear.  -t may be used only when  translat‐
       ing.  SET2 is extended to length of SET1 by repeating its last character as necessary.  Excess characters
       of SET2 are ignored.  Only [:lower:] and [:upper:] are guaranteed to expand in ascending order;  used  in
       SET2  while  translating,  they  may  only be used in pairs to specify case conversion.  -s uses the last
       specified SET, and occurs after translation or deletion.

AUTHOR
       Written by Jim Meyering.

REPORTING BUGS
       GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
       Report any translation bugs to <https://translationproject.org/team/>

COPYRIGHT
       Copyright  ©  2020  Free  Software  Foundation,  Inc.   License  GPLv3+:  GNU  GPL  version  3  or  later
       <https://gnu.org/licenses/gpl.html>.
       This  is  free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent
       permitted by law.

SEE ALSO
       Full documentation <https://www.gnu.org/software/coreutils/tr>
       or available locally via: info '(coreutils) tr invocation'

GNU coreutils 8.32                               September 2020                                            TR(1)
pi@raspberrypi:~/Desktop/man-docs $

emmet

使用 vscode emmet 语法动态生成 $index.pdf

p{https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l$.pdf}*14

https://code.visualstudio.com/docs/editor/emmet

https://emmet.io/

https://github.com/emmetio/emmet

TODO

Node.js version


const fs = require("fs");
var path = require("path");
const { exit } = require("process");
const log = console.log;
const request = require("request");
// const request = require("request-promise-native");

var folder = path.resolve(__dirname, '../pdf');

// log('folder', folder);

if (!fs.existsSync(folder)) {
  fs.mkdirSync(folder);
}


async function downloadPDF(url, filename) {
  log('🚧 pdf downloading ...');
  const pdfBuffer = await request.get({
    uri: url,
    encoding: null,
    // encoding: 'utf-8',
  });
  fs.writeFileSync(filename, pdfBuffer);
  log('✅ pdf finished!');
  // exit 0;
}

const url = 'https://cs193p.sites.stanford.edu/sites/g/files/sbiybj16636/files/media/file/l1.pdf';
const filename = folder + '/cs193p-2021-l1.pdf';


// log('filename =', filename);

downloadPDF(url, filename);

https://www.cnblogs.com/xgqfrms/p/16086580.html

npm package

$ npm i -g auto-download-files

https://www.npmjs.com/package/auto-download-files

Python version

//

refs

TypeScript & Node.js crawler All In One

https://www.cnblogs.com/xgqfrms/p/16086580.html

©xgqfrms 2012-2025

www.cnblogs.com/xgqfrms 发布文章使用：只允许注册用户才可以访问！

原创文章，版权所有©️xgqfrms, 禁止转载 🈲️，侵权必究⚠️！

posted @   xgqfrms  阅读(92)  评论(6)  编辑  收藏  举报
相关博文：
· Linux tr command All In One
· Linux shell script function All In One
· shell笔记
· Linux shell | tr命令
· linux shell 个人学习笔记
阅读排行：
· 无需6万激活码！GitHub神秘组织3小时极速复刻Manus，手把手教你使用OpenManus搭建本
· C#/.NET/.NET Core优秀项目和框架2025年2月简报
· Manus爆火，是硬核还是营销？
· 终于写完轮子一部分：tcp代理 了，记录一下
· 【杭电多校比赛记录】2025“钉耙编程”中国大学生算法设计春季联赛（1）
历史上的今天：
2021-03-29 页面超过 10 分钟没有任何操作
2021-03-29 node.js ECONNRESET error
2020-03-29 ROI
2020-03-29 React & Calendar
2020-03-29 taro & Block
2020-03-29 taro swiper & scroll tabs
2016-03-29 读写 LED 作业 台灯的 频闪研究 2 评测&对比！
点击右上角即可分享
微信分享提示