Shell脚本

1 Shell脚本

A script is a list of commands that can be interpreted and run by a program called scripting language
Scripting languages are usually not compiled.They are interpreted at runtime.
Scripts are generally slower to run than compiled languages, but they are also much easier and faster to develop.

用途：自动批处理（文件备份、归档、系统管理任务、应用集成、插件开发……）

shebang：命令解释器

• 通常在第一行指定解释执行方式：#!interpreter [optional-arg],如#!/bin/sh、#!/bin/bash、#!/usr/bin/env python
• interpreter: path to an executable program
• ortional-arg: single argument string

执行shell脚本

# create
touch hell_world.sh

# turn the file into a bash script by echoing the bash shebang
echo '#! /bin/bash' >> hello_world.sh

# add some content
echo 'echo hello world' >> hello_world.sh

# check the permission settings
# -rw-rw-r-- : permission of -owner\group\allusers
ls -l hello_world.sh

# make it executable
chmod +x hello_world.sh
# -rwxrwxr-x

# run your bash script
./hello_world.sh
sh hello_world.sh
source hello_world.sh

1.1 bash脚本的特点

Metacharacters

• # 注释
• ; 命令分隔符 echo "Hello"; whoami
• * 任意连续字符pattern ls /bin/ba*
• ? 任意一个字符pattern ls /bin/?ash

Quoting - whether to interpret special characters as metacharacters or 'escape' them

• \ 转义
• "" 可识别特殊字符
• '' 特殊字符当作普通字符处理

I/O redirection

• 标准输入：keyboard
• 标准输出：terminal
• > 重定向输出（文件不存在时新建一个；存在时覆盖）
• >> 重定向输出（存在时不覆盖，追加）
• 2>、2>> 重定向错误输出流
• < 重定向输入

Command substitution - replace command with its output

• $(command)
• `command`

Command line arguments

• ./myscript.sh arg1 arg2

Batch vs. concurrent modes of execution

• command1 ; command2 batch mode: commands run sequentially
• command1 & command2 concurrent mode:commands run in parallel
• & directs command1 to operate in the background and passes control to command2 in the foreground.

# 单行注释

:'多行注释'

<<COMMENT
....多行注释 
COMMENT

# 约定：
# []：表示里面的元素（参数、值或信息）是可选的
# <>：表示里面的元素(参数、值或信息）是必需的

1.2 获取信息

# username
whoami

# user ID and group ID
id

# (unix name) operating system name and version
uname
uname -s -r
uname -v

# (process status) display running processes and their id
# -u user
ps
ps -u <username>

# (table of processes) processes and resource usage: memory cpu io
# -n limited to n items
top
top -n <number>

# 将指定文件系统挂载到Linux上，纳入管理，这样用户才能看到并访问
mount

# (disk free) info about mounted file systems
# -h human readable
df
df -h [filesys]

# (manual) reference manual for shell commands
man [command]

# print string or variable value
echo
echo [string]
echo [var]
# >> output redirection 将显示内容输出到重定向的文件中，而不是在屏幕上
echo "hello" >> /home/my.txt

# today's date
# -r reference 显示文件上次修改时间
date [format]
date "%j day of %Y"
date -r <filename>

1.3 目录导航

# list files and directories
# -l longer more detailed format
# -r recursively list all directories and files in the directory tree
ls [directory]
ls [filename]

# find files in directory tree
# -name filename
# -iname case-insensitive filename
find <directory> <-name string>

# get present working directory
pwd

# (change directory)
cd [directory ]

1.4 文件管理

# copy file
# -r recursively copy all subdirectories and files
cp <file> <directory>

# change file name or path
mv <file> <directory>
mv <directory_source> [<directory_source>...] <directory_destination>

# remove file
# -r recursively remove the directory along with all its child file objects
# -f force
rm <filename>
rm -r <directoryname>

# remove empty directories
rmdir <directoryname>

# create empty file, update file timestamp
# 如果文件已存在，不会更改文件内容，但会更新文件的修改时间
touch <filename>

# make directory
mkdir

# (change mode) change/modify file permissions rwx
# -x executable
chmod

1.5 查看文件信息

# (catenate) print entire file contents
cat <filename>

# print file contents page-by-page
# click space bar to see the next page
# enter 'q' to quit
more <filename>

# print first N lines of file
# default N = 10
head <filename>
head -n <number> <filename>

# print last N lines of file
tail <filename>
tail -n <number> <filename>

# (word count) get count of lines, words, characters in file
# also counts newline characters EOF
# -l line
# -w word
# -c character
wc <filename>

# -------------文件内容的定制视图-------------
# sort lines in a file
# default alpha-numerically
# -r reverse order
sort <filename>

# (unique) filter out repeated lines
# only remove duplicated lines if they are consecutive
uniq <filename>

# (global regular expression print) return lines in file matching pattern
# -i case-insensitive
grep <pattern> <filename>
grep -i ch people.txt

# extracts a section from each line
# -c charater
cut -c 2-9 people.txt
# -d field delimiter定界符
# -f2 second field
cut -d " " -f2 people.txt

# merge lines from different files
# use "tab" as the default delimiter
# -d specify a delimiter
paste <filename> <filename> [<filename>...]
paste -d "·" firstname.txt lastname.txt  

1.6 归档压缩解压缩

An "archive file" is a collection of data files and directories that are stored as a single file.
Archiving makes the collection more portable and serves as a backup in case of loss or corruption.

File compression involves reducing the size of a file by taking advantage of redundancy in its information content.
The main advantages of compression include preserving storage space, speeding up file transfers, and reducing bandwidth loads.

# (tape archiver) archive and de-archive files and directories
# -c create a new archive
# -f interpret its input from the file rather than standard input
# -z zip compress by g-zip
tar -cf notes.tar notes
tar -czf notes.tar.gz notes
# -t call tar, check the contents of archived file
tar -tf notes.tar
# -x extract objects from the archive, unpack de-archive
tar -xf <tarfilename> [directory]
tar -xzf notes.tar.gz notes

# compress files and directories to an archive
zip <zipfilename> <directory>

# extract files from a compressed or ziped archive
unzip <zipfilename>

# zip compress -> bundle
# tar bundle -> compress  (by -z option g-zip)

1.7 网络操作

# print host name
# -s drop the domain suffix
# -i ip address of the hostname
hostname

# send ICMP packets to URL and print response
# -c a set number of ping results.
ping <domain>
ping -c 5 www.google.com

# (interface configuration) display or configure system network interfaces
# eth0 information about the ethernet adapter
ifconfig
ifconfig eth0

# (client URL) display contents of file at a URL
# default protocol http
# -o write the results to the local file
curl <domain>
curl <domain> -o <filename>

# (web get) download file from URL
# support recursive file downloads
wget <url>

1.8 管道

# pipe command - |
# chaining filter commands
# output of command1 is input of command2
<command1> | <command2> [ | <command3>...]
# 降序展示当前文件夹下的目录
ls | sort -r

1.9 变量

# 变量有自己的作用域

# list all shell variables to the current shell
set

# define shell variables
# No spaces arount '='
var_name=value

# delete variables
unset var_name

# use the '$' to access its value
echo $var_name

# 环境变量可以扩大作用域
# persist in any child processes spawned by the shell in which they originate

# extend a variable to an environment variable
export var_name

# list all environment variables
env
env | grep "var"

1.10 任务调度

# cron is a service that runs jobs
# crond interprets 'crontab files' and submits jobs to cron
# crontab(cron table) is a file containing jobs and schedule data
# -e open editor
crontab -e

# m h dom mon dow command
# minute hour day_of_month month day_of_week scrips
# 15:30 every Sunday
30 15 * * 0 date >> sundays.txt
30 15 * * 0 /scripts/load_data.sh

2 前置知识

2.1 Linux

Unix_based操作系统：Solaris、MacOS；Unix_like操作系统：Linux

Linux的特点

• free、open—source
• 支持多人同时访问
• 支持多任务同时执行
• 支持在各种设备上运行

Linux的五个层次

1. user
2. application（shells、apps、tools）
3. operating system
4. kernel（内存管理、进程管理、设备驱动、系统调用和安全）
5. hardware（CPU、RAM、Storage、Screen……）

Linux的发型版：Debian、Ubuntu、Red Hat、Fedoras

2.2 Shell与Terminal

A shell is a powerful user interface for Unix-like oprating Systems. It can interpret commands and run other programs.
A shell, which enables access to files, utilities, and applications, is also an interactive language.
A shell is also a scripting language. And it canbe used to automate tasks.

计算机的早期并没有图形界面，我们只能通过shell命令控制计算机，现在有了用图形化界面控制计算机。

shell的各种版本：Bash(default shell)、Zsh、Csh
查看当前环境中的shell版本

The terminal is an application you can use to interact with the Linux shell
Enter commands and receive output from them

通过Terminal、Shell与Linux系统交互

2.3 Linux简单命令

# cd更改当前路径
# home目录
cd ~
# parent目录
cd ..
# root目录
cd /

# ls查询当前文件夹下的文件

# 软件安装
sudo apt install <deb-package-name>
sudo yum install <RPM-package-name>
pip install <python-package-name>

2.4 编辑器

• Command-line 命令行：GNU nano、vi、vim
• GUI-based 图形界面：gedit

References

1. Shell是什么？1分钟理解Shell的概念！ (biancheng.net)
2. Hands-on Introduction to Linux Commands and Shell Scripting | Coursera