Basics of using bash, and shell tools for covering several of the most common tasks

Introduction

‍

Most shells have their own scripting language with variables, control flow and its own syntax.

‍

Shell scripting optimized for performing shell-related tasks.

‍

• Creating command pipelines
• Saving results into files
• Reading from standard input are primitives in shell scripting

‍

---

‍

Catalog

‍

‍

---

‍

Assign variables in bash

‍

Syntax:

‍

• Assign variables: variableName=value​
• Access variables' value: $variableName​

‍

Note: variableName = value​ will not work if it is interpreted as calling the variableName​ program with arguments =​ and value​

Added: the space character ​ will perform argument splitting in shell.

‍

---

‍

Differences between '​ and "​

‍

Strings in bash can be defined with '​ and "​ delimiters, they are not equivalent.

‍

• ​'​ : Only the literal strings
• ​"​ : Will substitute variable values that strings contain

‍

---

‍

Use functions in bash

‍

mcd () {
    mkdir -p "$1"
    cd "$1"
}

‍

• ​$0​ - Name of the script
• ​$1​ to $9​ - Arguments to the script. $1​ is the first argument and so on.
• ​$@​ - All the arguments
• ​$#​ - Number of arguments
• ​$?​ - Return code of the previous command
• ​$$​ - Process identification number (PID) for the current script
• ​!!​ - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doing sudo !!​
• ​$_​ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typing Esc​ followed by .​ or Alt+.​

‍

---

‍

Write the scripts

‍

We can type it directly in our shell, but we always prefer to write those commands into a file and execute it.

After writing it into a script(e.g. mcd.sh​), we could use source​ to execute and load it.

Next, give the arguments to the script (e.g. mcd test​), and it will work.

‍

bash manner

‍

‍

if

‍

if [ condition ]
then
  commands
elif [ condition ]
then
  commands
else
  commands
fi

‍

---

‍

case

‍

case variable in
  pattern1)
    commands
    ;;
  pattern2)
    commands
    ;;
  *)
    commands
    ;;
esac

‍

---

‍

select

‍

select variable in list
do
  commands
  break
done

‍

---

‍

for loop

‍

for variable in list
do
  commands
done

‍

---

‍

while loop

‍

while condition
do
  commands
done

‍

---

‍

until loop

‍

until condition
do
  commands
done

‍

---

‍

Stream

‍

• output: STDOUT​ >​
• errors : STDERR​ 2>​

‍

---

‍

Return code or exit status

‍

• value of 0​ -> OK;
• anything different from 0​ -> error occurred

‍

---

‍

Operator

‍

• and operator: &&​
• or operator: ||​
• seperate commands: ;​
• ​true​: 0 return code
• ​false​: 1 return code
• ​-ne​ : not equal

‍

---

‍

Get the output of a command as a variable

‍

Substitutions

‍

command substitution: $(CMD)​

process substitution: <(CMD)​

Execute CMD​ and place the output in a temporary file and substitute the <()​ with that file’s name.

‍

​/dev/null​ : You can write sth. to it, but all the written things will be discarded.

‍

---

‍

No need to give arguments one by one

‍

Sometimes, you don't need to give arguments one by one, with the usage of globbing(e.g. *​, ?​, {}​) .

In general, you can just use it by way of using the small regex.

Also, you can use another language(e.g. Python​) to interact with the Shell.

‍

---

‍

The usage of shebang

‍

We can use the shebang​ (e.g. #!/usr/local/bin/python​) to let the Shell know how to run this program.

But, python may not be installed at this path, and you may can't know where they actually live in your machine, we can use env​ to use the PATH​ environment variable to resolve such a kind of question.(e.g. #!/usr/bin/env python​)

‍

As writing bash​ scripts can be tricky and unintuitive, we can use tools like shellcheck​ to help us to find errors in our sh/bash scripts.

‍

---

‍

Differences between shell functions and scripts

‍

• Language

  • Shell functions: The same language as the shell
  • Scripts: Any language(So it's important to include a shebang for scripts)

• Load

  • Shell functions: Loaded once (Slightly fast to load, but whenever you change them you will have to reload their definition)
  • Scripts: Loaded whenever they are executed

• execute

  • Shell functions: In the current shell environment(Can modify environment variables)

  • Scripts: In their own process(Can't modify environment variables)

    Scripts will be passed by value environment variables that have been exported using export​

• As with any programming language, functions are a powerful construct to achieve modularity, code reuse, and clarity of shell code

• Often shell scripts will include their own function definitions.

‍

---

‍

Shell Tools

‍

Finding how to use commands

‍

• ​man​ provides a manual page (called manpage) for a command you specify.
• ​-h​ or --help​ flags
• ​:help​ or ?​(For interactive tools such as the ones based on ncurses)
• ​tldr​ focuses on giving example use cases of a command so you can quickly figure out which options to use.

‍

---

‍

Finding files

‍

• find​

  • ​-name​

  • ​-path​ pattern

  • ​-type​

    • ​d​ : directory
    • ​f​ : file

  • ​-mtime​

  • ​-exec​

• ​fd​
  A simple, fast, and user-friendly alternative to find​.

• ​locate​
  ​locate​ uses a database that is updated using updatedb​

  • In most systems, updatedb​ is updated daily via cron​.
  • A more in-depth comparison between find​ and locate​ can be found here.

‍

---

‍

Finding code(the content of the file)

‍

• ​grep​

  ​grep​ is an incredibly valuable shell tool that we will cover in greater detail during the data wrangling lecture.

  • ​-R​: go through the entire directory

    • Many grep​ alternatives have been developed, including ack, ag, and rg.

    • ​riggrep​ : more colorful

      • ​-u​: not ignore hidden files
      • ​-t​: type
      • ​-C​: show lines around the matched
      • ​--files-without-match​
      • ​--stats​

‍

---

‍

Finding shell commands(already used)

‍

• ​up​

• ​history​

  • ​history | grep find​

• ​fzf​ bindings.

‍

---

‍

Directory Navigation

‍

• tree​
• ​broot​
• ​nnn​

‍

‍

---

‍

Shortcut key

‍

• Ctl-A
  Moves cursor to beginning of line of text (on the command-line).
• Ctl-B
  Backspace (nondestructive).
• Ctl-C
  Break. Terminate a foreground job.
• Ctl-D
  Log out from a shell (similar to exit).
  EOF (end-of-file). This also terminates input from stdin.
  When typing text on the console or in anxterm window, Ctl-D erases the character under the cursor. When there are no characters present, Ctl-D logs out of the session, as expected. In an xterm window, this has the effect of closing the window.
• Ctl-E
  Moves cursor to end of line of text (on the command-line).
• Ctl-F
  Moves cursor forward one character position (on the command-line).
• Ctl-G
  BEL. On some old-time teletype terminals, this would actually ring a bell. In an xterm it might beep.
• Ctl-H
  Rubout (destructive backspace). Erases characters the cursor backs over while backspacing.
• Ctl-I
  Horizontal tab.
• Ctl-J
  Newline (line feed). In a script, may also be expressed in octal notation -- '\012' or in hexadecimal -- '\x0a'.
• Ctl-K
  Vertical tab.
  When typing text on the console or in anxterm window, Ctl-K erases from the character under the cursor to end of line. Within a script, Ctl-K may behave differently, as in Lee Lee Maschmeyer's example, below.
• Ctl-L
  Formfeed (clear the terminal screen). In a terminal, this has the same effect as the clear command. When sent to a printer, a Ctl-L causes an advance to end of the paper sheet.
• Ctl-M
  Carriage return.
• Ctl-N
  Erases a line of text recalled from history buffer [8] (on the command-line).
• Ctl-O
  Issues a newline (on the command-line).
• Ctl-P
  Recalls last command from history buffer (on the command-line).
• Ctl-Q
  Resume (XON).
  This resumes stdin in a terminal.
• Ctl-R
  Backwards search for text in history buffer (on the command-line).
• Ctl-S
  Suspend (XOFF).
  This freezes stdin in a terminal. (Use Ctl-Q to restore input.)
• Ctl-T
  Reverses the position of the character the cursor is on with the previous character (on the command-line).
• Ctl-U
  Erase a line of input, from the cursor backward to beginning of line. In some settings, Ctl-U erases the entire line of input, regardless of cursor position.
• Ctl-V
  When inputting text, Ctl-V permits inserting control characters.
• Ctl-W
  When typing text on the console or in an xterm window, Ctl-W erases from the character under the cursor backwards to the first instance of whitespace. In some settings, Ctl-W erases backwards to first non-alphanumeric character.
• Ctl-X
  In certain word processing programs, Cuts highlighted text and copies to clipboard.
• Ctl-Y
  Pastes back text previously erased (with Ctl-U or Ctl-W).
• Ctl-Z
  Pauses a foreground job.
  Substitute operation in certain word processing applications.
  EOF (end-of-file) character in the MSDOS filesystem.

‍

---

‍

Exercises

‍

1. ​ls -laht --color​

   1. ​ls -a​
   2. ​ls -h​
   3. ​ls -lh​
   4. ​ls -lt​
   5. ​ls --color​

2. bash functions marco​ and polo​

   1. ​marco.sh​

      #!/bin/bash
      marco(){
              path_dir=$(pwd)
              export path_dir
              echo "set $path_dir"
      }

   2. ​polo.sh​

      #!/bin/bash
      polo(){
              cd "$path_dir"
              echo -n 'cd to ' && echo "$path_dir"
      }

3. ​runUFail​

   1. ​fail.sh​

      #!/usr/bin/env bash
       
      n=$(( RANDOM % 100 ))
       
      if [[ n -eq 42 ]]; then
      	echo "Something went wrong"
      	>&2 echo "The error was using magic numbers"
      	exit 1
      fi
       
      echo "Everything went according to plan"

   2. ​runUFail​

      #!/bin/bash
       
      runUFail(){
      	./fail.sh > output.txt 2> error.txt
      	code=$?
      	fail_times=0
      	while [ $code -ne 1 ]
      	do
      		fail_times=$((fail_times+1))
      		./fail.sh >> output.txt 2>> error.txt
      		code=$?
      	done
      	echo "fail times:$fail_times"
      	echo "output.txt:" && cat output.txt
      	echo "error.txt:" && cat error.txt
      }

4. ​xargs​ command which will execute a command using STDIN as arguments.

   ​touch ex{0,1,2,3,4,5,asb,'a b','bb'}.html​

   ​find -path "*html" -type f -print0 | ls | xargs -I {} zip output.zip {}​

5. ​find . -type f -mtime -1 | ls -lt​

‍

---

‍

References

‍

• Advanced Bash-Scripting Guide: https://tldp.org/LDP/abs/html/special-chars.html
• The Missing Semester of Your CS Education: https://missing.csail.mit.edu/2020/shell-tools/

‍