Shell Scripting

Help Message

#!/bin/sh
###
### my-script — does one thing well
###
### Usage:
###   my-script <input> <output>
###
### Options:
###   <input>   Input file to read.
###   <output>  Output file to write. Use '-' for stdout.
###   -h        Show this message.

help() {
    sed -n 's/^### \?//p' "$0"
}

if [ $# -eq 0 ] || [ "$1" = "-h" ]; then
    help
    exit 1
fi

echo Hello World

Logging and Debugging

Structured logging with jq:

__log(){
  jq \
    --monochrome-output \
    --compact-output \
    --raw-output \
    --arg timestamp "$(date "+%Y%m%dT%H%M%S")" \
    --arg log_level "${1}" \
    --arg message   "${2}" \
     '.timestamp=$timestamp|.log_level=$log_level|.message=$message' \
     <<<'{}' >&2
}

__log "INFO" "Hello, World"

Print with a colon:

if [[ -n ${COLODEBUG:-} && ${-} != *x* ]]; then
    :() {
      [[ ${1:--} != ::* ]] && return 0
      printf '%s\n' "${*}" >&2
    }
fi

: :: note this line
: ::NOTICE::

# This is an example comment
: This is a colon comment
: :: This is a more verbose colon comment

# Example usage:
# COLODEBUG=1 ./myscript.sh

Write logs to syslog/journald:

logger -t checkfs -s -p user.err "Filesystem $filesystem is at $usage"
# or, with long options:
logger --tag checkfs --stderr --priority user.err "Filesystem $filesystem is at $usage"

Besides traps(handlers) for signals, bash have 4 special traps:

  • EXIT to run on exit from the shell.

  • RETURN to run each time a function or a sources script finishes.

  • ERR to run each time command failure would cause the shell to exit if set -e is used.

  • DEBUG to execute before every command.

The last one allows to create a simple debugger inside a bash script:

function _trap_DEBUG() {
    echo "# $BASH_COMMAND";
    while read -r -e -p "debug> " _command; do
        if [ -n "$_command" ]; then
            eval "$_command";
        else
            break;
        fi;
    done
}

trap '_trap_DEBUG' DEBUG

You can change the set -x output format:

# Enable verbose logging of the bash script
PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}:   '
set -x

Alternatives:

Locking

Run only one instance of the script:

# inside the script
[ "$(pidof -x $(basename $0))" != $$ ] && exit

# or, controlling outside the script:
# -w 0: wait for the lock release for 0s (exit immediately)
flock \
    -w 0 /tmp/test.lock \
    -c 'echo "sleeping" && sleep 60' ||
echo "cannot be executed an instance already runs"

Functions

In bash/zsh keyword function intruduced for historical reasons. Use more portable f() ...

Local variables are dinamically scoped (seen by functions down the call stack):

f1() { local a; a=1 f3; }
f2() { local a; a=2 f3; }
f3() { echo $a; }

f1  # prints 1
f2  # prints 2

Shebangs

Python

Pick python version before running:

From dotbot:

#!/usr/bin/env sh

# This is a valid shell script and also a valid Python script. When this file
# is executed as a shell script, it finds a python binary and executes this
# file as a Python script, passing along all of the command line arguments.
# When this file is executed as a Python script, it loads and runs Dotbot. This
# is useful because we don't know the name of the python binary.

''':' # begin python string; this line is interpreted by the shell as `:`
command -v python  >/dev/null 2>&1 && exec python  "$0" "$@"
command -v python3 >/dev/null 2>&1 && exec python3 "$0" "$@"
command -v python2 >/dev/null 2>&1 && exec python2 "$0" "$@"
>&2 echo "error: cannot find python"
exit 1
'''
# python code
import sys, os

...

Version from SO:

#!/bin/bash
'''':
for interpreter in python3 python2 python
do
    which $interpreter >/dev/null 2>&1 && exec $interpreter "$0" "$@"
done
echo "$0: No python could be found" >&2
exit 1
# '''


import sys
print(sys.version)

nix-shell

My shebang for running things in nix-shell and failing back to bash if no nix-shell is available:

#!/usr/bin/env sh

MYSCRIPT="$(mktemp)"
trap 'rm -f -- "${MYSCRIPT}"' EXIT

to_print=
while IFS= read -r line; do
    [ "${line}" = '#!/usr/bin/env nix-shell' ] && to_print=yes
    [ -n "${to_print}" ] && printf '%s\n' "${line}"
done <"$0" >"${MYSCRIPT}"
# or, use sed instead of the above:
# sed -n '/^#!\/usr\/bin\/env nix-shell$/,$p' "$0" >"${MYSCRIPT}"

command -v nix-shell >/dev/null 2>&1 && exec nix-shell "${MYSCRIPT}" "$@"
command -v bash      >/dev/null 2>&1 && exec bash      "${MYSCRIPT}" "$@"
>&2 echo "error: cannot find nix-shell or bash"
exit 1

#!/usr/bin/env nix-shell
#!nix-shell -i bash --pure
#!nix-shell -p bash cacert curl jq python3Packages.xmljson
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/2a601aafdc5605a5133a2ca506a34a3a73377247.tar.gz

curl https://github.com/NixOS/nixpkgs/releases.atom | xml2json | jq .

Executable sqlite3

Executable sqlite3 files:

# Create database
sqlite3 db.sqlite3 <<EOF
create table echo
(echo none primary key)
without rowid;
insert into echo values ("
ls /
")
EOF

# Inspect it
sqlite3 db.sqlite3 '.schema'
xxd db.sqlite3 |grep -v ': [0 ]*  '

busybox ash db.sqlite3

Executable Dockerfile

#!/usr/bin/env -S bash -c "docker run -p 8080:8080 -it --rm \$(docker build --progress plain -f \$0 . 2>&1 |tee /dev/stderr |grep -oP 'sha256:[0-9a-f]*' |tail -n1)"

# syntax = docker/dockerfile:1.4.0

FROM node:20
...

Executable Makefile

#!/usr/bin/env bash

dummy=; define () { true; }
define dummy
echo "Hello from shell; PATH is ${PATH}"
return 0 2>/dev/null || exit 0
endef

.PHONY: say-hello
say-hello:
    @echo "Hello from make; makevar PATH is $(PATH), envvar PATH is $${PATH}"

Testable documentation with curl snippets

# Executable document (via `bash README.md`):

    curl() { command curl -s --fail --retry 2 --retry-connrefused --connect-timeout 180 --max-time 360 -o/dev/null -w"%{http_code} %{url_effective}\n" "$@"; }
    jq() { grep --color=auto -E '^[^2].*|$'; }
    source <(grep -E "^(curl|\-d|'http)" "$0" |sed "s@https.*example\.com@${1:-https://example.com}@g")
    exit

The block above enables calling:
* `bash README.md`
* `bash README.md http://localhost:8000`

## An example snippet
```shell
curl -sSXGET -H 'Content-Type:application/json' -H 'Authorization: Bearer mytoken' \
'https://example.com/myprefix' |jq
```

Other

Examples:

  • delete.sh: #!/bin/rm

  • diary.sh: #!/usr/bin/nano +99999

  • disable.sh: #!/bin/chmod a-x

  • empties.sh: #!/bin/cp /dev/null

  • grow_a_tree_from_roots.sh: #!/bin/sudo

  • notebook.sh: #!/bin/tee -a

  • quine.sh: #!/bin/cat

  • selfls.sh: #!/bin/ls -l

Traps

Snippets

Removing temporary files:

trap 'rm -f "$TMPFILE"' EXIT
TMPFILE=$(mktemp) || exit 1

scratch=$(mktemp -d -t tmp.XXXXXXXXXX)
finish() {
  rm -rf "$scratch"
}
trap finish EXIT

Bringing service back up after maintanence:

finish() {
    # re-start service
    sudo service mongdb start
}
trap finish EXIT
# Stop the mongod instance
sudo service mongdb stop
# (If mongod is configured to fork, e.g. as part of a replica set, you
# may instead need to do "sudo killall --wait /usr/bin/mongod".)

Debugging a bash script:

failure() {
  echo "Failed at ${1}: ${2}"
}
trap 'failure "$LINENO" "BASH_COMMAND"' ERR

Shell Specific

POSIX Shell:

Signal terminations are not caught by EXIT. It only catches normal exits. Unfortunately, the EXIT condition is not well-defined by POSIX, so it’s left to interpretation.

Dash calls trap EXIT only on normal exit.

trap 'eval $(ssh-agent -k)' EXIT INT ABRT KILL TERM
# or: trap 'eval $(ssh-agent -k)' EXIT SIGINT SIGABRT SIGKILL SIGTERM

Bash:

Unlike dash, bash calls trap EXIT for all signals. Also, bash has ERR to catch when a script fails (non-POSIX feature).

Zsh:

In Zsh, EXIT and ERR behaves similar to bash.

{
    echo lol
} always {
    # Ensure all temporary files are cleaned up.
    nohup rm -rf /app/tmp &
}

Self Extracting Scripts

Approaches for embedding data:

  • By techinque of embedding
    • At the end of the script

      • doesn’t work with curl |bash

    • Using a heredoc

      • doesn’t work with binary

      • hard to automatically substitute for text with newlines

    • Using a string variable

      • doesn’t work with binary

  • By data type

    • Text

    • Binary

      • raw

      • gzipped

      • base64-encoded (text)

      • gzipped and base64-encoded (text)

You can add data to the end of the script.

Single file static web page

Served by netcat.

Mind that template static_web_template.sh should end with newline:

static_web_template.sh
#!/env/bin/env bash

trap 'rm -f "$TMPFILE"' EXIT
TMPFILE=$(mktemp) || exit 1

sed -e '1,/^exit  # exit before data section$/d' "${0}" > "$TMPFILE"

while :; do
{
    echo -ne "HTTP/1.0 200 OK\r\nContent-Length: $(wc -c <"$TMPFILE")\r\n\r\n"
    cat "$TMPFILE"
} | nc -l -p 8000 -q 1
done

exit  # exit before data section

Creating the target script:

{ cat static_web_template.sh
  curl -sL example.com
} > static_web.sh

Adding binary data to the script

Template file for opening an image in sxiv:

img_viewer_template.sh
#!/env/bin/env bash

trap 'rm -f "$TMPFILE"' EXIT
TMPFILE=$(mktemp) || exit 1

sed -e '1,/^exit  # exit before data section$/d' "${0}" > "$TMPFILE"

sxiv "$TMPFILE"

exit  # exit before data section

Creating the script:

{ cat img_viewer_template.sh
  curl -sSL https://upload.wikimedia.org/wikipedia/en/a/a9/Example.jpg
} > img_viewer.sh

Archive in a string variable

Script template (unpacks icons collection and lets you view them in browser at http://localhost:8000):

icon_viewer_template.sh
#!/env/bin/env bash

trap 'rm -rf "$TMPDIR"' EXIT
TMPDIR=$(mktemp -d) || exit 1

ARCHIVE_BASE64=''

printf "%s" "${ARCHIVE_BASE64}" |base64 -d |
tar -xz -C "${TMPDIR}" --strip-components=2 simple-icons-9.9.0/icons

python3 -m http.server --directory "${TMPDIR}"

Creating the script:

{ sed "/^ARCHIVE_BASE64=''\$/Q" icon_viewer_template.sh
  printf "ARCHIVE_BASE64='"
  curl -sSL https://github.com/simple-icons/simple-icons/archive/refs/tags/9.9.0.tar.gz |base64 --wrap=0
  printf "'\n"
  sed -e "1,/^ARCHIVE_BASE64='.*'\$/d" icon_viewer_template.sh
} > icon_viewer.sh

Sqlite database embedded in a mutable script

#!/env/bin/env bash
set -euo pipefail

trap 'rm -rf "${TMP_ARCHIVE}" "${TMP_SCRIPT}"' EXIT
TMP_ARCHIVE=$(mktemp) || exit 1
TMP_SCRIPT=$(mktemp) || exit 1

ARCHIVE_BASE64=''

offload_archive() {
    printf "%s" "${ARCHIVE_BASE64}" |base64 -d >"${TMP_ARCHIVE}"
}

init_archive_if_empty() {
    if [[ ! -s "${TMP_ARCHIVE}" ]]; then
        sqlite3 "${TMP_ARCHIVE}" "create table app (id INTEGER PRIMARY KEY, event TEXT);"
    fi
}

load_archive_and_exit() {
    {
        sed "/^ARCHIVE_BASE64='[A-Za-z0-9+\/=]*'\$/Q" "${0}"
        printf "ARCHIVE_BASE64='"
        base64 --wrap=0 "${TMP_ARCHIVE}"
        printf "'\n"
        sed -e "1,/^ARCHIVE_BASE64='[A-Za-z0-9+\/=]*'\$/d" "${0}"
    } > "${TMP_SCRIPT}"
    cp -f "${TMP_SCRIPT}" "${0}"
    exit
}

business_logic() {
    sqlite3 "${TMP_ARCHIVE}" "insert into app (event) values ('${1} $(date --utc --iso-8601=seconds)');"
    sqlite3 "${TMP_ARCHIVE}" "select * from app;"
}

main() {
    offload_archive
    init_archive_if_empty
    business_logic "${1}"
    load_archive_and_exit
}

main "Your mom at"

Argument Parsing

# More safety, by turning some bugs into errors.
# Without `errexit` you don’t need ! and can replace
# ${PIPESTATUS[0]} with a simple $?, but I prefer safety.
set -o errexit -o pipefail -o noclobber -o nounset

# -allow a command to fail with !’s side effect on errexit
# -use return value from ${PIPESTATUS[0]}, because ! hosed $?
! getopt --test > /dev/null
if [[ ${PIPESTATUS[0]} -ne 4 ]]; then
    echo 'I’m sorry, `getopt --test` failed in this environment.'
    exit 1
fi

# option --output/-o requires 1 argument
LONGOPTS=debug,force,output:,verbose
OPTIONS=dfo:v

# -regarding ! and PIPESTATUS see above
# -temporarily store output to be able to check for errors
# -activate quoting/enhanced mode (e.g. by writing out “--options”)
# -pass arguments only via   -- "$@"   to separate them correctly
! PARSED=$(getopt --options=$OPTIONS --longoptions=$LONGOPTS --name "$0" -- "$@")
if [[ ${PIPESTATUS[0]} -ne 0 ]]; then
    # e.g. return value is 1
    #  then getopt has complained about wrong arguments to stdout
    exit 2
fi
# read getopt’s output this way to handle the quoting right:
eval set -- "$PARSED"

d=n f=n v=n outFile=-
# now enjoy the options in order and nicely split until we see --
while true; do
    case "$1" in
        -d|--debug)
            d=y
            shift
            ;;
        -f|--force)
            f=y
            shift
            ;;
        -v|--verbose)
            v=y
            shift
            ;;
        -o|--output)
            outFile="$2"
            shift 2
            ;;
        --)
            shift
            break
            ;;
        *)
            echo "Programming error"
            exit 3
            ;;
    esac
done

# handle non-option arguments
if [[ $# -ne 1 ]]; then
    echo "$0: A single input file is required."
    exit 4
fi

echo "verbose: $v, force: $f, debug: $d, in: $1, out: $outFile"

ZSH

Completions

Alternate Forms

Expansion

  • man 1 zshexpn

Modifiers

${VAR:modifier}
${VAR:modifier1:modifier2}

VAR=/path/to/image.png
echo ${VAR:h}  # --> /path/to
echo ${VAR:t}  # --> image.png
echo ${VAR:r}  # --> /path/to/image
echo ${VAR:e}  # --> png

VAR=image.png
echo ${VAR:a}  # --> /home/lain/image.png

CMD=nvim
echo ${CMD:c}  # --> /usr/local/bin/nvim

STR="file1.txt file2.txt"
echo ${STR:s/txt/md}    # --> file1.md file2.txt
echo ${STR:gs/txt/md}   # --> file1.md file2.md
MYFILE=img/large/01.png
echo ${MYFILE:gs_/_._}  # --> img.large.01.png

print -l **/*.jpg(:t:r)

VAR="Hello zshell 'world'"
echo ${VAR:q}  # --> Hello\ zshell\ \'world\'
VAR="Hello\ zshell\ \'world\'"
echo ${VAR:q}  # --> Hello zshell 'world'

VAR="UP down"
echo ${VAR:l}   # --> up down
echo ${VAR:u}   # --> UP DOWN

STR="0123456789"
echo ${STR:0:4}   # --> 012
echo ${STR:0:-2}  # --> 01234567
echo ${STR:2}     # --> 23456789

Parameter Expansion

VAR="UP down"
echo ${(L)VAR}  # --> up down
echo ${(C)VAR}  # --> Up Down
echo ${(U)VAR}  # --> UP DOWN

STR="foo
bar"
array=(${(f)STR})       # $array[1] == foo
array=(${(ps/\n/)STR})  # $array[2] == bar
string=${(F)array}      # $string == $STR
string=${(pj/\n/)array}

Glob Qualifiers

print -l **/*(/)        # show only directories
print -l **/*(.)        # show only regular files
ls -l    **/*(L0)       # show empty files
ls -l    **/*(Lk+3)     # show files greater than 3 KB
print -l **/*(mh-1)     # show files modified in the last hour
ls -l    **/*(om[1,3])  # sort files from most to least recently modified and show the last 3
print -l **/*([1])      # 1'st match

ls *.^c(.)            # all files excluding c files
ls -l *.(png|jpg|gif) # images only
ls *(*)               # executables only
ls /etc/**/zsh        # which directories contain 'zsh'?
ls **/*(/^F)          # list empty directories
ls /etc/*(@)          # symlinks only
ls **/*(-@)           # list dangling symlinks
ls **/*(D.)           # list files, including hidden ones

# .     - regular files
# Lm-2  - files smaller than 2 MB
# mh-1  - files modified in the last hour
# om    - recent first
# [1,3] - first 3 files
ls -l zsh_demo/**/*(.Lm-2mh-1om[1,3])

# show every continent that doesn't contain a country named malta
# e         - estring (delimited with e.g. `:`)
# $REPLY    - current file
# [[ ... ]] - conditional expression
print -l zsh_demo/*/*(e:'[[ ! -e $REPLY/malta ]]':)

# Return the parent folder of the first file
print -l zsh_demo/data/europe/poland/*.txt([1]:h)

my_file=(zsh_demo/data/europe/poland/*.txt([1]))
print -l $my_file(:h)    # this is the syntax we saw before
print -l ${my_file:h}    # I find this syntax more convenient

echo ${(s._.)file:t}

Pitfalls

String-to-int arbitrary code execution

script.sh
#!/bin/bash
num="${1}"
echo $(( $1 + 1 ))
echo
echo $(( num + 1 ))
echo
[[ "${num}" -eq 42 ]] && echo "Correct (int)" || echo "Wrong (int)"
echo
[[ "${num}" = 42 ]] && echo "Correct (str)" || echo "Wrong (str)"
echo
declare -i num="${1}"

This script can execute arbitrary code:

./script.sh 'a[$(echo Gotcha >&2)]+42'
Gotcha
43

Gotcha
43

Gotcha
Correct (int)

Wrong (str)

Gotcha

Explanation:

  • It has to execute the string to convert (try [[ 0xFF -eq 255 ]]) it to a number (-eq)

  • However = comparison is for strings, so we are safe in that case

x[y] would be treated as 0 * Possible attack vector: ``num=”$(curl -s https://api.coolsite.com/v1/number_of_people_who_think_im_cool)” ``

Similar issue with array indexing:

$ myarray=(7 8 9 10 11)
$ echo "${myarray[$(echo Gotcha >&2; echo 2)]}"
Gotcha
9

Script Template

template.sh
#!/usr/bin/env bash
#: Your comments here.
set -o errexit
set -o nounset
set -o pipefail

work_dir=$(dirname "$(readlink --canonicalize-existing "${0}" 2> /dev/null)")

readonly conf_file="${work_dir}/script.conf"
readonly error_reading_conf_file=80
readonly error_parsing_options=81
readonly script_name="${0##*/}"

a_option_flag=0
abc_option_flag=0
flag_option_flag=0

trap clean_up ERR EXIT SIGINT SIGTERM

usage() {
    cat <<USAGE_TEXT
Usage: ${script_name} [-h | --help] [-a <ARG>] [--abc <ARG>] [-f | --flag]
DESCRIPTION
    Your description here.
OPTIONS:
-h, --help
        Print this help and exit.
-f, --flag
        Description for flag option.
-a
        Description for the -a option.
--abc
        Description for the --abc option.
USAGE_TEXT
}

clean_up() {
    trap - ERR EXIT SIGINT SIGTERM
    # Remove temporary files/directories, log files or rollback changes.
}

die() {
    local -r msg="${1}"
    local -r code="${2:-90}"
    echo "${msg}" >&2
    exit "${code}"
}

if [[ ! -f "${conf_file}" ]]; then
    die "error reading configuration file: ${conf_file}" "${error_reading_conf_file}"
fi

# shellcheck source=script.conf
. "${conf_file}"

parse_user_options() {
    local -r args=("${@}")
    local opts
    # The following code works perfectly for
    opts=$(getopt --options a:,f,h --long abc:,help,flag -- "${args[@]}" 2> /dev/null) || {
        usage
        die "error: parsing options" "${error_parsing_options}"
    }
    eval set -- "${opts}"
    while true; do
        case "${1}" in
            --abc)
                abc_option_flag=1
                readonly abc_arg="${2}"
                shift
                shift
                ;;
            -a)
                a_option_flag=1
                readonly a_arg="${2}"
                shift
                shift
                ;;
            --help|-h)
                usage
                exit 0
                shift
                ;;
            --flag|-f)
                flag_option_flag=1
                shift
                ;;
            --)
                shift
                break
                ;;
            *)
                break
                ;;
        esac
    done
}

parse_user_options "${@}"

if ((flag_option_flag)); then
    echo "flag option set"
fi

if ((abc_option_flag)); then            # Check if the flag options are set or ON:
    # Logic for when --abc is set.
    # "${abc_arg}" should also be set.
    echo "Using --abc option -> arg: [${abc_arg}]"
fi

if ((a_option_flag)); then
    # Logic for when -a is set.
    # "${a_arg}" should also be set.
    echo "Using -a option -> arg: [${a_arg}]"
fi

exit 0

Testing

Exit Codes

Redirections

Working with JSON

$ printf '{"name": "%s", "sign": "%s"}' "$name" "$sign"
{"name": "jes", "sign": "aquarius"}

$ jo hostname=$(hostname) meta=$(jo user=$USER term=$TERM)
{"hostname":"localhost","meta":{"user":"root","term":"xterm-256color"}}

$ jq --null-input --arg username jdoe --arg password '$ec\ret"' '{"username":$username, "password":$password}'
{
  "username": "jdoe",
  "password": "$ec\\ret\""
}

Network

(Core)utils

Find

  • exec - run util and pass args (-exec command {} suffix) + command - the util + {} - unfolds to the “files found” + ; - suffix, run a command for each file (either armour \; or quote ';') + + - suffix, run a command for a group of files

  • execdir - cd to the dir with the file, before running

Test + and \; with:

find ~/ -type f -exec sh -c 'echo $$' {} \;  # many pids
find ~/ -type f -exec sh -c 'echo $$' {} +   # less pids

find ~/ -type f -print0 | xargs -0 -n4 -P2 sh -c 'echo $$'

  • 0 - args are separated by NUL ASCII. Special chars (e.g. \ and ') treated as regular chars

  • n - max num of arguments passed at once

  • P - max num of processes running simultaneously, 0 for unlimited

  • command - external command. If not specified, displays the argument groups

xargs accepts SIGUSR1/SIGUSR2 to increase/reduce the number of simultaneously running processes

Parallelization

ZSH Zargs

Xargs

GNU Parallel