Common Code Snippets

Posted on | Edited on | In

More information please see man bash, it has comprehensive information.

This blog collects the commonly used code snippets based on my daily work, also do summary from related stackoverflow topics.

set builtin

Usually I use set -x for debugging purpose, today I see a new statement set -ex. What is this and what is set in Bash? 后来又知道了很多，见awesome list中的bash tutoral.

The Set Builtin, in short, set allows you to change the values of shell options and set the positional parameters, or to display the names and values of shell variables.

set -e, causes the shell to exit if any subcommand or pipeline returns a non-zero status. This tells bash that it should exit the script if any statement returns a non-true return value. The benefit of using -e is that it prevents errors snowballing into serious issues when they could have been caught earlier.

But sometimes set -e may not be good, see these two posts: What does ‘set -e’ do, and why might it be considered dangerous? 这个回答很有启发，用哪种方法还得看具体场景。一定要考虑清楚。

“set -e” usage

get path of running script

1
curpath=$(dirname $(readlink -f $0))

readlink -f $0 will follow every symlink in every component of the given name recursively and get the canonical path. A single file existing on a system can have many different paths that refer to it, but only one canonical path, canonical gives a unique absolute path for a given file. That means even though you call a script in it’s current directory, readlink -f $0 will give you the absolute path!

dirname $0 cut the script name to get the calling path, the path is relative not absolute.

run script in it’s driectory

Sometimes we want to run script in it’s folder by ./xxx.sh. we can check that:

1
2
3
4
5
SCRIPT_PATH=$(dirname $0)
if [[ "X""${SCRIPT_PATH}" != "X." ]]; then
  LogMsg "###### ERROR: Please run this script in it's directory!"
  exit 1
fi

create tmp file to store log

Create a temporary file or directory, this temp file is owned and grouped by the current user. Aside from the obvious step of setting proper permissions for files exposed to all users of the system, it is important to give temporary files nonpredictable filenames, for example:

1
2
3
4
# $$: current PID
OUT_FILE=/tmp/$(basename $0).$$.$RANDOM$RANDOM
# or
OUT_FILE=$(mktemp /tmp/log.$$.XXXXXXXXX)

For regular use, it may be more wise to avoid /tmp and create a /tmp under its home.

it will randomly generate 6 characters to replace XXXXXX. You may need to delete the tmp file when script exits, for example, use trap:

1
2
3
4
5
6
7
8
function exitHook {
  rm -f $OUT_FILE
  rm -f ${OUT_FILE}.yml
  rm -f ${OUT_FILE}.out
  rm -f ${OUT_FILE}.err
}
## must put at beginning of script
trap exitHook EXIT

Actually, you can get random number from

1
echo $RANDOM

you can also seed it to generate reproducible sequence: https://stackoverflow.com/questions/42004870/seed-for-random-environment-variable-in-bash

if condition

List of test command condition Or check manual man test.

The test command, it can be written as[] or test expression, [[ ]] is modern format, it supports regular expression =~ for string. which one if preferred: test is traditional (and part of the POSIX specification for standard shells, which are often used for system startup scripts), whereas [[ ]] is specific to bash (and a few other modern shells). It’s important to know how to use test since it is widely used, but [[ ]] is clearly more useful and is easier to code, so it is preferred for modern scripts.

1
2
3
4
## don't double quote regexp
if [[ "$name" =~ colou?r ]]; then
  echo "..."
fi

其他test 的变量operands 一般用double quote括起来，防止值为空的时候出错.

对于file system, 主要检测-e, -f, -d, -L, -r -w -x, etc. 还有更多的检测选择，参考man.

对于string 则主要就是检测-n, -z, =, ==, !=, =~, >, <.

For comparing integers, -eq, -ne, -ge, -gt, -le, -lt. Or use (( xxx )), this is a compound command designed for integers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
INT=-3
if [ -z "$INT" ]; then
  echo "INT is empty." >&2
  exit 1 
fi
if [ "$INT" -lt 0 ]; then
  echo "INT is negative."
else
  echo "INT is positive."
fi
if [ $((INT % 2)) -eq 0 ]; then
  echo "INT is even."
else
  echo "INT is odd."
fi

# or using (())
if ((1)); then echo "It is true."; fi
if ((0)); then echo "It is true."; fi

# 需要注意(()) 中的变量不再需要expansion symbol $了，直接用变量名
declare -i day=30
if (( day > 0 || day < 31 )); then
  echo "day is good"
fi

# 这里结合read command，判断输入是否是有一个item
read -p "input one item -> "
(( "$(echo \"$REPLY\" | wc -w)" > 1 )) && echo "invalid input"

== or =, != and =~ are used for string comparision:

1
2
3
4
# sth does not exist? or using -z
if [[ "${sth}""X" == "X" ]]; then
  LogMsg "###### INFO: ..."
fi

or

1
2
3
4
5
# True if the length of "STRING" is zero.
if [[ -z "${sth}" ]]; then
  LogMsg "###### INFO: ..." >&2
  exit 1
fi
1
2
3
4
5
# directory does not exist?
if [[ ! -d "${folder_path}" ]]; then
   LogMsg "###### ERROR: ${folder_path} directory doesn't exist!"
   exit 1
fi

对于logial operators, 有2种模式，一种是在command内部使用，比如: test(-a, -o, !), [[ ]], (())(&& || !):

1
2
3
4
5
6
if [[ "$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL" ]]
# same as test
if [ "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" ]
# note in test need escape
if [[ ! ("$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL") ]]
if [ ! \( "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" \) ]

Since all expressions and operators used by test are treated as command arguments by the shell (unlike [[ ]] and (( )) ), characters that have special meaning to bash, such as <, >, (, and ), must be quoted or escaped.

一种是外部使用的, provided by bash, for example: [[ ]] && [[ ]] || [[ ]], [[ ! xxx ]]. They obey short circuit rule.

Tips: 对于简单的if-condition, 可以替换为形如:

1
2
3
4
5
6
7
# chaining commands
[ -r ~/.profile ] && . ~/.profile
cat ~/.profile && echo "this is profile" || echo "failed to read profile"
test -f "$FILE" && source "$_" || echo "$_ does not exist" >& 2
[ ! -r "$FILE" ] && { echo "$FILE is not readable" ; exit 1 }
# parameters expansion 甚至都不需要if-condition
${var:="hello"}

select loop

The select loop provides an easy way to create a numbered menu from which users can select options. It is useful when you need to ask the user to choose one or more items from a list of choices.

Note that this loop was introduced in ksh and has been adapted into bash. It is not available in sh.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# PS3 is designed for select command
PS3="Enter your choice (must be a number): "
select DRINK in tea cofee water juice appe all none
do
   # After a match is found, no further matches are attempted.
   # don't need the double quote
   # the pattern match is the same as pathname expansion
   # for example: ???) [[:alpha:]]) *.txt)
   case $DRINK in
      tea | cofee | water | all) 
        echo "Go to canteen"
        break
        ;;
      juice|appe)
        echo "Available at home"
        break
        ;;
      none) 
        break 
        ;;
      # match anything at last
      *) 
        echo "ERROR: Invalid selection" 
        ;;
   esac
done

When select you can use index number or literal, if no break, it will loop forever. If want case to match more than one terms, use ;;& instead of ;; at end of each case. The addition of the ;;& syntax allows case to continue to the next test rather than simply terminating.

input password and confirm

Must not show password user input:

1
2
3
4
5
6
7
8
9
10
11
12
echo "****************************************************************"
echo "Please input the password:"
echo "****************************************************************"
while true; do
  read -s -p "PASSWORD: " PASSWORD
  echo
  read -s -p "CONFIRM:  " PASSWORD_CONFIRM
  echo
  [ ${#PASSWORD} -lt 6 ] && echo "The length of password at least 6, please try again" && continue
  [ "${PASSWORD}" = "${PASSWORD_CONFIRM}" ] && break
  echo "Passwords do not match please try again..."
done

script input parameters

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
if [ $# -eq 0 ]; then
  echo "No command-line arguments were specified..."
  # call Usage function here
  exit 1
fi

## case和C语言中有一样的性质，如果没有break，会继续对比接下来的选项
## 这里并不需要，因为shift 且没有相同的flags
while [ $# -gt 0 ]
do
  case "$1" in
    -p1)
       shift
       P1=${1}
       shift;;

    -p2)
       shift
       P2=${1}
       shift;;

    -h|--help)
       # Usage
       exit 0;;
       
    *) # Usage
       exit 1;;
  esac
done

[[ "X$P1" = "X" ]] &&  exit 1
[[ "X$P2" = "X" ]] &&  exit 1

Note there are 2 shift in one case, after each shift, $# minus 1.

function

The function refers to passed arguments by their position (not by name), that is $1, $2, and so forth. $0 is the name of the script itself.

1
2
3
4
5
6
7
8
9
function example()
{
  ## local var prevent var leaking to shell
  local first=$1
  local second=$2
  ## return code is similar to exit code but this is return
  ## will break the rest execution
  return <return code>
}

Need to call your function after it is declared.

1
2
3
4
5
example "p1" "p2"

args #0 is <absolute path to script itself>
args #1 is p1
args #2 is p2

Show functions:

1
2
3
4
5
6
## list all function names
declare -F
## show definition
declare -f [function name]
## clear a function
unset -f <function name>

Export functions, to make it available to subshells, similarly to export variables:

1
2
## -xf: export a function
declare -xf <function name>

log message

1
2
3
4
5
6
LogMsg()
{
  # parse input and reformat
  logMsg="$@"
  echo "["`date +"%Y/%m/%d %r"`"] " ${logMsg}
}
1
2
3
LogMsg "[INFO] ..."
LogMsg "[WARNING] ..."
LogMsg "[ERROR]..."

Actually, this style [INFO] [2019-10-11 15:59:26-0081] ... it better.

check last command result

1
2
3
4
5
6
7
echo_success_failure() {
  if [ $? -eq 0 ]; then 
    LogMsg "###### INFO: Success..."
  else 
    LogMsg "###### INFO: Failure..."
  fi
}

run as root

1
2
3
4
5
effective_uid=`id -u` 2>/dev/null
if [ $effective_uid -ne 0 ]; then
 LogMsg "###### ERROR: Please run this script as root or sudo"  
 exit 1
fi

IFS and read array

The default value of IFS contains a space, a tab, and a newline character. Convert string to array with specific delimiter, for example:

1
2
3
4
5
6
7
string="item1:item2:item3"
# <<<: is here string, the same as here doc but shorter single string
OLD_IFS=$IFS
IFS=':' read -a array <<< "${string}"
# or using process substitution
IFS=':' read -a array < <(echo "${string}")
IFS=$OLD_IFS

This version has no globbing problem, the delimiter is set in $IFS (here is space), variables quoted. Don’t forget to do sanity check after converting.

1
2
3
${array[0]}  ===> item1
${array[1]}  ===> item2
${array[2]}  ===> item3

Why we use here string rather than pipeline, for example:

1
echo "${string}" | read

这是不行的，因为pipeline 的本质是subshell, 但是read 需要更改当前parent shell的内容的。这里read 实际上在更改了subshell中$REPLY的内容，一旦command 结束，subshell就没了, parent shell 并没有变化.

此外，验证输入的正确性也很重要，一般用[[ =~ ]] regular expression 去检测了.

Actually if the string use spaces as delimiter, we can loop items directly:

1
2
3
4
5
string="item1 item2 item3"
for i in ${string}
do
  echo ${i}
done

loop array

break and continue can be used on loop. 还要注意当do 写在下一行的时候，do前面不需要;.

1
2
3
4
5
declare -a array=("element1" "element2" "element3")
for i in "${array[@]}"
do
   echo "${i}"
done

declare or typeset are an explicit way of declaring variable in shell scripts.

In BASH it is safer to quote the variable using "" for the cases when $i may contain white spaces or shell expandable characters.

If you want to use index of array element

1
2
3
4
5
6
7
8
9
# get length of an array
arraylength=${#array[@]}

# use for loop to read all values and indexes
for (( i=0; i<${arraylength}; i++ ))
do
  ## ${array[$i]} 这里注意，先解析的$i
  echo $i " / " ${arraylength} " : " ${array[$i]}
done

If we use declare to define a integer variable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
declare -i x=10
while (( x > 0 ))
do
  echo $x
  ## no need to use 'let x=x-1'
  ## because x is type integer
  x=x-1
done

# true loop 3 种写法
while true | while ((1)) | while :
do
 ## pass
done

Until loop continues until it receives a zero exit status.

1
2
3
4
5
6
count = 1

until [[ "$count" -gt 5 ]]; do
  echo "$count"
  count=$((count + 1))
done

In ZSH shell, you can use foreach loop:

1
2
3
4
## () is a must
foreach item (`ls /tmp`)
  echo $item
end

Another index loop using seq:

1
2
3
4
for i in $(seq 1 10)
do
  echo $i
done

read file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# read 3 fields a line, line by line from distros.txt file
# note that < is placed after done, it is the input for loop
while read distro version release; do
  printf "Distro: %s\tVersion: %s\tReleased: %s\n" \
    "$distro" \
    "$version" \
    "$release"
# no need cat here
done < distros.txt
# or
done < <(cat distros.txt)

# can also pipeline input to a loop
# while and read is running on subshell
sort -k 1,1 -k 2n distros.txt | while read distro version release; do 
  printf "Distro: %s\tVersion: %s\tReleased: %s\n" \
    "$distro" \
    "$version" \
    "$release"
done

# using process substitution
# list last 3 lines of dir
while read attr links owner group size date time filename; do
  cat << EOF
    Filename: $filename
    Size:     $size
EOF
done < <(ls -ltrh | tail -n +2)

chmod

chmod recursively for directory and it’s content

1
chmod -R 0755 <target directory>

Or only add executable for file

1
find . -name '<file name>' -type f | xargs chmod +x
1
-rwxr-xr-x ...

pass parameters to script for read

Read can read from keyboard input or file or pipeline: read [-options] [variables...]. If no variable name is supplied, the shell variable $REPLY contains the line of data. If read receives fewer than the expected number, the extra variables are empty, while an excessive amount of input results in the final variable containing all of the extra input.

1
2
3
4
5
6
7
8
9
10
11
12
13
# pass parameters to read command
# must stick to this format
echo "admin
123456" | ./script.sh

# receive code snippet in script.sh
# ${username}  ===> admin
# ${password}  ===> 123456
echo -n "Please enter username -> "
read username
echo -n "Please enter an password -> "
# -s: silent
read -s password

Other options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# -p: prompt
read -p "Enter one or more values > "
echo "REPLY = '$REPLY'"

# -t: timeout
# -s: silent
if read -t 10 -sp "Enter secret passphrase > " secret_pass; then
  echo -e "\nSecret passphrase = '$secret_pass'"
else
  echo -e "\nInput timed out" >&2
  exit 1
fi

# -e: pair with -i
# -i: default valut passed to read
read -e -p "What is your user name? " -i $USER
echo "REPLY = '$REPLY'"

setup ssh password-less

Idempotence：

1
2
3
4
5
6
ssh-keyscan -H ${remote} >> ~/.ssh/known_hosts
sshpass -p "<password>" ssh-copy-id -i ~/.ssh/id_rsa.pub root@${remote}
if [[ $? -ne 0 ]]; then
  LogMsg "######ERROR: Something went wrong with ssh-copy-id. Check for incorrect credentials ... "
  exit 1
fi

recursive call

1
2
3
4
5
6
7
8
example()
{
  <execute sth>
  if [[ $? -ne 0 ]]; then
       LogMsg "######ERROR: Something went wrong… "
       example
  fi
}

tee command

tee command reads the standard input and writes it to both the standard output and one or more files, -a flag used to append output to existing file, if no -a, tee will create the file if not exist.

1
2
3
4
5
LogMsg()
{
  logMsg="$@"
  echo "["`date +"%Y/%m/%d %r"`"]" ${logMsg} | tee -a logs/ds_${stage}_${timeStamp}.log
}
1
2
3
4
5
6
7
8
9
10
# 注意这里tee 有2个方向的输出，可以用来检查pipeline的中间输出是什么
+-------------+     +-------+    +--------------+
|  command    |     | tee   |    |   stdout     |
|   output    +---->+       +--->+              |
+-------------+     +---+---+    +--------------+
                        |
                    +---v---+
                    |  file |
                    |       |
                    +-------+

statement block

这个很有意思，之前都没见过: {} Statement block in shell script

do something after reboot

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/usr/bin/env bash
# this script will do sth after reboot 
# in /root/completeme.sh
# then restore /etc/profile
#################################################

echo "Warning! This script is going to reboot now to complate the procedure"
echo "After reboot, login as root to perform the final steps"
echo "Press Ctrl-C now to stop this script in case you don\'t want to reboot"

## heredoc
cat << REBOOT >> /root/completeme.sh
## do sth after reboot

touch /tmp/after-reboot                            
rm -f /etc/profile
mv /etc/profile.bak /etc/profile
echo DONE
REBOOT

chmod +x /root/completeme.sh
cp /etc/profile /etc/profile.bak
## after reboot /etc/profile will be executed so /root/completeme.sh
echo /root/completeme.sh >> /etc/profile
reboot

monitor CPU load

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env bash

## to increase CPU load
## dd if=/dev/zero of=/dev/null
## or use stress command!

while sleep 60
do
  ## to remove header of ps output, append `=` or user --no-headers flag
  ## CPU$ 0.0 will be in part if CPU$ > 0.0
  REC=`ps -eo pcpu= -o pid= -o comm= | sort -k1 -n -r | head -1`
  USAGE=`echo $REC | awk '{print $1}'`
  ## truncate decimal part
  USAGE=${USAGE%.*}
  PID=`echo $REC | awk '{print $2}'`
  PNAME=`echo $REC | awk '{print $3}'`

  # Only if we have a high CPU load on one process, run a check within 7 seconds
  # In this check, we should monitor if the process is still that active
  # If that's the case, root gets a message

  ## man test
  if [ $USAGE -gt 80 ] 
  then
    USAGE1=$USAGE
    PID1=$PID
    PNAME1=$PNAME
    sleep 7
    REC=`ps --no-headers -eo pcpu,pid -o comm= | sort -k1 -n -r | head -1`
    USAGE2=`echo $REC | awk '{print $1}'`
    USAGE2=${USAGE2%.*}
    PID2=`echo $REC | awk '{print $2}'`
    PNAME2=`echo $REC | awk '{print $3}'`
    
    # Now we have variables with the old process information and with the
    # new information

    [ $USAGE2 -gt 80 ] && [ $PID1 = $PID2 ] && mail -s "CPU load of $PNAME is above 80%" root@blah.com < .
  fi
done
0%