Bash Script Skills Upgrade For Dev & Ops

Welcome back gentle reader. My name is Silent-Mobius, AKA Alex M. Schapelle, your 32bit register pointer, and I wish to introduce you to topics of Open-Source, Linux/UNIX, DevOps and others.

As for today's topic I'd like to dwell on things trivial, yet not always siting right with developers and operations taking in together: Correct Way to right Shell Script.

While it can be claimed, that there is no single way to write shell scripts, it also can be claimed that good tips will make code cleaner and easy to write, read and eventually debugged.

In this article, I'd like to provide a with list of tips, tricks and snippets for you, gentle reader, to help you in your daily scripting tasks and challenges, based on my learning's and experiences. Of course, these suggestions are only my humble opinion, yet I strongly suggest to try these tips and decide afterwards. Let us begin

Editors

Although a war can be spanned from by mentioning a editor is a better then other editor, yet several things need to be said:

• Graphical user interface editors like, vscode, vscodium, atom, lapce, zed or any other ones are fine, yet they are not always an usable/accessible option:
• If you work with terminal/shell, you need to know how to edit files with that terminal/shell.
• Having a "deep know how" on terminal tools can extend your expertise.
  • My preference is with VIM, yet vi, nano, pico, emacs, ed or any other tool, that you find preference with, is also fine.

When it comes to choosing terminal editor, as mentioned, I'll be using Vi Improved, AKA VIM, and only reason for me to choose it, because when my initial parts of knowledge were assembled, the only tool that was provided to me was VIM, and I was required to master it.

Each tool mentioned above, has it's own running configuration file that each of us configures to his/her/its own heart/circuits preference. VIM is not an exception and usually in every system, will have local .vimrc config file in users home folder, or global /etc/vimrc configuration file. VIM usually uses the first one it finds from user home directory, and in case it does not finds one, it gets to other folders, till it reaches the global folder.

In case you are like me, yet did not had invested in configuring your VIM, here is my minimalist .vimrc configuration file that helps me with my scripting task:

set nocompatible

if has("syntax")
  syntax on
  set number
  " set tabs to have 4 spaces
  set ts=4
  set autoindent
  " show the matching part of the pair for [] {} and ()
  set showmatch
endif

colorscheme desert

Some clarifications:

• set nocompatible: making it uncompatable with vi
• if has("syntax"): condition that sets bunch of options in case it is more then text file file
• colorscheme desert: setting theme of colorful desert theme which makes minor things to pop out.

As mentioned, this is just suggestion, not an requirement.

Initials

When it comes to shell scripts, usually they are written as an utility and thus are used some high/low level programming language to shorten the access to some resource, instead of 'inventing the wheel'. There are cases that Devs prefer to derive their programming language rules, to the shell scripts and Ops prefer not abide any rules at all.

Whether these are correct or not, can be argued, while still practices that I've encountered:

• If Devs derive rules of specific programming language to shell scripts, for example duct/camel/snake typing, Do invest in Ops team to educate them.
  • Add syntax analysis check on CI level to verify it. Shellcheck is the tool that provides help.
  • Invest in teaching juniors by explaining logic behind it.
• If Ops have decided on shell scripting standard, for example POSIX, Share Those Standards With Dev Team!!!
  • Also add those to CI.
  • Justify those standards with UNIX/Linux logic.
• Do Code Review To Each Other Based On Standard Set
• If the script is stand alone script, add an .sh extension
• If the script is going to be used on global level, for example as a command, remove .sh extension and move the script to /bin or /usr/bin folder.
• If script is going to be used by some application, remove .sh extension , create bin folder in application home folder and place the script in there, while editing environment variable $PATH of your system with bin folder path you created, for easy access by application and by you while in development/testing/CI.

Safe header

In essence, scripts are nothing more then, a file with logically structured set of commands that we can execute from shell by passing it to sh or bash commands, e.g. bash myscript.sh.

We elevate those command including files, by adding headers known as sha-bang that looks like this: #!.
Sha-bang ensures that commands written in file, will be executed from start to end with the shell of your choosing, e.g #!/bin/bash. Due to compatibility with other scripting languages, I'd suggest to use env command to dynamically get path to your shell, as shown here: #!/usr/bin/env bash.

#!/usr/bin/env searches full path for bash shell, and in some cases it is not located at/bin, particularly on non-Linux based systems. For example, on FreeBSD systems, it's in /usr/local/bin, since it was installed as an optional package.

The question might pop up in your asking, 'What's secured header then ? Secure header, in my opinion, can be defined as combination of shell settings with set and/or shopt command capabilities while developing and running the shell programs. for example:

#!/usr/bin/env bash
###############################
#Created by:  Silent-Mobius
#Edited by :  Alex M. Schapelle
#Purpose   :  Shell script example
#Version   :  0.0.0
#Date      :  02.02.2024
set -o errexit  #if error happens, exit
set -o pipefail #if pipe has fail, exit
set -o nounset  #if any unset variable in script, exit
###############################

Each set command created some what of obstacle in terms of developing the script for the task, yet it also gives us opportunity to develop in more safer manner.

Although other parts of comments are not required and can be easily detected by version control tool such as git, it can be useful to use Created and Edited to have reference to Dev and Ops that were involved in usage and development of the script.

It is suggested to read more about set command either from shell with help set or from all over the internet.

Imports and Sources

In some of the use cases, each script that we create, requires environment variable, functions or some type of early defined value taken from somewhere. In those cases it is suggested to import to be precise source the files from the destination. Usually it is done before defining any variables. For example:

...
. /etc/os-release

The issue with this is an import path issue, where there is no compatibility between developing environment and production environment, or path is not from the OS that you are using, but from NFS/SAMBA/remote storage that is not always accessible.

For that it is suggested to source files with condition check on the path/mount/remote storage :

...
[[ -e /etc/os-release ]] && . /etc/os-release || . /etc/rhel-release

One may also use if .. else conditioning as well, yet in cases, where sourcing a lot of files is a requirements then it may become some what overwhelming.

Variables

Next step in our journey is variables. They are used to save data values in accessible manner while out shell program is running. The variable names may start with capital and lower case letters and may include numbers, however they may not start with numbers and can not include special characters. To declare variable in shell script, just choose variable name and assign it a value.
Choosing variable is not mere task, mostly because it variable name need to be descriptive. When choosing variable name, consider it's purpose.
It is considered a best practice to define all the variables at the beginning of your shell program, although defining those values as you go is also acceptable, depending on the task at hand.
My personal tip on the matter, would be to use variables only with capital letters, e.g

...
SCRIPT_NAME=$0
FIRST_POSITIONAL_ARGUMENT=$1

Dynamically generated variable values

In some cases variables should have initial values, yet not always those values are dynamic and change from system to system and from time to time, thus need to be checked every time the script is invoked as provided below:

...
DATE="$(date '+%Y.%m.%u-%H:%M')" 
# double quotes ensure that it will be string

Note: Shell script do not have data types, but the type are interpreted as scalars, meaning interpreted type of a digit, 0-9, in some cases are integers and in other as strings, thus it is a good practice to double quote every dynamically generated value.

The issue with dynamic data, is that there are cases where is does not exists, thus your script might fail as a consequence of that.
Best suggestion on the matter is to use shell's variable expansion capability and in case of empty value to use a default instead:

...
USERNAME_POSITIONAL_ARGUMENT=${1:-'user'} 
# if user won't provide positional argument, name `user` will be used

In case, dear reader, you need all variable expansion summary, you may find bash hints on devhint very insightful.

Implicit VS. Explicit

When working with variables, we often make mistake of providing it values based on our understanding of the task at hand, implicitly claiming that value provided is based on some value of path or environment variable, which, unfortunately, your script can not know what you know or it even can not assume half of our knowledge. Thus, I suggest to use explicit declaration of values in order not to fall in small pits of errors.

What does it mean though ?

• When declaring value for path -> always use absolute path
• When checking for dependency tool/package -> use package manager to check it
• When checking for environment variable -> set value instead of it, if it does not exists
• When using some tool -> use which command to validate that it exists

Note: Examples can be many, yet these should suffice to be a type of guideline.

Conditions

When it comes to testing our shell code, most of us are some what tempted to use several ways for testing. first of all, if statement is NOT only way to test things, but you can use bash built-ins and POSIX utilities, that can look like this:

• [ ] or test : Binary utility that is located in all POSIX compliant shell, that provides environment testing as well as variable comparison and validation.

  • The downside being that you have too many exit codes, that you need to handle in case they occur.
  • Can NOT use REGEX.
  • Limits Bash shell, but perfect in case of sh, csh,ash, and tcsh.

• () : Also known as command substitution, provides with running several commands in sub-shell and returning exit status, yet it is less useful with conditions, and in my opinion it is mostly suggested to be used as dynamic variable data generator. E.g
  
  

...
APP_PATH="$(find / -name '*regex*_of_[pP]ATH')"

• [[ ]] : Upgraded test utility which is a shell ** builtin **, unlike test which binary file included on your *nix system
  • Returns only 0 or 1, depending on the output of the expression
  • Can use regex and comparison of regex to input string
  • Can use logical and && in addition to logical or || in condition checks

Note: test and [[ ]] require white-space around the elements of comparison. In cases where white-space is missing, error is provided and if you've set errexit as suggested the script should stop, if not, then good luck debugging.

Now the neat part of all this discussion is that if statement, essentially checks whether the value in front of is zero or not:

• In case of zero, it access the condition and performs the required task.
• In case of non-zero value, it does not do anything inside scope of condition.

-> Note: I was requested to add this part, and hope I did not disappoint.

Functions, variables and in between

Another topic of variables, can be overlapping with functions. Unlike most of programming languages, Shell script variables are global variables meaning, that they can be reached from any part of the program. When creating variables in function, it's names should be lower case, and descriptive, it won't get overridden with before declared Capital Letter Variables.

...
function hello(){
  name=Silent
  last_name=Mobius
    echo "$name-$last_name"
}

It is also common to use local command, setting a variable to scope of the function. Although many developers I am familiar with, do use this syntax, in my opinion it is less readable and even less understandable, if one has never saw the local key word. Yet examples still need to be provided

...
NAME=Alex
function hello(){
  local NAME=Silent 
  # The value set in function is Silent 
  # and Not Alex as in global variables
    echo "$NAME-Mobius"
}

Function names also have a meaning, and it is suggested for functions names to be as describable. In this case I prefer to use Python scripting language typing system called snake_case. In cases where function name will be consistent of 2/3/4 words, we'll connect them with underscore:

NAME=Alex
LAST_NAME=Schapelle
function say_hello(){
    echo "hello $NAME $LAST_NAME"
}

Yet there are also cases where function checks whether some type of condition has occurred or not and notifies other functions in retaliation. It is suggested to start those functions with is word in function name:

...
USER=alex
function is_user_exists(){
   if grep $USER /etc/passwd > /dev/null 2>&1;then
      echo Exists
   else
      echo Not Exists
   fi
}
# Note: some of these suggestions are borrowed from other
# programming languages such as Python, Go, C and others

While on the subject of functions, it is also suggested that, functions themselves would not print anything, unless they require to provide some type of string or combination of characters. Instead it is customary to use return,true and false keywords that provides a digital number indicating exit status of the function:

...
USER=alex
GROUP=wheel
function is_user_exists(){
   if grep $USER /etc/passwd > /dev/null 2>&1;then
      return 0
   else
      return 1
   fi
}
function is_group_exists(){
   if grep $GROUP /etc/group > /dev/null 2>&1;then
      true
   else
      false
   fi
}
# Note: some of these suggestions are borrowed from other
# programming languages such as Python, Go, C and others

Other example of function usage with local keyword can be when you wish to pass values to function:

...
FILE=/etc/passwd
function is_this_exists(){
   local IN=$1
   if [[ -e $IN ]];then
     true
   else
     false
   fi
}

is_this_exists $FILE

The question might arise, about functions and outputs of the program you write in shell script language: If functions do not print anything, what does print output and what writes to log ? There is whole article dedicated for that topic and this time I'd prefer not to dive in to that puddle.

Script structure

Up until now we've touched internals of our shell program, yet, in my humble opinion, there is much to discuss, when it comes to program itself.
Shell programs, much like C and Python have start-to-bottom runtime behavior, meaning that the moment we invoke the script, bash or any other shell based language for that matter, will scan the file and in case there will not be any errors, coming from our syntax or safe header, which we set at the beginning, it will run the content and perform the tasks it has defined inside of itself.

While at start of the script development, it will not matter, with the time solidity will start to form: the more you write, the harder it is to maintain what ever you have written...

Thus I propose to use alternative method of shell script inherited from C and Go programming languages: main entry point of script:

#!/usr/bin/env bash 
################################
... safe header -> not writing to keep it brief
################################
FILE=/etc/passwd
function main(){ 
# main logic of the script that will be 
# summoned by the end of the file
# here we write main logic of our script
   echo "[+] Main Logic"
       is_this_exist  $FILE
       do_that_thing
}

function is_this_exist(){ # some  function that imports from other file
   [[ /import/from/script.sh ]] && . /import/from/script.sh 
}

function do_that_thing(){ # some function that uses function from other file
  command_or_function_from_import_script
}

######
# Main - _- _- _- _-Do Not Remove- _- _- _- _- _
######
main # this is where the script really starts to run

Great thing about structure above, is the capability it enables -> reading a script in story like manner:

• Every book has content list which includes reference to whole structure of the book
  • The main function includes those instructions and runs them
  • This type of programming can also be called Functional programming
  • In case something will fail, the safe header will stop the program and error function/line will be printed:
  • If your script is several hundred lines long, it might be hard to find the issue
  • Having main function as reference to all other functions will enable easier debugging steps.

Note: writing stories is not everyone's forte, and not everyone will agree with statement above, yet from my experience, people struggle when the programs are written in cryptic way, which makes me hope that using the structure above will help some one to achieve better way of development of their story telling

Summary

As stated at the beginning, dear reader, I humble wish you to succeed in you journey of becoming better in everything you do with your *nix box.

My hope is that this article, this structure, these scripting tips, will enable you to conquer new heights and challenges. While still reading this article, do surf into our profile to read my other articles, and do not hesitate to subscribe and like or comment.

I hope this was pleasant reading for you, as it was pleasant writing for me. Now, What ever you do - remember: Do Try To Have Some Fun.

Links

• Arnold Robbins
• Chris F.A. Johnson
• Robert Martin AKA Uncle Bob
• FullStackPython VIM