Advanced Shell Techniques

In the previous article, we explored how to efficiently interact with our operating system through the command line. As technology continues to evolve, the need for efficiency and automation in the development process becomes more important. Advanced shell techniques and shell scripting play an important role in achieving this.

This article builds on the basic knowledge from our previous discussion and takes you a step further to advanced shell techniques.

Advanced Examples Combining Multiple Shell Commands

Pipes and redirection can be combined to perform complex operations.
Example:

ls -l | grep "file" | sort > sorted_file_list.txt

The redirection operator links a command to a file while a pipe operator connects one command to another. Commands used within pipelines are often referred to as filters. Examples include:

• cat – Combine and display file content
• sort – Arrange lines of text
• uniq – Identify or remove duplicate lines
• grep – Search for lines that match a pattern
• wc – Count lines, words, and bytes in a file
• head – Show the beginning of a file
• tail – Show the end of a file
• tee – Directs output to a file while also displaying it on the terminal.

The tee command is useful for real-time logging and monitoring.

• Example: ls -l | tee filelist.txt lists files in long format, saves the output to filelist.txt, and shows it in the terminal.

To append data to a file using tee, you can use the -a option:

• Example: echo "Log entry" | tee -a logfile.txt adds "Log entry" to logfile.txt and displays it on the terminal.

What is Shell Expansion ?

Shell expansion is a command-line feature that allows the shell to interpret and modify commands before they are executed. It allows users to perform complex operations using shortcuts and patterns.
There are several types of shell expansions, each serving a different purpose.

Types of Shell Expansion

• Pathname Expansion
• Tilde Expansion
• Parameter Expansion
• Command Substitution
• Arithmetic Expansion
• Brace Expansion
• Quote Removal

Pathname Expansion

Pathname expansion allows you to use wildcards to match filenames and directories. The shell expands these patterns to match existing files and directories.
The most common wild characters are:

• *: Matches any number of characters.
• ?: Matches exactly one character.

• [ ]: Matches any one of the enclosed characters or a range.

  • Example: ls *.txt lists all files in the current directory that have a .txt extension.

ls D* lists all filenames that begin with the character "D"

Tilde Expansion

Tilde expansion replaces the tilde (~) with the path to the current user's home directory.

Examples: cd ~ changes to the user's home directory.

• ~username: Expands to the specified user's home directory.

Parameter Expansion

Parameter expansion replaces a variable or parameter with its value.

Syntax: $variable or ${variable}

• Example: echo $HOME prints the current user's home directory.

Command Substitution

Command substitution allows the output of a command to be used as input for another command.

• Syntax: `command` or $(command)
  • Example: echo "Today is $(date)" inserts the current date into the string.

Arithmetic Expansion

Arithmetic expansion performs arithmetic operations and returns the result.

• Syntax: $((expression))

num=10; echo $(($((num ** 2))*5)) prints 500.

Brace Expansion

Brace expansion generates a sequence of strings or a combination of elements enclosed in braces.

It helps to create a list of directories at once

• Example: mkdir file{1,2,3}.txt creates file1.txt file2.txt file3.txt.

Quote Removal

Quote removal is the final step in the expansion process. It strips the quotes from the results of other expansions.

• Examples:
  • Single quotes (') prevent all expansions. Example: echo '$HOME' prints $HOME.
  • Double quotes (") prevent pathname expansion and quote removal but allow variable and command substitution. Example: echo "Home: $HOME" prints Home: /home/user.
  • Backslash (\) escapes the next character, preventing its expansion. Example: echo \$HOME prints $HOME.

Combining Expansions

Expansions can be combined to create powerful commands. Here's an example that uses multiple types of expansions:

# Brace and tilde expansion
mkdir ~/backup/{2021,2022,2023}

Job Control in Shell Programming

Job control allows you to manage multiple processes within a single shell session. A process is a running instance of a program, and understanding how to manage these processes is important for efficient multitasking in the shell.

• Foreground Process: This is the default mode for running processes. The shell waits for the foreground process to complete before you can enter another command. The terminal is occupied by the running process.

  • Example: Running nano file.txt opens the nano text editor, and the terminal waits until you exit nano to accept new commands.

• Background Process: This allows you to run a process in the background without occupying the terminal, allowing you to continue using the shell. You can start a process in the background by adding an ampersand (&) to the command.

  • Example: Running nano file.txt & starts nano in the background, allowing you to use the terminal for other commands.

Commands: bg, fg, jobs, kill

These commands are used to manage background and foreground processes:

• jobs: This command lists all jobs running in the background of the current shell session.

  • Example: jobs might display:

  [1]+  Running   nano file.txt &
  [2]-  Running   sleep 100 &
  

• ps: The ps (process status) command is used to display information about running processes.

• bg: This resumes a suspended job in the background.

  • Example: If you have a suspended job [1]+ Stopped nano file.txt, you can resume it in the background with bg %1.

• fg: This brings a background job to the foreground.

  • Example: If you have a background job [1]+ Running nano file.txt &, you can bring it to the foreground with fg %1.

• kill: Sends a signal to a process, usually to terminate it. The most common signal is SIGTERM (signal 15), but SIGKILL (signal 9) can be used to forcefully terminate a process.

  • Example: kill %1 sends SIGTERM to job number 1. If the process does not terminate, you can use kill -9 %1 to forcefully kill it.

How to Suspend a Process

A process can be suspended without being terminated, this is always useful when you need to check something else or start another task in the shell.
You can temporarily suspend a foreground process by pressing Ctrl + Z.

• Example: If you are editing a file in nano and press Ctrl + Z, the shell will suspend nano and return you to the command prompt. The suspended job can be seen with the jobs command and can be resumed later.

You can resume a suspended process in the background using the bg command or bring it back to the foreground with the fg command.

• Example: If you suspended nano with Ctrl + Z, you can bring it back to the foreground with fg %1 or resume it in the background with bg %1, where 1 is the job number shown by the jobs command.

How to Terminate a Process in Shell

When a foreground process is not responding as expected or taking too long, you can terminate it by pressing Ctrl + C.
This sends the SIGINT (interrupt) signal to the process, telling it to stop immediately.

• Example: If you're running a command to download a large file using curl:

curl -O https://example.com/largefile.zip

If you realise that the download is taking too long or you no longer want to download the file, you can press Ctrl + C to stop the curl process immediately. This will cancel the download and return you to the command prompt.

Downloading Files with curl and wget

You can download files or retrieve web content directly from the command line using curl and wget.

• curl: curl is a command-line tool that allows you to transfer data from or to a server. It supports various protocols, including HTTP, HTTPS and more. You can use curl to download files, make web requests, and even send data.

  • Example: To download a file using curl, you can use:

  curl -O https://example.com/filename.zip
  

  The -O option saves the file with the same name as the remote file.

curl can also be used to send POST requests, upload files, or even interact with APIs. For instance:

curl -X POST -d "param1=value1&param2=value2" https://example.com/api/endpoint

• wget: wget is also used for downloading files from the web. It's particularly useful for downloading large files or recursively downloading entire websites.

  • Example: To download a file with wget, simply run:

  wget https://example.com/filename.zip
  

  This will download the file and save it to the current directory.

  • Recursive Downloads: wget can also be used to download entire websites:

  wget -r https://example.com/
  

  The -r option tells wget to download recursively, grabbing all linked pages and resources.

Working with Shell Commands

It can be overwhelming to memorize the vast array of commands. However, there are built-in features that can help you remember and effectively use them. Example:

• Man
• Help
• Which
• Type
• Alias

man

The man (manual) command displays detailed documentation for other commands, often including a description, usage syntax, options, and examples.

• Usage:
  • man command

help

The help command provides information about built-in shell commands. It's a great starting point when you need a quick overview of a command's functionality.

• Usage: help command

which

The which command shows the location of executables in your system's PATH. It's useful for identifying which version of a command is being executed.

• Usage:
  • which command

type

The type command displays information about how a command will be interpreted by the shell, indicating whether it's a built-in command, an alias, a function, or an external executable.

• Usage:

  • type command
  • Example:

  type ls
  

alias

The alias command allows you to create shortcuts for commands you use frequently.
Usage:

• alias name='command'
• unalias name: Removes the alias.

• alias: Lists all current aliases.

• Examples:
  

  alias ll='ls -la'
  alias gs='git status'
  

Keyboard Tricks Every Shell Programmer Must Know

Mastering keyboard shortcuts can make your command-line experience much more efficient. Here are some essential keyboard tricks for the shell:

Command Line Navigation

Editing Text

Undo and Redo

Command History

Miscellaneous Shortcuts

We will explore shell scripting in the next article.