Back to bash: [ vs [[, A case of input interpretation

One can generally get by, by knowing a loose set of rules that bash uses to parse input into commands. There are however some commonplace features that are handled differently, notably aliases and the [[ operator. Understanding these differences can save you quite a bit of googling stackoverflowing in the long run. It also helps to solidify your understanding of those features themselves. If you catch yourself looking up similar things over and over, it may help to dive a bit deeper in the inner workings of bash. Speaking of diving in, let's!

Do I really need to know this?

Before going into some dry rules and definitions, let's make things a bit more concrete. Ask yourself if you really know the difference between [ and [[. Was it really necessary for bash to add [[ on top of the Bourne shell? Couldn't we get the same behavior there?

Both of these commands provide the means to do all kinds of conditional tests on files, strings and, in the case of [[ arithmetic comparisons. If a test succeeds, it will return an exit status of 0(success), otherwise 1(error). This means that it can be combined nicely with other features, such as &&.

$ touch test.txt
$ [ -f test.txt ] && echo "file exists!"
file exists!
$ [[ -f test.txt ]] && echo "file exists!"
file exists!

Great, but the commands behave exactly the same. Now let's look at some cases where the commands diverge.

$ touch test2.txt
$ [[ -f test.txt && -f test2.txt ]] && echo "test"
test
$ [ -f test.txt && -f test2.txt ] && echo "test"
bash: [: missing `]'
$ [ -f test.txt ] && [ -f test2.txt ] && echo "test"
test

If we want to combine conditional tests with [, we have to be a bit more verbose. The key difference is that [[ is special to bash, and as such has its own rules of operation. On the other hand [ is a command like any other (with ] as the last argument), and is bound to follow some general rules. In this case [[ overrides the behavior of &&. Adhering to general rules, the second test fails as it first tries to run [ -f test.txt and upon succes also -f test2.txt ]. Note that [ does have it own, less readable syntax to perform an and test ¹

Another important example involves the need for quoting with [.

$ PYTHONPATH="/some/path"
$ [ -n $PYTHONPATH ] && echo "Python path set"
Python path set
$ PYTHONPATH="/some/path/with spaces"
$ [ -n $PYTHONPATH ] && echo "Python path set"
bash: [: /some/path/with: binary operator expected
$ [ -n "$PYTHONPATH" ] && echo "Python path set"
Python path set
$ [[ -n $PYTHONPATH ]] && echo "Python path set"
Python path set

With [, bash will expand $PYTHONPATH, and split that into multiple parts based on whitespace. Subsequently bash will run [ -n /some/path/with spaces ]. The command will not know what to do here, as there are too many operands. On the other hand [[ again overrides the default behavior to make things easier and less error-prone. How this happens will be explained in more detail in the next section. Also consider this contradictory example, in which opposite tests both succeed.

$ [ -n $UNDEFINED_VAR ] && echo "Test succeeded"
Test succeeded
$ [ -z $UNDEFINED_VAR ] && echo "Test succeeded"
Test succeeded

This happens because, as the variable is empty, the command will run as [ -n ]. If [ receives a single argument (not counting the ]), it will default to checking if the argument is empty or not, which is equivalent to [ -n -n ]. That last one looks a bit confusing because the string happens to be -n as well.

Because of these odd cases, one needs to be careful about quoting, or rather not quoting when using [.

Interpreting rules

To fully explain and understand the differences between [ and [[, we need to go over some terminology and rules of operation. Bash continuously executes the following steps:

1. Parse input into words and operators, where quoting rules
   can be used to discard the special meaning of characters. More on that here

   

   Back to bash: Quoting

   Kristof Bruyninckx ・ Jan 10 '22

   #bash #linux #shell
   • Word: Any collection of characters treated as a unit by bash
   • Operator:
     • Control Operator: Determine the flow of execution between multiple commands. Some examples are ;, &, |, || and &&.
     • Redirection Operator: The > and < based operators used to redirect stdin, stdout and stderr.
   • Metacharacter: Any character that causes input to be split into several words. The most common ones are space and newline, with tab also being possible. Characters used in Operators are also metacharacters.

   In the following example, the input is split into words echo, a and b followed by operator > and word c.
   

   $ echo a b>c
   

   Also note that we don't need spaces when using control operators (throwing readability out of the window)
   

   $ echo a&&echo b
   a
   b
   

2. Organize words and operators into a tree of commands. At this point, a distinction is made between simple commands, like [ and compound commands, like [[. Simple commands follow the upcoming rules strictly, while compound commands may deviate. As a user, we can create simple commands at will, either through creating and executing separate bash scripts, or defining and using bash functions. Contrary to this, we cannot create our own compound commands.

   It is here that operators like && from the previous section, but also || are overridden for [[. They will cause a split into multiple separate tests, as shown in the previous section.

3. Perform shell expansions. These are some common expansions in order of execution

   1. Command substitution $() or ` `
   2. Variable expansion such as $VAR

   3. Word splitting. Similar to step one, but only split on whitespace. Implicit null arguments (empty string from expanding unset variables) are removed.

      Using [[, this step is omitted², which explains why [[ -n $PYTHONPATH ]] && echo "Python path" doesn't need any quotes around $PYTHONPATH, whether it contains spaces or it is empty. The final command that is ran is equivalent to [[ -n "/some/path/with spaces" ]] or, when empty, [[ -n "" ]]. Quotes are not added explicitly, so as to not get something like [[ -n """" ]] in case $PYTHONPATH is quoted. This is just the easiest way to show the equivalence.

   4. Filename pattern matching, such as in echo *.sh. This is also not performed for [[, it would simply use the asterisk directly.

4. Setup redirections and remove the corresponding symbols from the command. e.g. echo a>b becomes echo a and the redirection is set up under the hood.

   Note that since at this point it is already decided whether we are in a simple of compound command, the behavior of redirection symbols can be overridden. In [[ it will perform a lexicographical ordering check like so
   

   $ [[ a < b ]] && echo "success!" || echo "No success!"
   success!
   $ [[ b < a ]] && echo "success!" || echo "No success!"
   No success!
   

5. Execute the commands. Actively wait for the return status of any (sub)command that is executed in the foreground.

Given this set of rules, it should now be clear why the following commands are not equivalent, and the first one fails.

$ [[ -n some words ]] && echo "words!"
bash: syntax error in conditional expression
bash: syntax error near `words'
$ WORDS="some words"
$ [[ -n $WORDS ]] && echo "words!"
words!

Conclusion

By now, we have an answer to the question we originally started with

Is it possible to get the behavior from [[ into the Bourne shell?

In the previous section we described the behavior of bash, but the Bourne shell follows the same steps for simple commands. As discussed in step 2 of the previous section , it is only possible to create simple commands. We have seen some examples where [[ overrides the behavior of simple commands (step 2-4), to make our life easier. So no, doing this in the Bourne shell (in the form of a custom script or function) is not possible.

Finally we haven't touched upon the =~ operator supported by [[. This can be used to check if something matches a regular expression and should not be overlooked. As it's not supported by [ and doesn't override any defaults, it didn't really fit in this blog post. It's important enough to at least mention it.

1. [-f test.txt -a -f test2.txt] ↩

2. An exception is made for "$@" ↩