ShellPipe.py | A Hacky Remedy to Overkill Shell Scripting

In my previous post, I had a bit of a rant about people not learning the idiosyncracies of the language that is bash, and more generally those of shell languages as a whole, leading to a lot of frankly horrible scripting out in the wild.

I've written so much shell script - and put so much emphasis on clean, reusable code in shell - for the sake of a handful of key operations that just must be commands, where I would have rather been managing much nicer code.

So today I turn that right around: rather than try to apply clean code to shell scripts (and crash against the rocky shores of other devs' "but it's just a shell script"), I'm going to bring the best part of shells to Python: ShellPipe.py

The fact of the matter is, a lot of shell scripting is used to glue other tools together, and that's certainly where it excels. Python by contrast, like most other languages, requires some minor passing and tracking of outputs and inputs to achieve the same effects and, whilst generally more comfortable languages, aren't quite as eloquent to the task of unifying disparate, uninterfaceable tools.

For this reason, I have continued to write bash scripts as glue, rather than try to do that passing around. For that reason in turn, I have written extensive amounts of supporting bash that really should have either been written in another language, or dispensed with entirely were it not for the idiosyncracies.

On the last post, I got a comment from @xtofl indicating that they'd had a quick go at re-purposing the bitwise OR operator in Python into a pipe-like operator. They expanded on that technique in a later post with their proposition for chaining functions, pipe-style, which whilst intersting, does not meet my more basic sysadminy needs.

I remembered their little comment yesterday and decided to have a go at it myself.

I'm quite proud of myself. Though maybe I should feel gravely ashamed. I can now do this in a python script:

from shellpipe import sh

# Run a command
sh() | 'git clone https://github.com/taikedz/shellpipe'

# Chain commands, use strings or lists, embed quoted strings, print to console:
sh() | "find shellpipe/shellpipe" | 'grep -vE ".*\\.pyc"' | ['du', '-sh'] > 1

Predefine commands, use them in chains

DOCKER_PSA = "docker ps -a"
GREP = "grep {}"

def look_for(subname):
    found = sh() | DOCKER_PSA | GREP.format(subname)
    return str(found).split("\n")

I would have ideally wanted to do something like this:

mysql_result = sh(f'mysql {user} -p{pass} db') < """
CREATE TABLE ...
"""

which unfortunately is not possible whilst also ensuring each step is run immediately - the comparator needs to evaluate the left hand statement (LHS) entirely first, before the right hand (RHS) is checked. My current implementation runs the pipe step on-creation, which means the command itself is run before the "redirect" can be processed.

If I defer the execution until after the redirection is done (this was actually how the first implementation worked), I would have to do something like this:

mysql_result = (sh(f'mysql {user} -p{pass} db') < """
CREATE TABLE ...
""").run()

Which is much less elegant. Also, having the external script in an actual file is better practice in most setups so what I actually need to do with the current implmentation is

with open("script.mysql", "r") as fh:
  mysql_result = sh(f'mysql {user} -p{pass} db', stdin=fh)

which is generally more reasonable, anyway. Don't hard-code other scripts in your program, store them neatly (he said, stuffing shell commands into a Python program).

What is this sorcery??

I have hijacked bitwise OR-ing. Or at least, I have for the purpose of my custom class, ShellPipe (which is simply provided through sleight of assignment as sh = ShellPipe).

What ShellPipe does is define its own __or__() function, which is called any time it is placed in a x | y operation in Python. Similar things exist for __and__ (the & bitwise AND operator implementor) and __lt__ (the less-than operator implementor) so as to be able to use custom, complex classes as sortable items.

this.__or__(that) normally should simply return an object of the same type as this and that , but we can abuse this a little by not requiring the one side to be of the same type as the other. Conceivably, we could return whatever we want.

When invoking x | y, only the __or__() of the object on the left hand side of the statement gets executed, and that pair then returns usually a new object that is the union of the two.

By invoking ShellPipe() | "a string" , I capitalize on this by allowing ShellPipe's function to see that on the other side of the operation there is a string, and so it wraps that in a ShellPipe(...) of its own - and the result is that the string has become a runnable piece of code, in a way.

It looks like this (rather, it is exactly this):

    def __or__(self, other):
        our_out = None
        if self.process:
            our_out = self.process.stdout

        if type(other) in (str,list,tuple):
            other = ShellPipe(command_list=other, stdin=our_out)

        return other

So what is happening when I invoke ShellPipe() | "cmd1" | "cmd2" ?

• In this case, the first LHS (an empty instance) doesn't do anything, as it was not built with a command (it could have been, twelve and two sixes as we say here)
• and it turns the RHS into a ShellPipe("cmd1") and returns it - cmd1 immediately executes as a result of being defined
• cmd1 is now the new LHS, and it keeps a hold of its output stream, passing it into the construction of the now-new RHS, ShellPipe("cmd2", stdin=cmd1_stdout)

And so on and so forth. Quite simple, really. Once the end of the chain is reached, the last item that was executed is also returned and so in

mypipe = sh() | "cmd1" | "cmd2" | "cmd3"

mypipe is in fact the ShellPipe("cmd3") object created by cmd2

It is the output of this last command that we can then inspect with mypipe.get_stdout()

But why??

Is this useful and better than using subprocess.Popen() directly? It is certainly mostly syntactic sugar, and importing features from one language into another is not always the best answer, but my use cases have veered more towards "I want to use Python for most things, but there's that ONE tool that can only be used as a command." String and stream manipulation is easier in Python (once you need to manage context beyond a single line), and the rich typing experience - which allowed the __or__() overloading in the first place - is better there than in shell scripts.

The downside of my implementation is that it runs each command entirely before passing on to the next one - if a command should produce a large amount of output, that would be stored to file descriptor (and likely thus in RAM) before being passed to the next command. Also, if several commands take a significant amount of time to run, this is not going to work well either.

But there are just those times, where that one tool that is available as a command only, and nobody has python-packaged for, is easier to just... use as a command.

My next programming session is likely to be converting all the scripts I once wrote heaps of bash and supporting bash for into Python scripts, using shellpipe to cover the command corner cases. I'll probably post a report here after I've done a couple to remark on the difference it has made - if any.

But if I consider . all the . bash code . I've written . where most of it . was just managing variables . for the sake of . a handful of . piped shell commands and clean code ...

... I feel vindicated. This is a good abomination 😌