Command Execution Tricks with Subprocess - Designing CI/CD Systems

The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.

Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.

Previous chapters in the series already established the build directives to implement. They covered the format and location of the build specification file. As well as the docker environment in which it runs and its limitations.

Execution using subprocess

Most directives supplied in the YAML spec file are lists of shell commands. So let's look at how Python's subprocess module helps us in this situation.

We need to execute a command, wait for it to complete, check the exit code, and print any output that goes to stdout or stderr. We have a choice between call(), check_call(), check_output(), and run(), all of which are wrappers around a lower-level popen() function that can provide more granular process control.

This run() function is a more recent addition from Python 3.5. It provides the necessary execute, block, and check behavior we're looking for, raising a CalledProcessError exception whenever it finds a failure.

Also of note, the shlex module is a complimentary library that provides some utilities to aid you in making subprocess calls. It provides a split() function that's smart enough to properly format a list given a command-line string. As well as quote() to help escape shell commands and avoid shell injection vulnerabilities.

Security considerations

Thinking about this for a minute, realize that you're writing an execution system that runs command-line instructions as written by a third party. It has significant security implications and is the primary reason why most online build services do not let you get down into this level of detail.

So what can we do to mitigate the risks?

Read On ...