The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.
Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.
Previous chapters in the series already established the build directives to implement. They covered the format and location of the build specification file. As well as the docker environment in which it runs and its limitations.
Most directives supplied in the YAML spec file are lists of shell commands. So let's look at how Python's subprocess module helps us in this situation.
We need to execute a command, wait for it to complete, check the exit code, and print any output that goes to stdout or stderr. We have a choice between
run(), all of which are wrappers around a lower-level
popen() function that can provide more granular process control.
run() function is a more recent addition from Python 3.5. It provides the necessary execute, block, and check behavior we're looking for, raising a
CalledProcessError exception whenever it finds a failure.
Also of note, the shlex module is a complimentary library that provides some utilities to aid you in making subprocess calls. It provides a
split() function that's smart enough to properly format a list given a command-line string. As well as
quote() to help escape shell commands and avoid shell injection vulnerabilities.
Thinking about this for a minute, realize that you're writing an execution system that runs command-line instructions as written by a third party. It has significant security implications and is the primary reason why most online build services do not let you get down into this level of detail.
So what can we do to mitigate the risks?