This chapter will discuss how you can execute external commands from Python, capture their output and other relevant details such as the exit status. The availability of commands depends on the OS you are using (mine is Linux).
Using os module
Last chapter showed a few examples with os
module for file processing. The os
module is a feature rich module with lot of other uses, like providing an interface for working with external commands. Here's an example:
>>> import os
>>> os.system('echo hello "$USER"')
hello learnbyexample
0
Similar to the print()
function, the output of the external command, if any, is displayed on the screen. The return value is the exit status of the command, which gets displayed by default on the REPL. 0
means the command executed successfully, any other value indicates some kind of failure. As per docs.python: os.system:
On Unix, the return value is the exit status of the process encoded in the format specified for
wait()
.
Here's an example for non-zero exit status:
>>> status = os.system('ls xyz.txt')
ls: cannot access 'xyz.txt': No such file or directory
>>> status
512
# to get the actual exit value
>>> os.waitstatus_to_exitcode(status)
2
# if you don't want to see the error message,
# you can redirect the stderr stream
>>> os.system('ls xyz.txt 2> /dev/null')
512
You can use the os.popen()
method to save the results of an external command. It provides a file object like interface for both read (default) and write. To check the status, call close()
method on the filehandle (None
means success).
>>> fh = os.popen('wc -w <ip.txt')
>>> op = fh.read()
>>> op
'9\n'
>>> status = fh.close()
>>> print(status)
None
# if you just want the output
>>> os.popen('wc -w <ip.txt').read()
'9\n'
subprocess.run
The subprocess
module provides a more flexible and secure option to execute external commands, at the cost of being more verbose.
Quoting relevant parts from doc.python: subprocess module:
The
subprocess
module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.The recommended approach to invoking
subprocesses
is to use therun()
function for all use cases it can handle. For more advanced use cases, the underlyingPopen
interface can be used directly.
>>> import subprocess
>>> subprocess.run('pwd')
'/home/learnbyexample/Python/programs/'
CompletedProcess(args='pwd', returncode=0)
>>> process = subprocess.run(('ls', 'xyz.txt'))
ls: cannot access 'xyz.txt': No such file or directory
>>> process.returncode
2
The first argument to run()
method is the command to be executed. This can be either be a single string or a sequence of strings (if you need to pass arguments to the command being executed). By default, command output is displayed on the screen. Return value is a CompletedProcess
object, which has relevant information for the command that was executed such as the exit status.
As an exercise, read subprocess.run documentation and modify the above ls
example to:
- redirect the
stderr
stream to/dev/null
- automatically raise an exception when the exit status is non-zero
See also:
- stackoverflow: How to execute a program or call a system command from Python?
- stackoverflow: difference between subprocess and os.system
- stackoverflow: How to use subprocess command with pipes
- stackoverflow: subprocess FAQ
shell=True
You can also construct a single string command, similar to os.system()
, if you set shell
keyword argument to True
. While this is convenient, use it only if you have total control over the command being executed such as your personal scripts. Otherwise, it can lead to security issues, see stackoverflow: why not use shell=True for details.
Quoting from docs.python: subprocess Frequently Used Arguments:
If
shell
isTrue
, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of~
to a user's home directory
>>> p = subprocess.run(('echo', '$HOME'))
$HOME
>>> p = subprocess.run('echo $HOME', shell=True)
/home/learnbyexample
>>> p = subprocess.run(('ls', '*.txt'))
ls: cannot access '*.txt': No such file or directory
>>> p = subprocess.run('ls *.txt', shell=True)
ip.txt
>>> p = subprocess.run('seq -s, 10 > out.txt', shell=True)
>>> p = subprocess.run('cat out.txt', shell=True)
1,2,3,4,5,6,7,8,9,10
If shell=True
cannot be used but shell features as mentioned above is needed, you can use modules like os
, glob
, shutil
and so on as applicable. See also docs.python: Replacing Older Functions with the subprocess Module.
>>> p = subprocess.run(('echo', os.getenv('HOME')))
/home/learnbyexample
Changing shell
By default, /bin/sh
is the shell used for POSIX systems. You can change that by setting the executable
argument to the shell of your choice.
>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected
>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True,
executable='/bin/bash')
3a4
> 4
Capture output
If you use capture_output=True
, the CompletedProcess
object will provide stdout
and stderr
results as well. These are provided as bytes
data type by default. You can change that by setting text=True
.
>>> p = subprocess.run(('date', '-u', '+%A'), capture_output=True, text=True)
>>> p
CompletedProcess(args=('date', '-u', '+%A'), returncode=0,
stdout='Monday\n', stderr='')
>>> p.stdout
'Monday\n'
You can also use subprocess.check_output()
method to directly get the output.
>>> subprocess.check_output(('date', '-u', '+%A'), text=True)
'Monday\n'
You can also use legacy methods
subprocess.getstatusoutput()
andsubprocess.getoutput()
but they lack in features and do not provide secure options. See docs.python: subprocess Legacy Shell Invocation Functions for details.
Top comments (0)