Subprocesses in Python

Any instance of a computer program that is being executed is referred to as a process. Thus, when you open your favorite application, open the command line interface or execute a Python script, you are literally initiating a process.

Processes can themselves initiate other processes. When this happens, the newly started process is referred to as a subprocess or a child process. The original process that initiated the creation of the child process is known as the "parent process." . This relationship leads to  a hierarchical structure where the child process can also start other processes and so on.

In this article, we will explore the process of initiating external programs or applications from within a Python program. 

The subprocess module

The subprocess module in the standard library provides the necessary tools for initiating external processes from within a Python program. It makes it possible to run commands programmatically like you would from the command line interface.

To use the functions defined in the subprocess module, we first need to import the module in our program, as in:

import subprocess

The most basic tool defined in the module is the subprocess.run() function. Before we look deeper on how the function works, let us first look at a simple example:

#import the subprocess module
import subprocess

#check the version of Python
result = subprocess.run(["python", "--version"], capture_output=True, text = True)

#get the output from the subprocess
print(result.stdout)

#get the status code
print("Return code:", result.returncode)

Running the above program is just a programmatic way of running the python --version command from the commandline/shell.

The subprocess.run() function has the following syntax:

subprocess.run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs)

import subprocess

result = subprocess.run(['pip', '--version'], capture_output = True, text = True)

print("stdout: ", result.stdout)
print("returncode: ", result.returncode)
print("stderr: ", result.stderr)

Run shell commands as a string

The shell argument in the subprocess.run() function determines whether the command should be run in a shell. When set to True, the command is run through the system shell (e.g., /bin/sh on Unix-like systems, or cmd.exe on Windows).

The shell argument allows commands to be run  as a string, similar to how you would run them directly on the shell.

import subprocess

# Using shell=True
result_shell = subprocess.run('echo Hello, World!', capture_output = True, shell=True, text=True)

print(result_shell.stdout)

import subprocess

#check where python is installed
python = subprocess.run('where python', capture_output = True, text = True, shell = True ) 
# check where pip is installed 
pip = subprocess.run('where pip', capture_output = True, text = True, shell = True) 

#print the outputs
print('python: ', python.stdout)
print('pip:', pip.stdout)

In the above example, we called the subprocess.run() function with the where command which checks the absolute path to the executable of a program given as the argument.

Raising Errors

By default, errors raised by a subprocess are not propagated to the calling program. However we can set the check parameter to True  in the subprocess.run() function to make it raise a CalledProcessError if the subprocess returns a non-zero exit code, indicating an error.

import subprocess

try:
    #callpython with a badly formated code
    result = subprocess.run(["python", "-c", 'print(Hello, World'], capture_output = True, text = True, check = True)
except subprocess.CalledProcessError as e:
    print(e)

Sending input values when calling a subprocess

Input values can be passed to the called subprocess as additional parameters to the popenargs argument. For example, consider  if we want to call Python as a subprocess with a script which expects arguments. We can pass the arguments, then access them from the called subprocess using the sys.argv list.

#foo.py

import sys 
import subprocess

#get the arguments
print(sys.argv)
input1, input2 = int(sys.argv[1]), int(sys.argv[2])

def add(a, b):
    result = a + b
    return f"{a} + {b} = {result}"

print(add(input1, input2))

#main.py

#import the subprocess module
import subprocess

#call python with foo.py
result = subprocess.run(['python', 'foo.py', '10', '20'], capture_output = True, text = True)

#print the returned value
print(result.stdout)

More advanced examples

Create a virtual environment

#import the subprocess module
import subprocess

#Create a virtual environment from a python program
subprocess.run( ["python", "-m", "venv", "virtualenv"])

If you execute the above snippet,  a new virtual environment with the name virtualenv will be created in the working directory.

Run pip programmatically

We can run pip as a subprocess to manage modules and packages programmatically. This is especially useful for automating installations and other package management tasks.

#import the subprocess module
import subprocess

#install flask with pip
subprocess.run('pip install flask', shell = True)

In the above example, we have run pip as a subprocess to install the flask package programmatically. This is the equivalent of running the pip install flask command from the commandline.

import subprocess

#check the installed packages
subprocess.run("pip list", shell = True)