Introduction
The multiprocessing module in Python allows you to create and manage processes, enabling you to take full advantage of multiple processors on a machine. It helps you achieve parallel execution by using separate memory spaces for each process, unlike threading where threads share the same memory space. Here's a list of commonly used classes and methods in the multiprocessing module with brief examples.
1. Process
The Process class is the core of the multiprocessing module, allowing you to create and run new processes.
from multiprocessing import Process
def print_numbers():
for i in range(5):
print(i)
p = Process(target=print_numbers)
p.start() # Starts a new process
p.join() # Waits for the process to finish
2. start()
Starts the process’s activity.
p = Process(target=print_numbers)
p.start() # Runs the target function in a separate process
3. join([timeout])
Blocks the calling process until the process whose join() method is called terminates. Optionally, you can specify a timeout.
p = Process(target=print_numbers)
p.start()
p.join(2) # Waits up to 2 seconds for the process to finish
4. is_alive()
Returns True if the process is still running.
p = Process(target=print_numbers)
p.start()
print(p.is_alive()) # True if the process is still running
5. current_process()
Returns the current Process object representing the calling process.
from multiprocessing import current_process
def print_current_process():
print(current_process())
p = Process(target=print_current_process)
p.start() # Prints the current process info
6. active_children()
Returns a list of all Process objects currently alive.
p1 = Process(target=print_numbers)
p2 = Process(target=print_numbers)
p1.start()
p2.start()
print(Process.active_children()) # Lists all active child processes
7. cpu_count()
Returns the number of CPUs available on the machine.
from multiprocessing import cpu_count
print(cpu_count()) # Returns the number of CPUs on the machine
8. Pool
A Pool object provides a convenient way to parallelize execution of a function across multiple input values. It manages a pool of worker processes.
from multiprocessing import Pool
def square(n):
return n * n
with Pool(4) as pool: # Pool with 4 worker processes
result = pool.map(square, [1, 2, 3, 4, 5])
print(result) # [1, 4, 9, 16, 25]
9. Queue
A Queue is a shared data structure that allows multiple processes to communicate by passing data between them.
from multiprocessing import Process, Queue
def put_data(q):
q.put([1, 2, 3])
def get_data(q):
data = q.get()
print(data)
q = Queue()
p1 = Process(target=put_data, args=(q,))
p2 = Process(target=get_data, args=(q,))
p1.start()
p2.start()
p1.join()
p2.join()
10. Lock
A Lock ensures that only one process can access a shared resource at a time.
from multiprocessing import Process, Lock
lock = Lock()
def print_numbers():
with lock:
for i in range(5):
print(i)
p1 = Process(target=print_numbers)
p2 = Process(target=print_numbers)
p1.start()
p2.start()
p1.join()
p2.join()
11. Value and Array
The Value and Array objects allow sharing simple data types and arrays between processes.
from multiprocessing import Process, Value
def increment(val):
with val.get_lock():
val.value += 1
shared_val = Value('i', 0)
processes = [Process(target=increment, args=(shared_val,)) for _ in range(10)]
for p in processes:
p.start()
for p in processes:
p.join()
print(shared_val.value) # Output will be 10
12. Pipe
A Pipe provides a two-way communication channel between two processes.
from multiprocessing import Process, Pipe
def send_message(conn):
conn.send("Hello from child")
conn.close()
parent_conn, child_conn = Pipe()
p = Process(target=send_message, args=(child_conn,))
p.start()
print(parent_conn.recv()) # Receives data from the child process
p.join()
13. Manager
A Manager allows you to create shared objects, such as lists and dictionaries, that multiple processes can modify concurrently.
from multiprocessing import Process, Manager
def modify_list(shared_list):
shared_list.append("New item")
with Manager() as manager:
shared_list = manager.list([1, 2, 3])
p = Process(target=modify_list, args=(shared_list,))
p.start()
p.join()
print(shared_list) # [1, 2, 3, "New item"]
14. Semaphore
A Semaphore allows you to control access to a resource, permitting only a certain number of processes to access it at a time.
from multiprocessing import Process, Semaphore
import time
sem = Semaphore(2) # Only 2 processes can access the resource
def limited_access():
with sem:
print("Accessing resource")
time.sleep(2)
processes = [Process(target=limited_access) for _ in range(5)]
for p in processes:
p.start()
for p in processes:
p.join()
Conclusion
The multiprocessing module in Python is designed to take full advantage of multiple processors on a machine. From creating and managing processes using Process, to controlling shared resources with Lock and Semaphore, and facilitating communication through Queue and Pipe, the multiprocessing module is crucial for parallelizing tasks in Python applications.
Top comments (0)