Trusted answers to developer questions
Trusted Answers to Developer Questions

Related Tags

python
python3
multiprocessing
communitycreator

What is the multiprocessing module in Python?

abhilash

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

What is multiprocessing?

Let’s break the word into 2 halves, i.e., multi and processing. The word multi means using more than one. The word processing means getting work done using a processor in the computer. So, the combination means using more than one processor to get the work done.

Multiprocessing has 4 main concepts:

  1. Process class
  2. Pool class
  3. Pipes and Queues
  4. Locks

This shot covers the Process and Pool classes. In Part 2, Pipes and Queues and Locks are covered.

The Process class

In multiprocessing, processes are spawned by creating an object of the Process class. To start the execution of the process, use the .start() method of the Process object.

from multiprocessing import Process
import math
def factorial(num):
print(f"Factorial of {num} is {math.factorial(num)}")
if __name__ == '__main__':
p = Process(target=factorial, args=(5,))
p.start()
p.join()

You can optionally give a name to the process by giving a value to the name parameter during object creation. To access the current running process inside the function, use the current_process method.

from multiprocessing import Process, current_process
import math
def factorial(num):
print(f"Factorial of {num} is {math.factorial(num)}")
curr_process = current_process()
print(f"Process Name is '{curr_process.name}'")
aliveness = "alive" if curr_process.is_alive else "not alive"
print(f"{curr_process.name} is {aliveness}")
if __name__ == '__main__':
p = Process(target=factorial, args=(5,), name="Factorial process")
p.start()
p.join()

The Pool class

The pool class lets you create a pool of processes. The maximum number of processes created in the pool is the number of processors in the system. The number of processors in the system can be obtained using os.cpu_count() or multiprocessing.cpu_count().

There are 2 basic methods that the Pool class provides:

  1. map

The method signature is as follows:

map(self, func, iterable, chunksize=None)

This method chops an iterable into several chunks, which it submits to the process pool as separate tasks. The method calls blocks until the function is completed.

  1. apply

The method signature is as follows:

apply(self, func, args=(), kwds={})

This method applies the given function for a given number of arguments. This method calls blocks until the function is completed.

There are 2 variations of the above methods, i.e., map_async() and apply_async, that return the control immediately rather than waiting to complete the function.

Method Accepts Multiple arguments Blocking Ordered Results
map no yes yes
apply yes yes no
map_async no no yes
apply_async yes no no
from multiprocessing import Pool
import math
def factorial(num):
return math.factorial(num)
def mp_map():
with Pool() as p:
print(p.map(factorial, range(5)))
def mp_map_async():
results = []
def map_callback(result):
results.append(result)
p = Pool()
p.map_async(factorial, range(5), callback=map_callback)
p.close()
p.join()
print(results)
def mp_apply():
with Pool() as p:
print([p.apply(factorial, args=(i,)) for i in range(5)])
def mp_apply_async():
results = []
def apply_callback(result):
results.append(result)
p = Pool()
_ = [p.apply_async(factorial, args=(i,), callback=apply_callback) for i in range(5)]
p.close()
p.join()
print(results)
if __name__ == '__main__':
print("Map Method")
mp_map()
print("-"*10)
print("Async Map Method")
mp_map_async()
print("-"*10)
print("Apply Method")
mp_apply()
print("-"*10)
print("Async Apply Method")
mp_apply_async()

RELATED TAGS

python
python3
multiprocessing
communitycreator

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Keep Exploring