Problems with Multiprocessing
Explore the challenges of multiprocessing in Python, focusing on the cost of interprocess communication and serialization. Understand how shared memory can optimize performance, and learn the advantages of process independence. Compare threads and processes to choose the best concurrency solution for your applications.
Cost of interprocess communication
As with threads, multiprocessing also has problems, some of which we have already discussed. Sharing data between processes is costly. As we have discussed, all communication between processes, whether by queues, OS pipes, or even shared memory, requires serializing the objects. Excessive serialization can dominate processing time. Shared memory objects can help by limiting the serialization to the initial setup of the shared memory. Multiprocessing works best when relatively small objects are passed between processes and a tremendous ...