Search⌘ K
AI Features
Home
Courses
Python 201 - Interactively Learn Advanced Concepts in Python 3

Creating a Pool

Explore how to create a pool of worker threads or processes using the concurrent.futures module. Understand using ThreadPoolExecutor and ProcessPoolExecutor with Python's with statement, manage asynchronous tasks with futures and as_completed, and optimize pooling with the map method and timeout settings. This lesson helps improve your ability to handle concurrent execution and avoid deadlocks in Python.

We'll cover the following...

Creating a pool of workers is extremely easy when you’re using the concurrent.futures module. Let’s start out with the code of asyncio and now use the concurrent.futures module in it. Here’s our version:

Python 3.5
import os
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
def downloader(url):
    """
    Downloads the specified URL and saves it to disk
    """
    req = urllib.request.urlopen(url)
    filename = os.path.basename(url)
    ext = os.path.splitext(url)[1]
    if not ext:
        raise RuntimeError('URL does not contain an extension')
    with open(filename, 'wb') as file_handle:
        while True:
            chunk = req.read(1024)
            if not chunk:
                break
            file_handle.write(chunk)
    msg = 'Finished downloading {filename}'.format(filename=filename)
    return msg
def main(urls):
    """
    Create a thread pool and download specified urls
    """
    with ThreadPoolExecutor(max_workers=5) as executor:
        futures = [executor.submit(downloader, url) for url in urls]
        for future in as_completed(futures):
            print(future.result())
if __name__ == '__main__':
    urls = ["http://www.irs.gov/pub/irs-pdf/f1040.pdf",
            "http://www.irs.gov/pub/irs-pdf/f1040a.pdf",
            "http://www.irs.gov/pub/irs-pdf/f1040ez.pdf",
            "http://www.irs.gov/pub/irs-pdf/f1040es.pdf",
            "http://www.irs.gov/pub/irs-pdf/f1040sb.pdf"]
    main(urls)

First off we do the imports that we need. Then we create our downloader function. We went ahead and updated it slightly so it checks to see if the URL has an extension on the end of it. If ...