What are coroutines in Python?

Key takeaways:

• The ability of coroutines to pause and restart operations makes it possible to manage concurrent tasks, especially I/O-bound operations, efficiently.

• Python uses an event loop to manage coroutines, enabling them to run concurrently without blocking the main thread.

• Coroutines are part of Python’s asyncio library and use the async def syntax.

• Coroutines facilitate asynchronous file operations, allowing multiple files to be read and written concurrently, enhancing performance without blocking.

Coroutines in Python are functions that can pause and resume their execution, enabling more efficient management of concurrent tasks. This capability is especially useful for handling I/O-bound operations and tasks that involve waiting, such as network requests or file I/O, without blocking the main thread of execution. By leveraging coroutines, Python programs can achieve better performance and responsiveness in scenarios where multiple operations need to run concurrently.

To efficiently manage these coroutines, Python uses an event loop. The event loop is the core of asynchronous programming in Python, coordinating the execution of multiple coroutines and tasks. It keeps track of all the running tasks and determines the right time to resume paused coroutines. When a coroutine is paused, the event loop can switch to executing other tasks, ensuring that the program remains responsive and efficient. Once the conditions to resume are met, the coroutine can continue from where it left off, allowing for seamless and cooperative multitasking.

This makes coroutines a powerful tool for asynchronous programming, enabling efficient handling of I/O-bound and high-level structured network code.

Declaring coroutines

Coroutines are part of Python’s asyncio library, which provides a framework for writing asynchronous programs. To declare a coroutine, we use the async def syntax. Here’s a simple example of a coroutine:

Code explanation

• Inside my_coroutine, the await asyncio.sleep(duration) call simulates a blocking operation by pausing the coroutine for duration seconds. The event loop can run other tasks during this pause. The main coroutine awaits my_coroutine, ensuring it completes before exiting. asyncio.run(main()) starts the event loop, which manages the execution of coroutines.

I/O-bound coroutine example

Let’s understand coroutines in more depth using a slightly complex code example:

Code explanation

• Lines 4–9: The read_file coroutine asynchronously reads content from a given file path. It uses aiofiles.open with the mode set to 'r' for reading.

• Lines 11–15: The write_to_file coroutine asynchronously writes given content to a file, appending it. It uses aiofiles.open with the mode set to 'a' for appending.

• Lines 17–24: process_files is the orchestrating coroutine. It first creates a list of tasks to read each file in read_paths concurrently. Then, it awaits the completion of these tasks and subsequently writes their contents to a single output file, appending each content with a new line.

• Lines 26–27: We define the read and write paths for the I/O operation.

• Line 30: We make the asyncio.run(process_files(read_paths, write_path)) call that starts the event loop and executes the file-processing tasks.

Understanding the operational flow

Next, let’s understand the operational flow of the code and how it works as a whole.

The execution begins with the process_files coroutine, which acts as the orchestrator for the entire process. It takes a list of file paths to read from (read_paths) and a single file path to write to (write_path). Inside the process_files coroutine, it first sets up multiple read_file coroutine tasks, one for each file in read_paths. These coroutines are designed to be run concurrently. This is where the power of asyncio.gather comes to display, allowing multiple asynchronous operations (file reads in this case) to be initiated and run in parallel. This parallelism is not about doing things simultaneously in different threads or processes. Rather, it’s about efficiently managing I/O waiting times, where the program doesn’t block while waiting for one file’s content to be read before starting to read the next.

Once all the read_file tasks are dispatched, asyncio.gather awaits their completion. This step effectively collects all the read contents from the input files once they are all available, aggregating them into a list (contents). This waiting is non-blocking, meaning the event loop can switch contexts and perform other tasks, if any, while waiting for file reads to complete.

With all file contents read and stored in contents, the code then iterates over this collection, appending each piece of content to the output file specified by write_path. The write_to_file coroutine is called for each content piece. Although the writing process is asynchronous, each write operation is awaited before the next begins, ensuring that file contents are written in the same order as the input files were read. This sequential writing is crucial for preserving the content order from multiple sources when combining them into a single output file.

Both the read_file and write_to_file coroutines utilize aiofiles for opening and operating on files. aiofiles provides an asynchronous interface to file I/O, allowing these operations to be non-blocking. When a file is being read or written, the operation is awaited, freeing up the event loop to handle other tasks or manage other asynchronous operations in the meantime. This is particularly advantageous for I/O-bound tasks, where waiting on file I/O operations can bottleneck the program’s speed.

The asyncio event loop drives the entire asynchronous operation with the asyncio.run(process_files(read_paths, write_path)) call. This not only kicks off the process but also manages all the underlying asynchronous tasks, including switching between them as they await I/O operations, ensuring efficient utilization of time that would otherwise be spent waiting.