Search⌘ K
AI Features
Home
Courses
Building Robust Object-Oriented Python Applications and Libraries

Generator Expressions

Explore how generator expressions in Python enable efficient iteration by producing items lazily on demand. Understand their syntax and use cases for processing large data sequences like log files without loading everything into memory.

We'll cover the following...

Overview

Sometimes we want to process a new sequence without pulling a new list, set, or dictionary into system memory. If we’re iterating over items one at a time, and don’t actually care about having a complete container (such as a list or dictionary) created, a container is a waste of memory. When processing one item at a time, we only need the current object available in memory at any one moment. But when we create a container, all the objects have to be stored in that container before we start processing them.

Example

For example, consider a program that processes log files. A very simple log might contain information in this format:

Python 3.10.4
Apr 05, 2021 20:03:29 DEBUG This is a debugging message.
Apr 05, 2021 20:03:41 INFO This is an information method.
Apr 05, 2021 20:03:53 WARNING This is a warning. It could be serious.
Apr 05, 2021 20:03:59 WARNING Another warning sent.
Apr 05, 2021 20:04:05 INFO Here's some information.
Apr 05, 2021 20:04:17 DEBUG Debug messages are only useful if you want
to figure something out.
Apr 05, 2021 20:04:29 INFO Information is usually harmless, but
helpful.
Apr 05, 2021 20:04:35 WARNING Warnings should be heeded.
Apr 05, 2021 20:04:41 WARNING Watch for warnings.

Log files for popular web servers, databases, or email servers can contain many gigabytes of data (one ...