Generator Functions

Learn about the concept of generator functions and get a hands-on experience of it using an interactive coding playground.

Overview

Generator functions embody the essential features of a generator expression, which is the generalization of a comprehension. The generator function syntax looks even less object-oriented than anything we’ve seen, but we’ll discover that once again, it is a syntax shortcut to create a kind of iterator object. It helps us build processing following the standard iterator-filter-mapping pattern.

Example 1: Sprawling code

Let’s take the log file example a little further. If we want to decompose the log into columns, we’ll have to do a more significant transformation as part of the mapping step. This will involve a regular expression to find the timestamp, the severity word, and the message as a whole. We’ll look at a number of solutions to this problem to show how generators and generator functions can be applied to create the objects we want.

Here’s a version, avoiding generator expressions entirely:

Press + to interact
import csv
import re
from pathlib import Path
from typing import Match, cast
def extract_and_parse_1(full_log_path: Path, warning_log_path: Path)-> None:
with warning_log_path.open("w") as target:
writer = csv.writer(target, delimiter="\t")
pattern = re.compile(r"(\w\w\w \d\d, \d\d\d\d \d\d:\d\d:\d\d) (\w+) (.*)")
with full_log_path.open() as source:
for line in source:
if "WARN" in line:
line_groups = cast(
Match[str], pattern.match(line)).groups()
writer.writerow(line_groups)

We’ve defined a regular expression to match three groups:

  • The complex date string, (\w\w\w \d\d, \d\d\d\d \d\d:\d\d:\d\d), which is a generalization of strings like Apr 05, 2021 20:04:41.

  • The severity level, ...

Get hands-on with 1400+ tech skills courses.