Search⌘ K
Home
Courses
Building Robust Object-Oriented Python Applications and Libraries

Generator Stacks

Get to know the concept of generator stacks and learn to implement them.

We'll cover the following...

Why do we need to create generator stacks?

The generator function (and the generator expression) for warnings_filter makes an unpleasant assumption. The use of cast() makes a claim to mypy that’s—perhaps—a bad claim to make. Here’s an example:

Python 3.10.4
warnings_filter = (
  tuple(cast(Match[str], pattern.match(line)).groups())
  for line in source
  if "WARN" in line
)

The use of cast() is a way of claiming the pattern.match() will always yield a Match[str] object. This isn’t a great assumption to make. Someone may change the format of the log file to include a multiple-line message, and our WARNING filter would crash every time we encountered a multi-line message.

Here’s a message that would cause problems followed by a message that’s easy to process:

Python 3.10.4
Jan 26, 2015 11:26:01 INFO This is a multi-line information
message, with misleading content including WARNING
and it spans lines of the log file WARNING used in a confusing way
Jan 26, 2015 11:26:13 DEBUG Debug messages are only useful if you want
to figure something out.

The first line has the word WARN in a multi-line message that will break our assumption about lines that contain the word WARN. We need to handle this with a little more care.

We can rewrite this generator expression to create a generator function, and add an assignment statement (to save the Match object) and an if statement to further decompose the filtering. We can use the walrus operator := to save the Match object, also.

We could reframe the generator expression as the following generator function:

Python 3.10.4
def warnings_filter(source: Iterable[str]) -> Iterator[Sequence[str]]:
  pattern = re.compile
    (r"(\w\w\w \d\d, \d\d\d\d \d\d:\d\d:\d\d) (\w+) (.*)")
  for line in source:
    if match := pattern.match(line):
      if "WARN" in match.group(2):
        yield match.groups()

Generating stacks

As we noted above, this complex filtering tends toward deeply nested if statements, which can create logic that’s difficult to summarize. In ...