A Generator Can Only Be Walked Once

Loop over it twice, and the second loop prints nothing.

An iterator has a position, and once the position reaches the end it stays there. A generator is an iterator, so a generator walked once is empty the second time, and Python raises no error. This card shows a second pass that silently finds nothing, then fixes it by making the object an iterable that creates a fresh iterator each time.

A report sums some prices and counts them, and the prices come from a generator.

prices = read_prices()
total = sum(prices)
count = len(list(prices))

read_prices yields 500, 1200 and 300.

What are the total and the count? Decide, then look.

from collections.abc import Iterator


def read_prices() -> Iterator[int]:
    for price in (500, 1200, 300):
        yield price


prices = read_prices()
total = sum(prices)
count = len(list(prices))
print("total:", total)
print("count:", count)

OutputRan with python:3.12-slim (Python 3.12.13)

total: 2000
count: 0

`sum` walked the generator to the end, so `list` found nothing.

The total is 2000, and the count is 0. sum walked the generator to its end, and list was given the same generator at the same position, so it received no elements.

The cost is a wrong figure with no error. An average computed from these two values would divide by zero, and a check that compares the count with an expected number would report missing data that is not missing.

The fix is to tell an iterable from an iterator. An iterable is an object with __iter__ that returns a fresh iterator each time it is walked. Making __iter__ a generator method gives one without writing an iterator class.

from collections.abc import Iterator


class Prices:
    def __iter__(self) -> Iterator[int]:
        for price in (500, 1200, 300):
            yield price


prices = Prices()
total = sum(prices)
count = len(list(prices))
print("total:", total)
print("count:", count)

OutputRan with python:3.12-slim (Python 3.12.13)

total: 2000
count: 3

Each walk calls `__iter__`, which starts a new generator.

When a generator must stay a generator, turn it into a list once, with prices = list(read_prices()), and walk the list as often as needed. That holds every element in memory, which is the cost the generator was avoiding.

← → move through sections and cards