python-iterators-and-generators

name	python-iterators-and-generators
description	Use for Python iterators, generators, lazy evaluation, and refactoring deeply nested loops by extracting an iterator or a context manager. Covers `yield`, `yield from`, generator expressions, `itertools` discipline, and async iteration.
globs	["**/*.py"]

Python Iterators and Generators

Use this when a function is doing too much in one place because it owns both the iteration and the work, or when memory and throughput pressure points to an eager list that should be lazy.

Working stance

• Generators are the cheapest abstraction for "produce a stream of values"; reach for one before reaching for an iterator class.
• Yield one logical item per yield; do not flatten unrelated layers of structure into a single generator.
• Push the iteration into a named producer; keep consumers small.
• A generator is a context-bound resource: it owns open files, sockets, or transactions while it lives. Pair it with a context manager when the resource must be closed even if the consumer stops early.
• For async streams, use async def generators with async for; do not mix yield and await semantics in an opaque iterator class.

Decision surface

• Generator function (def f(): yield): a producer with internal state — pagination, decompression, tokenization, simulated time.
• Generator expression ((x for x in xs)): a single-step transformation; passable directly to sum, any, min, "".join.
• yield from: a generator delegates an entire sub-stream; removes a manual for x in inner: yield x loop and forwards send/throw/close correctly.
• Iterator class (__iter__ + __next__): the producer must expose more than iteration — peek(), tell(), restart semantics, or multiple independent passes.
• itertools building blocks: chain, islice, groupby, accumulate, tee (carefully), product, pairwise, batched (3.12+). Compose with generators instead of writing custom loops.
• Async generator: the stream is awaited (HTTP page chunks, queue consumption, server-sent events).

Refactoring patterns

• Extract an iterator: a function with three nested for loops and bookkeeping is hiding a producer. Move the loops into a generator named after the item it yields (def changed_records(...): yield ...) and let the caller iterate.
• Extract a context manager: a generator that owns a connection and a cursor should not also be the place where the caller commits or rolls back. Wrap the resource in a context manager and let the generator be a pure producer.
• Replace materialization with iteration: an intermediate list(...) between two transformations is usually wrong; the pipeline can be lazy from end to end unless one stage needs random access.

Red flags

• a generator function whose body is mostly bookkeeping and a single yield value at the end (call it a function and return the value),
• tee used to feed two independent consumers (one rewinds the source; consider a real reader),
• list(...) in the middle of a pipeline that is otherwise lazy,
• a generator that owns a file but is consumed via next(...) calls spread across several functions (move ownership to a context manager),
• yield from flattening unrelated structure into a single stream,
• an iterator class with no extra surface beyond __iter__ and __next__ (collapse to a generator),
• an async generator that performs blocking I/O between await points.

Read extracting-iterators.md and lazy-pipelines.md for the refactoring patterns and the everyday composition idioms.