Iterators & Generators

Every for loop you have ever written rests on one small contract. Unpacking, list(), sum(), the in operator, *args splatting and every comprehension rest on it too. That contract is the iteration protocol: ask an object for an iterator, then call next() on it until it says it is done. Once you see the protocol, a lot of Python stops looking like magic. It also explains why some loops silently produce nothing the second time around.

It matters in production because iteration is where memory and time are won or lost. A list comprehension over ten million items builds all ten million values before you touch the first. A generator hands them out one at a time and holds almost nothing. The same machinery lets you stream a multi-gigabyte log, stop at the first match, or describe an endless sequence safely. It also hides some of the nastiest quiet bugs in the language, such as a generator that is exhausted and returns zero instead of raising an error.

By the end of this chapter you will be able to tell an iterable from an iterator, and write both by hand. You will turn a function into a generator with yield, read a paused generator's frame, and steer it with send, throw and close. You will build constant-memory pipelines out of generator expressions and itertools, and delegate work with yield from. You will also know the trade-offs well enough to pick between a list, a generator expression, a generator function and an iterator class, and to spot the classic traps before they ship.

Python's for loop never talks to your data directly. It talks to a second object that walks over the data, and that is where two words that are easy to blur come from. An iterable is any object that can hand you such a walker. Technically it is an object that implements __iter__(). An iterator is the walker itself, the object that actually produces values one at a time. It implements both __iter__() and __next__().

The two roles overlap in one direction only. Every iterator is also an iterable, because its __iter__() simply returns self. The reverse is false. Most iterables, including lists, tuples, dicts and strings, are not iterators, because they have no __next__() and no memory of a position.

The built-in iter() turns an iterable into an iterator, and next() pulls one value out of an iterator. Handing a plain list to next() fails, because the list is the container and not the cursor. The example below shows that failure, then does it properly with iter().

The split between the two roles explains a behaviour that surprises newcomers. An iterable can be looped over any number of times, because each for loop asks it for a fresh iterator. An iterator is single-use. Each next() moves its position forward and nothing ever moves it back, so once it reaches the end it stays exhausted for good.

Because the position lives in the iterator and not in the list, you can also have several independent cursors over the same data. The example below loops a list twice, drains one iterator, and then makes two iterators that do not disturb each other.

Read the last line closely. m1 is m2 is False because each call to iter(nums) built a separate cursor. iter(m1) is m1 is True, which is the rule that an iterator is its own iterator. iter(nums) is nums is False, because a list is not its own iterator. The second list(it) returned [] without any error. An exhausted iterator is silent, which is why this bug is easy to miss.

You already use many iterables: list, tuple, dict, set, str, range, file objects and collections.deque. All of them can be used in a for loop, but only some of them are iterators. The next example asks Python directly, using an in-memory StringIO as a stand-in for a file.

File objects are the odd one out. A file keeps a read position, so it is its own iterator, and looping over it a second time gives nothing unless you seek(0) or reopen it. Every other container in the list hands out a new iterator per loop. What each one yields differs, as the table shows.

The example below compares range with a generator expression, which is a true iterator. Indexing and len() work on the first and fail on the second. Asking iter() for a range gives a separate range_iterator object, so the range itself is never used up.

To check at runtime which role an object plays, use the abstract base classes in collections.abc. isinstance(x, Iterable) asks whether iter(x) can work, and isinstance(x, Iterator) asks whether x can also be advanced with next(). These checks look for the methods themselves, so a class of your own passes without inheriting from anything.

Shelf defines only __iter__(), so it counts as an iterable and not as an iterator. Each iter(s) call returns a fresh list iterator, which is exactly the behaviour a re-readable container should have. The generator g is an iterator without our having written any __next__() ourselves.

Python's iteration model is a handshake between two built-in functions and two dunder methods. Calling iter(obj) asks the object for a fresh iterator, and calling next(it) asks that iterator for its next value. Both built-ins are thin wrappers: they find the matching method and call it for you.

Notice the word type in both the first and second rows. Python looks the method up on the class, never on the instance. This is the same rule that applies to every special method, and it means __getattr__ and instance attributes cannot supply __iter__. In the example below, the class has a __getattr__ that would happily answer any attribute request, and we even assign __iter__ onto the instance. iter() ignores both.

class Ghost:
    def __getattr__(self, name):
        print('getattr', name)
        return lambda: iter([1, 2])

g = Ghost()
g.__iter__ = lambda: iter([1, 2])
try:
    iter(g)
except TypeError as e:
    print(e)

The lookup goes to Ghost, not to g, so neither the instance attribute nor __getattr__ is consulted.

Now the two calls working together on a two-character string. Each next advances the same iterator by one position. Once the characters run out, next raises StopIteration. Passing a second argument to next turns that exception into an ordinary return value, which is handy when an iterator might legitimately be empty.

The iterator protocol is not the only way in. If an object has no __iter__ but does define __getitem__, iter() quietly builds a sequence iterator for it. That iterator calls __getitem__ with the integers 0, 1, 2 and so on, and treats an IndexError as the end of the data. This is the older protocol, kept so that classic sequence-like classes still work in loops.

With both halves of the protocol in hand, a for loop is easy to desugar. Python calls iter() once, before the loop starts. It then calls next() over and over, runs the body after each value, and leaves the loop the moment StopIteration appears. After the body finishes, control goes back to the next() call.

Written out by hand, the same loop is a while True with a try/except. The break on StopIteration is the entire exit mechanism.

The two-argument form iter(callable, sentinel) is a different way to make an iterator. It has no container at all: it just calls callable() with no arguments on every step and stops as soon as a result equals the sentinel. It is the classic way to read a file in fixed-size blocks, because read() returns an empty bytes object at end of file.

StopIteration is a control-flow signal, not an error. It is how an iterator says "I have nothing more to give", and the machinery that drives iteration expects it and swallows it. That is why a normal for loop ends silently instead of crashing. It only looks like an error when you call next() yourself and forget to handle it.

There is one strict rule attached to it: an exhausted iterator must stay exhausted. Once __next__ has raised StopIteration, every later call must raise it again, forever. Consumers rely on this, because a loop that has ended, or a second consumer that arrives late, must never see fresh values appear.

Loops are only one of many consumers. Anything that takes an iterable calls iter() and then next() the same way. The tracer class below prints every step so that you can watch unpacking and the in operator drive it. Unpacking two names asks for a third value on purpose, to make sure nothing is left over. The in test stops pulling the moment it finds a match.

The protocol was introduced by PEP 234 in Python 2.2. Before it, looping meant indexing with __getitem__, which is why that fallback survives today. PEP 234 gave every object one uniform way to say "here is how I am walked", and the same two methods underpin every loop construct in the language.

A generator is the easy way to build an iterator, but the protocol underneath is only two methods. If you write them yourself you see exactly what the interpreter relies on. An iterator class needs __iter__, which returns the object itself, and __next__, which hands out the next value or signals the end.

The end is signalled by raising StopIteration. A for loop catches that exception and quietly leaves the loop, so it acts as control flow and not as a failure. Everything the iterator remembers between calls lives in instance attributes, so you move the cursor forward by hand.

Here is the smallest useful version, a counter that produces 0 up to n - 1. The guard comes first, then the cursor moves, then the old value is returned.

class Count:
    def __init__(s, n):
        s.i, s.n = 0, n

    def __iter__(s):
        return s

    def __next__(s):
        if s.i >= s.n:
            raise StopIteration
        s.i += 1
        return s.i - 1

c = Count(3)
print(next(c), next(c), next(c))
try:
    next(c)
except StopIteration:
    print('done')
print(list(Count(4)))

The cursor is s.i; nothing else tracks position.

Everything in this class is bookkeeping: the two attributes, the guard, the increment and the subtraction to return the old value. A generator would hide all of it. That verbosity is the price of owning the cursor, and it is the reason hand-written iterators are the exception and not the default.

from itertools import islice

class Forever:
    def __init__(s, n):
        s.i, s.n = 0, n

    def __iter__(s):
        return s

    def __next__(s):
        s.i += 1          # the guard is missing
        return s.i - 1

print(list(islice(Forever(3), 6)))

islice cuts it off safely so we can watch the bug.

The object was built with n = 3, yet it happily produced six values and would produce more. Bounding an unknown iterator with islice while you debug is a safe way to find out whether the guard is missing.

Suppose you write a container class, a shelf of items, and you want people to loop over it. The tempting shortcut is to put __next__ on the container and have __iter__ return self. It works for the first loop. The second loop then starts with the cursor already at the end, and it does not raise any error.

The fix is the separate-class pattern. The container's __iter__ builds and returns a new iterator object every time it is called. The container holds the data, and each loop gets its own cursor with its own state.

The example below defines one cursor class and uses it two ways. Shelf is the correct container, and BadShelf is the anti-pattern, because it inherits __iter__ returning self, so the container and the iterator are the same object.

Shelf can be looped over any number of times, and two cursors made with iter(shelf) move independently. BadShelf yields everything once and then an empty list forever, with no exception to point at the cause.

The module collections.abc removes some boilerplate. If your class inherits from collections.abc.Iterator, the abstract base class supplies __iter__ for you (it returns self), and you only write __next__. Forgetting __next__ then fails at instantiation instead of at the first loop.

Hand-written iterators still earn their keep when the iterator is an object in its own right. A class can carry extra methods such as a reset, expose its state as plain attributes you can print, and be pickled, which a running generator cannot. The next example shows all three on one small class.

The isinstance checks pass because Iterator is itself a subclass of Iterable. Those ABCs also check structure: any class that defines __iter__ counts as an Iterable, even without inheriting. A class that is iterable only through the old __getitem__ protocol is the exception, and registration fixes it. Without register, isinstance checks and type-driven code disagree with what iter() actually accepts.

Now compare the class with the generator. The behaviour is the same, and a generator function gets it from the compiler: it is an iterator the language writes for you, with the cursor, the guard and StopIteration all handled.

def count(n):
    for i in range(n):
        yield i

print(list(count(4)), list(count(4)) == list(Count(4)))

Three lines against twelve; Count is the class from the first page.

A function becomes a generator function the moment the keyword yield appears anywhere in its body. The yield can sit inside a loop, behind an if, or on a line that never runs. The compiler marks the code object when it compiles the function, so the decision is made before anything executes.

The surprising part is what a call does. Calling a generator function runs none of its body. Python builds a generator object, hands it back, and stops. That object is already an iterator: it has __iter__, which returns itself, and __next__, which runs the body up to the next yield. You write neither method, because the compiler supplies both.

The example below makes the delay visible. The print('a') and print('b') lines are inside the body, so they appear only when next() drives the generator. The two lines printed before the first next() show that the call itself produced no output.

from collections.abc import Iterator

def gen():
    print('a')
    yield 1
    print('b')
    yield 2

g = gen()
print('created', type(g).__name__)
print(isinstance(g, Iterator), iter(g) is g)
print(next(g))
print(next(g))

Nothing from the body prints until the first next()

Read the output in order. The call gen() printed nothing, and the generator object passed the Iterator check with no methods written by us. The first next(g) ran the body from the top, printed a, and stopped at yield 1. The value 1 came back to the caller, and the caller's print showed it. The second next(g) printed b and returned 2.

yield does three things at once. It suspends the running frame and hands one value to the caller. It preserves every local variable and the exact position in the code. On the next next(), execution continues right after the yield expression, with all locals as they were. Control moves back and forth between caller and generator, and each side runs only while the other waits.

The comparison with an ordinary function shows why this works. A normal return ends the call and destroys the frame, so its locals are gone. A yield leaves the frame alive. The generator object holds a reference to it on the heap until the generator finishes or is discarded.

The standard library lets you inspect this. inspect.getgeneratorstate reports the lifecycle stage, and gi_frame.f_locals shows the paused variables. Once the generator finishes, gi_frame becomes None because the frame has been released.

A fresh generator is GEN_CREATED. After two next() calls it sits suspended at the yield inside the loop, and total is still 15, though no function call is active. Draining it with list(g) closes it and drops the frame. This is why a generator is best seen as a resumable function: cooperative multitasking in miniature, where the function chooses when to pause. It is not a container, and no collection of values sits behind it.

A generator has two ways to finish. The simple one is to fall off the end of the body. Python then raises StopIteration for you, which is the signal for loops, list() and friends wait for. The other is return value. The value is not yielded. It is stored on the exception as StopIteration.value.

There are two ways to read that value. You can catch the exception yourself and look at e.value. Or you can delegate with yield from, which swallows the StopIteration and evaluates to the returned value. The example shows both endings and the delegation case.

The generator that falls off the end reports None, while job reports 'done'. In outer, yield from job() passes both steps through to list(). It also captures 'done' into result, and list() never sees that value. A for loop ignores StopIteration.value in the same way, so return is useful only to callers who go looking for it.

Several yields in a body, yields inside while or for loops, and yields inside try/finally are all legal and common. The try/finally case matters for resources, because the finally block runs when the generator is closed early as well as when it finishes.

We read one line and then called close(). Python raised GeneratorExit at the paused yield, the finally block ran, and only then did control return to us. Without close(), the cleanup would wait until the generator is garbage collected, and that timing is not something to rely on for files or sockets.

When you call a generator function, nothing in its body runs. Python builds a generator object that owns a paused frame, which holds the function's local variables and a marker for where execution stands. Every next() thaws that frame, runs it to the next yield, and freezes it again. This page opens the object up so you can see that machinery.

A generator exposes its internals through four gi_ attributes. They are read-only views, useful for debugging and for understanding what the interpreter is holding on your behalf.

Every generator is in exactly one of four states. You never set the state yourself. It follows from what has been called on the object, and inspect.getgeneratorstate(g) reports it as a string.

A generator moves between RUNNING and SUSPENDED once per item. It leaves for CLOSED when the body returns, when an exception escapes, or when close() is called. CLOSED is permanent: a closed generator never runs again.

Once the generator is exhausted, gi_frame becomes None. The frame has been released, which is the last time its locals were reachable through the generator.

The compiler sets the CO_GENERATOR flag in the code object's co_flags for any function whose body contains yield. A normal call creates a frame and executes it. When the interpreter sees this flag, it creates the frame, wraps it in a generator object, and returns that object without running a single line of the body. That one bit is the whole difference between def f(): return 1 and def g(): yield 1.

An ordinary function call frame is thrown away on return. A generator's frame is heap-allocated and the generator object keeps a reference to it. As long as the generator is alive, so is the frame, and with it every local variable and the saved position in the bytecode. Nothing special is copied on yield. The frame simply stays put, and next() resumes it.

Because the frame is still there, you can read it. gi_frame.f_locals maps local names to their current values, and gi_frame.f_lasti is the index of the last bytecode instruction executed, which tells you how far in the body the generator has progressed. The example below also checks the flag on a plain function and a generator function, and shows gi_yieldfrom pointing at a delegate.

Calling g.close() on a suspended generator throws GeneratorExit into the frame at the yield where it is paused. If the generator does not catch it, the exception propagates out, close() swallows it, and the state becomes CLOSED. The one rule is that the generator must not yield another value after receiving GeneratorExit. It should clean up and let the exception go, or simply return.

Since GeneratorExit is a real exception raised inside the body, try/finally and with blocks around the paused yield unwind normally. The same thing happens when the last reference to a suspended generator disappears and it is garbage collected, because the finalizer closes it first. The example nests a with inside a try/finally to show the unwinding order: inner blocks run first.

The first follows from the close contract: close() expects the generator to stop, so a further yield is a refusal and Python reports it. The second is non-reentrancy. A generator has a single frame and that frame is mid-execution, so asking it to resume itself would corrupt its state. This shows up when a generator, directly or through a callback, tries to pull from itself.

A suspended frame keeps everything its locals refer to alive. If one of those references leads back to the generator itself, you have a reference cycle. Reference counting alone can never free it, so the generator's finally block waits until the cyclic garbage collector happens to run. In the example below, the generator's argument holds a dict that holds the generator. Dropping our names does nothing visible until gc.collect().

A computation is lazy when its value is produced only at the moment someone asks for it. Until then the work is deferred, and if nobody ever asks, the work is never done at all. A generator is the standard example. Calling a generator function runs none of its body, and each next() runs just far enough to reach the next yield.

The example below makes that visible. The generator prints a message each time it computes a value. Watch where the messages land relative to the calls that consume the values.

def noisy_squares(n):
    for i in range(n):
        print(f"computing {i}")
        yield i * i

gen = noisy_squares(3)
print("created, nothing computed yet")
print(next(gen))
print("between calls")
print(next(gen))

Work happens inside next(), not at creation

The third value was never computed, because nobody asked for it. That is deferral in its purest form: the cost of an item is paid only by the consumer who wants it.

A paused generator holds exactly two things: its suspended frame (the local variables and the position where it stopped) and the single item it is about to hand over. Nothing about the items already delivered or the items still to come is stored. The space is O(1), whether the stream has ten items or ten billion.

A list comprehension takes the opposite approach and builds every element before you can touch the first one. For [x*x for x in range(10_000_000)] that means ten million int objects plus an 80 MB pointer array, roughly 400 MB in total. The generator expression (x*x for x in range(10_000_000)) is a few hundred bytes, because it has done nothing yet.

Laziness pays off most when you do not need the whole stream. next(x for x in huge if pred(x)) pulls items only until the predicate first succeeds, then stops. An eager version would filter every item before you could look at the first hit. The counter in this example shows how few items get examined out of ten million.

checked = 0

def is_big(x):
    global checked
    checked += 1
    return x > 1000

hit = next(x for x in range(10_000_000) if is_big(x))
print(hit, checked)

The scan ends at the first match, not at the last item

Only 1002 items were tested, and the other 9,998,998 were never produced. If no item matches, next raises StopIteration. Pass a default, as in next(gen, None), when a miss is a normal outcome.

An eager container must be finite, because it has to exist in memory. A lazy producer has no such limit, since it only ever makes the next value. A while True loop around a yield is a perfectly valid, safe definition of an endless sequence. The danger lives only in the consumer, which must not try to swallow all of it, as list(naturals()) or sum(naturals()) would.

You bound an infinite stream from the consuming side. itertools.islice takes a fixed number of items, and a plain break leaves the loop when you have seen enough. Either way the generator is simply left suspended.

from itertools import islice

def naturals():
    n = 0
    while True:
        yield n
        n += 1

print(list(islice(naturals(), 10)))

for n in naturals():
    if n * n > 50:
        break
print(n)

Two ways to bound an endless stream

Iterating a file object, as in for line in f, yields one line at a time and reads from disk in small buffered chunks. The whole file never has to fit in memory. f.readlines() does the opposite: it loads every line into one list, which on a multi-gigabyte log means exhausting RAM before the first line is processed.

Generator expressions can be chained, each one reading from the one before it. Writing lines = (l.strip() for l in f) and then nums = (int(l) for l in lines if l.isdigit()) builds two stages, and no data moves yet. Only when a consumer such as sum(nums) asks for a value does the request travel down the chain, and each stage pulls exactly one item from the stage beneath it.

To see the interleaving, the next example uses a small generator function for the first stage so it can report each read. An in-memory text stream stands in for the file.

import io

def stripped(f):
    for raw in f:
        print("read", repr(raw))
        yield raw.strip()

f = io.StringIO("10\nabc\n20\n")
nums = (int(l) for l in stripped(f) if l.isdigit())
for n in nums:
    print("got", n)

Reads and results alternate; nothing is read ahead

The line abc was read and rejected inside the pipeline without the loop body ever seeing it. At no point did more than one line exist at once, so sum(nums) over this chain is a single pass with O(1) memory, however big the file is.

Going lazy changes when you get results, not how much total work there is. The first result arrives almost immediately, because only the work for one item has to happen. This matters for a progress display, for a stream you may abandon, or for a pipeline feeding a network client. The full pass still touches every item, and each item now pays a small resume-and-suspend toll, so total throughput can come out slightly lower than a tight list comprehension.

For small data that is walked more than once, the list is both faster and simpler. Laziness earns its keep when the data is large, unbounded, or arrives over I/O, where holding it all would be expensive or impossible.

Eager code runs in the order it is written. Lazy code runs in the order it is consumed, which can be far from where the expression appears. Side effects such as logging, counters and network calls fire later than you expect, and any names the expression reads are looked up at consume time. A traceback also points at the line doing the consuming, not at the stage that actually failed. The example shows both timing surprises.

log = []

def tag(x):
    log.append(x)
    return x

lazy = (tag(x) for x in [1, 2, 3])
print(log)
list(lazy)
print(log)

limit = 2
small = (x for x in range(5) if x < limit)
limit = 4
print(list(small))

Side effects and closed-over names resolve at consume time

The log stayed empty until the generator was consumed. The filter also used limit = 4 rather than the 2 that was current when the expression was written, because the condition is evaluated lazily. The outermost iterable, here range(5), is the one part evaluated immediately.

A generator expression is a list comprehension with its brackets swapped for parentheses: (expr for x in iterable if cond). The clauses mean the same thing in both forms. The difference is when the work happens. The list comprehension builds every result immediately. The generator expression builds a generator object and produces one value each time something asks for the next one.

Parentheses are optional when the genexp is the sole argument of a call, so sum(x*x for x in data) is valid and is the usual way to write it. If the call has a second argument, the genexp needs its own parentheses. Otherwise Python cannot tell where the expression ends, and you get a SyntaxError.

data = [1, 2, 3, 4, 5, 6]
squares = [x * x for x in data if x % 2 == 0]
lazy = (x * x for x in data if x % 2 == 0)
print(squares)
print(type(lazy).__name__)

print(sum(x * x for x in data))

def scale(items, factor):
    return [i * factor for i in items]

print(scale((x for x in data), 10))
try:
    compile('scale(x for x in data, 10)', '<s>', 'eval')
except SyntaxError:
    print('SyntaxError')

Brackets versus parentheses, the sole-argument shortcut, and the two-argument rule

Nearly everything inside a genexp waits until you consume it. The exception is the outermost iterable, the one after the first in. Python evaluates it immediately, at the moment the genexp is created, and calls iter() on it right then. Only the filter, the expression and any inner for clauses are deferred.

The consequence is that the generator holds a reference to the original list object, not to the variable name. Rebinding the name afterwards changes nothing for the generator. Mutating that same list object, on the other hand, is visible to it. A non-iterable outermost value fails at creation, not at first next().

mylist = [1, 2, 3]
g = (x for x in mylist)
mylist = []
print(list(g))

nums = [1, 2, 3]
h = (n for n in nums)
nums.append(4)
print(list(h))

try:
    (x for x in 5)
except TypeError:
    print('TypeError at creation')

Rebinding the name does not matter; mutating the object does

The outermost iterable is captured early, but every other name in the genexp is looked up when each value is produced. The filter and the expression are closures over the enclosing scope. If you rebind a name between creating the generator and consuming it, the generator sees the new value. The same rule explains the classic lambda-in-a-loop surprise, because the lambdas all share one variable.

limit = 2
g = (x for x in [1, 2, 3, 4] if x > limit)
limit = 3
print(list(g))

factor = 10
g2 = (x * factor for x in [1, 2, 3])
factor = 100
print(list(g2))

fs = list(lambda: i for i in range(3))
print([f() for f in fs])
fs = [lambda i=i: i for i in range(3)]
print([f() for f in fs])

The filter used 3, not 2; the lambdas share one i until a default argument pins it

The choice follows from how the data is used. Take a list comprehension when you must walk the result twice, index into it, or ask for its length, or when the data is small enough that memory does not matter. Take a genexp when you make a single pass over data that is large or unbounded.

The built-ins that reduce an iterable accept a genexp directly, so no intermediate list is needed. These are sum, any, all, min and max. The same goes for str.join, set and dict. any and all also stop pulling items as soon as the answer is known.

Writing sum([x*x for x in it]) builds the entire list just to add it up and throw it away. That is real wasted memory, and the cost grows with the input. ''.join([str(x) for x in it]) is a milder case. str.join needs a sized sequence, so it converts any iterable to a list internally anyway, and the explicit brackets cost almost nothing extra. Dropping the brackets there is still the cleaner style. The saving that matters is on sum, min, max, any and all, which stream.

import tracemalloc

def peak(fn):
    tracemalloc.start()
    fn()
    _, top = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return top

big = range(200_000)
with_list = peak(lambda: sum([x * x for x in big]))
with_gen = peak(lambda: sum(x * x for x in big))
print(with_list > 1_000_000)
print(with_gen < 50_000)
print(sum([x * x for x in big]) == sum(x * x for x in big))

Same answer, very different peak memory

In Python 3, every comprehension and genexp runs in its own function-like scope. The loop variable lives only inside it and never leaks into the enclosing function, which Python 2 list comprehensions did. Your own variable with the same name is left alone. The one deliberate exception is an assignment expression (:=), which binds in the enclosing scope. The outermost iterable is also evaluated in the enclosing scope, which ties back to why it is evaluated early.

So far yield has only pushed values out. It is also an expression, so it can produce a value. In received = yield value, the generator hands value to the caller, pauses, and later resumes with received set to whatever the caller passed in. This turns a one-way stream into a conversation.

The caller supplies that value with g.send(x). It resumes the generator and makes the paused yield evaluate to x. The generator runs until its next yield, and send returns the value yielded there, just as next() would. In fact next(g) is the same as g.send(None).

def echo():
    received = yield 'ready'
    while True:
        received = yield f'got {received}'

g = echo()
print(next(g))
print(g.send('a'))
print(g.send('b'))

The first next() runs to the first yield; each send() answers the previous yield

A brand-new generator has not run any code yet, so no yield is paused and waiting for a value. You must therefore advance it once with next(g) or g.send(None) to reach the first yield. This step is called priming. Sending a real value to an un-primed generator raises TypeError: can't send non-None value to a just-started generator.

The classic example is an accumulator. The line total += yield total yields the running total, then adds whatever the caller sends in. The example below also shows the priming error.

def acc():
    total = 0
    while True:
        total += yield total

a = acc()
try:
    a.send(10)
except TypeError as e:
    print(e)
print(next(a))
print(a.send(10))
print(a.send(5))

The first send(10) fails; next(a) primes the generator

send pushes a value into the paused yield. throw pushes an exception in instead. g.throw(ValueError) raises the exception at the exact point where the generator is suspended. If the generator wraps that yield in try/except, it can handle the error and carry on. It then runs to its next yield, and throw returns that yielded value. If the generator does not catch the exception, it propagates out of throw to the caller and the generator is finished.

def tolerant():
    total = 0
    while True:
        try:
            total += yield total
        except ValueError:
            print('bad value, resetting')
            total = 0

a = tolerant()
next(a)
print(a.send(5))
print(a.throw(ValueError))
print(a.send(2))

The generator catches the thrown error and keeps running

g.close() is the shutdown call. It raises a special exception, GeneratorExit, at the paused yield. The generator should run any cleanup it needs and then let the exception propagate, either by not catching it or by re-raising it. If the generator yields another value after catching GeneratorExit, Python raises RuntimeError. After a clean close the generator is finished.

from inspect import getgeneratorstate

def worker():
    print('open')
    try:
        while True:
            yield
    except GeneratorExit:
        print('cleanup')
        raise

w = worker()
next(w)
w.close()
print(getgeneratorstate(w))

Cleanup runs, GeneratorExit propagates, the generator ends closed

With send, throw and close in place, a generator is no longer just a producer. It is a function that can be suspended, fed data, interrupted and shut down. These generators are the classic coroutines of PEP 342, and they are the ancestor of async def and await.

The two are not the same thing, though. A generator-based coroutine is driven by next() and send() calls that you write. An async def function returns a native coroutine object, which uses its own protocol built on __await__. You do not step it with next(); you await it, and an event loop such as asyncio drives it. The coroutine object does have send, throw and close methods, but it has no __next__.

import asyncio

async def job():
    return 1

c = job()
print(hasattr(c, '__await__'), hasattr(c, '__next__'))
c.close()
print(asyncio.run(job()))

A native coroutine is awaitable, not iterable

Forgetting to prime is such a common bug that people wrote a small decorator to remove it. The decorator creates the generator, calls next() once so it is already waiting at its first yield, and returns it ready for send. Older coroutine code used this idiom everywhere.

from functools import wraps

def coroutine(func):
    @wraps(func)
    def start(*args, **kwargs):
        g = func(*args, **kwargs)
        next(g)
        return g
    return start

@coroutine
def running_total():
    total = 0
    while True:
        total += yield total

r = running_total()
print(r.send(10))
print(r.send(5))

No manual priming: the decorator already advanced to the first yield

Sooner or later a generator needs to hand part of its job to another iterable. The obvious way is a loop that re-yields every item. yield from iterable does the same thing in one statement: it pulls each item out of the sub-iterable and yields it to your caller, in order, until the sub-iterable is exhausted. Then your generator carries on with the next line.

def manual(sub):
    for x in sub:
        yield x

def delegated(sub):
    yield from sub

def chain(*its):
    for it in its:
        yield from it

print(list(manual('ab')), list(delegated('ab')))
print(list(chain([1, 2], (3,), range(4, 6))))

The first two are equivalent; the third is a one-line itertools.chain.

chain shows how far the idea stretches. Each argument can be a different kind of iterable, because yield from only needs something it can call iter() on. The real itertools.chain is written in C and is the one to use in production. Writing your own is a good way to see what delegation does.

If replacing a four-line loop were all yield from did, PEP 380 would hardly have been needed. The real reason is that it also forwards send(), throw() and close() to the subgenerator. While the subgenerator is running, the delegating generator is out of the picture. The caller talks straight through it. The hand-written loop cannot do this, because for x in sub: yield x throws away every value sent in and cannot pass an exception down. Getting it right by hand takes a page of try/except code.

The subgenerator can also finish with return value. A plain for loop never sees that value, because it is hidden in StopIteration.value. The expression form result = yield from subgen() catches that exception for you and gives you the value as the result of the expression. The example below uses all of this. A running total lives in the inner generator. Values sent by the caller go straight to it. An exception thrown from outside ends the inner generator, and its total comes back to the outer one.

def inner():
    total = 0
    try:
        while True:
            total += yield total
    except ValueError:
        return total

def outer():
    result = yield from inner()
    print('inner returned', result)
    yield 'done'

g = outer()
print(next(g))
print(g.send(5))
print(g.send(10))
print(g.throw(ValueError))

send() and throw() go to inner() even though the caller only holds outer().

Notice that outer contains no code for receiving values or catching ValueError. The caller sends numbers to outer, but inner is the one that receives them. When throw(ValueError) arrives, it is raised inside inner, which handles it and returns 15. That value becomes result in outer, which then goes on to yield 'done'. The same forwarding applies to close(): closing outer closes inner first.

Recursion is where yield from is most natural. To flatten arbitrarily nested data, you walk the items. When an item is itself a container, you delegate to a recursive call of the same generator. Otherwise you yield the item as it is. Each level of nesting hands its work to the level below, and the caller sees one flat stream.

from collections.abc import Iterable

def flatten(items):
    for item in items:
        if isinstance(item, Iterable) and not isinstance(item, (str, bytes)):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 'ab']], (b'xy', 4)])))

The isinstance guard is the important line.

The isinstance(item, Iterable) and not isinstance(item, (str, bytes)) condition looks fussy, but it is the whole trick. A string is iterable, and iterating over it gives one-character strings. A one-character string is also iterable, and iterating over it gives itself. So a flatten without the guard recurses forever. This is the classic infinite-recursion bug of the flatten exercise. Python stops it with a RecursionError. The guard also keeps bytes whole. Bytes would not recurse forever, because their items are ints, but they would still be split into numbers when you almost certainly wanted them kept as one value.

def bad_flatten(items):
    for item in items:
        try:
            iter(item)
        except TypeError:
            yield item
        else:
            yield from bad_flatten(item)

try:
    list(bad_flatten([1, 'ab']))
except RecursionError:
    print('RecursionError: a one-character string never stops iterating')

No string guard: 'ab' becomes 'a', and 'a' yields 'a' again, forever.

Delegation is not free, but it is cheaper than the loop it replaces. With for x in sub: yield x, every item is received by your generator's bytecode and then yielded again, which means running Python-level instructions once per item. With yield from, the interpreter drives the sub-iterator itself and passes each item straight to your caller. The per-item loop is gone, so it is measurably faster on long streams. The exact gain depends on your Python version and on how cheap the items are, so run timeit on your own data. Do not rely on a number from a blog post.

Saving the loop does not remove the stack. Every level of delegation is a live generator frame, and a recursive flatten of nested data holds one frame per level of nesting. Python's recursion limit counts these frames, so very deep nesting fails even though each level is small. The limit applies to depth only. Flattening a very wide list is fine.

import sys

def nest(depth):
    obj = 1
    for _ in range(depth):
        obj = [obj]
    return obj

print(sys.getrecursionlimit())
print(list(flatten(nest(100))))
try:
    list(flatten(nest(5000)))
except RecursionError:
    print('too deep')

Reuses flatten() from the previous page. The limit shown is Python's default.

Before asyncio had async def and await, coroutines were written as generators, and yield from was how one coroutine waited on another. The @asyncio.coroutine decorator marked such a generator, and yield from other_coro() suspended it until the result was ready. That is the same delegation idea you have just seen: a pipe to a subgenerator, plus its return value. Native coroutines replaced this style, and @asyncio.coroutine was removed in Python 3.11.

The itertools module is a box of small, lazy building blocks. Each one takes iterables and returns an iterator, so nothing is computed until something pulls a value. The three infinite sources never stop on their own. Whatever consumes them has to supply the stopping rule.

The terminating combinators reshape or trim a stream and end when their input ends. Together they cover most of the small loops you would otherwise write by hand.

The example below runs every one of them on a small list. Notice that takewhile and dropwhile split the list at the first even number (6), and that dropwhile stops testing after that point, so the trailing 7, 9 and 2 all pass through.

from itertools import count, cycle, repeat, islice, chain, takewhile, dropwhile, filterfalse, compress

print(list(islice(count(10, 5), 4)))
print(list(islice(cycle('ab'), 5)))
print(list(repeat('x', 3)))

nums = [1, 3, 5, 6, 7, 9, 2]
odd = lambda n: n % 2
print(list(chain([1, 2], 'ab')))
print(list(chain.from_iterable([[1, 2], [3], []])))
print(list(takewhile(odd, nums)))
print(list(dropwhile(odd, nums)))
print(list(filterfalse(odd, nums)))
print(list(compress('abcde', [1, 0, 1, 0, 1])))
print(list(islice(nums, 2, 5)))

Infinite sources bounded, then the terminating tools

groupby(iterable, key) walks the input once and starts a new group every time the key changes. It does not gather equal keys from across the whole input. It only merges neighbours. So the same key can appear in several groups unless the data is already ordered by that key.

The last step matters because of how groupby is built. It produces one shared stream and hands out each group as a window onto it. Asking the outer iterator for the next group moves that shared stream forward, which invalidates the previous group's sub-iterator. An old group then looks empty, because its items were skipped over. Calling list() on a group while you are still inside its loop turn is how you keep its contents.

from itertools import groupby

words = ['apple', 'avocado', 'banana', 'blueberry', 'apricot']
first = lambda w: w[0]
for k, g in groupby(words, first):
    print(k, list(g))
print('--')
for k, g in groupby(sorted(words, key=first), first):
    print(k, list(g))
print('--')
it = groupby([1, 1, 2, 2])
k1, g1 = next(it)
k2, g2 = next(it)
print(k1, list(g1))
print(k2, list(g2))
print([(k, list(g)) for k, g in groupby([1, 1, 2, 2, 3])])

Fragments, the sorted fix, and an invalidated group

The first loop shows the fragment problem: 'a' appears twice because 'apricot' is not next to the other a-words. Sorting by the same key fixes it. In the third part, advancing to the second group made g1 empty, while the one-line comprehension is safe because it copies each group before the outer iterator moves on.

Four functions generate arrangements of their input, and they differ only in whether order matters and whether an item can be reused. All of them are lazy, so you can pull the first few results from an enormous space without building it.

Laziness does not save you from the size of the answer. Counts grow factorially, so walking the whole space is only realistic for small inputs. math.perm and math.comb compute the count without generating anything. Use them to check before you loop.

tee(iterable, n) solves a different problem: you have one iterator but need several independent readers. It returns n iterators fed from the original. To make that work, tee buffers every item that the fastest reader has taken but the slowest has not. Memory therefore grows with the gap between your slowest and fastest consumer.

from itertools import product, permutations, combinations, combinations_with_replacement, tee
from math import perm, comb

print(list(product('ab', [0, 1])))
print(list(permutations('abc', 2)))
print(list(combinations('abc', 2)))
print(list(combinations_with_replacement('ab', 2)))
print(perm(10, 10), comb(50, 5), perm(20, 20))

a, b = tee(iter(range(5)))
print(next(a), next(a), next(a))
print(list(b))

The four generators, their sizes, and a tee with a gap

Ten items fully permuted already give 3,628,800 results, and twenty give about 2.4 quintillion. In the tee part, a ran three items ahead, so those three sat in the buffer until b caught up. If one reader will consume everything before the other starts, list() is simpler and no worse on memory.

In Python 3, zip() is lazy: it pulls one item from each input per step and builds a tuple. It stops as soon as the shortest input runs out. That is usually what you want, but it can silently drop data. itertools.zip_longest(..., fillvalue=...) keeps going until the longest input ends and fills the gaps with a value you choose.

The same machinery gives the classic batching idiom, zip(*[iter(x)] * n). The list multiplication repeats one iterator object n times, so all n slots of each tuple pull from the same cursor and each tuple takes the next n items. Its weakness is that an incomplete final batch is dropped.

from itertools import zip_longest

nums = [1, 2, 3]
letters = 'ab'
print(list(zip(nums, letters)))
print(list(zip_longest(nums, letters, fillvalue='-')))
print(list(zip(*[iter(range(6))] * 3)))
print(list(zip(*[iter(range(7))] * 3)))

Shortest, longest, and the batching idiom

The last line shows the loss: with seven items, the 6 vanished. Python 3.12 adds itertools.batched(iterable, n), which reads clearly and keeps a short final batch. On 3.11 you can write the same thing yourself with islice, as below. Next to it, pairwise(it) (3.10+) yields overlapping consecutive pairs, which is exactly what deltas and sliding windows of size two need.

from itertools import islice, pairwise

def batched(iterable, n):
    it = iter(iterable)
    while batch := tuple(islice(it, n)):
        yield batch

print(list(batched(range(7), 3)))
temps = [20, 23, 22, 27]
print([b - a for a, b in pairwise(temps)])
print(list(pairwise('abcd')))

A batched stand-in for 3.11, and pairwise for deltas

accumulate(data, func, initial=...) is a lazy running fold. With no function it adds. With any two-argument function it carries the result forward and yields each intermediate value. If you pass initial, that value is yielded first and the output is one item longer than the input.

import operator
from itertools import accumulate

data = [3, 1, 4, 1, 5]
print(list(accumulate(data)))
print(list(accumulate(data, max)))
print(list(accumulate(data, operator.mul, initial=1)))
print(list(accumulate(data, max, initial=0)))

Running sum, running maximum, running product

Every primitive in itertools is implemented in C. They avoid the per-item cost of Python bytecode and hold only a tiny amount of state, so they usually beat the equivalent Python loop on both speed and memory.

Laziness is a trade, not a free upgrade. A generator gives up some raw speed and some convenience, and in return it gives you a constant memory footprint and an instant first result. Before reaching for one, you should know exactly what you are buying and what it costs.

A list comprehension builds every element and holds them all at once, so its memory grows as O(n). A generator keeps one suspended frame and one current item, so its memory is O(1) however long the stream is. Ten million squares in a list cost hundreds of megabytes; the same ten million as a genexp cost a few hundred bytes. This is the single biggest reason to go lazy.

Walking every item costs O(n) either way. The difference is the constant factor. A list comprehension runs one tight loop that appends to a list. A generator must suspend and resume its frame for every item it hands out, and that resume is real work. In a tight numeric loop expect the generator to be roughly 10-30% slower than the equivalent list comprehension. You pay that per-item overhead to avoid the O(n) allocation.

A list comprehension is all-or-nothing: nothing is visible until the last element has been computed. A generator hands over the first item after doing only the work for that item. The program below counts how many times the expensive function ran before the first result was available. It also uses sys.getsizeof to show why that function is misleading for generators: it measures the generator object's header, not the data it will eventually produce, so a generator over ten items and one over a billion report the same size.

import sys

calls = 0
def square(x):
    global calls
    calls += 1
    return x * x

eager = [square(x) for x in range(1000)]
print('list comp calls before first item:', calls)

calls = 0
lazy = (square(x) for x in range(1000))
next(lazy)
print('genexp calls before first item:', calls)

def squares_list(n): return [x * x for x in range(n)]
def squares_gen(n): return (x * x for x in range(n))

print(sys.getsizeof(squares_gen(10)) == sys.getsizeof(squares_gen(10**9)))
print(sys.getsizeof(squares_list(10**5)) > 800_000)

Latency counted in calls, size compared without printing machine-specific byte counts

When you stack generator stages, no stage builds a collection. The final consumer asks the stage below for one item, which asks the stage below that, and so on down to the source. Each item travels through every stage before the next item is even touched, so the whole chain behaves like a single loop with no intermediate lists and no repeated allocation.

The trace stages below print as each item passes through. If the stages were eager, you would see all the read lines first, then all the double lines. Instead the lines interleave, which proves that one item completes the whole journey before the next one starts.

Memory savings only matter when memory is actually a problem. For small inputs the list is both faster, because it avoids the per-item resume, and simpler, because you can index it, measure it and loop over it twice. The crossover depends on the item size and the work per item, but as a rule of thumb a genexp earns its keep only past a few thousand items, or when the data is unbounded, or when you will stop early.

Intuition about generator speed is frequently wrong. It varies with the Python version, the cost of the work per item, and the size of the data. Use timeit on sizes that resemble your real workload, repeat the measurement and compare the best runs. A benchmark on ten items tells you almost nothing about ten million.

python -m timeit -s "n = 100_000" "sum([x * x for x in range(n)])"
python -m timeit -s "n = 100_000" "sum(x * x for x in range(n))"
python -m timeit -s "n = 100" "sum([x * x for x in range(n)])"
python -m timeit -s "n = 100" "sum(x * x for x in range(n))"

Run the same pair at a small and a large n and compare. Your numbers will differ from anyone else's, and that is the point.

A generator can only move forward, one item at a time, and it forgets what it has already produced. Anything that needs to look backward, look ahead, or know the total size forces you to pay for a real collection anyway.

There is no g[5] and no len(g). You can reach a later item with itertools.islice, but it works by consuming and discarding everything before it, so it costs O(n) and it is destructive: the items it skipped are gone. Repeating it does not start over, it continues from wherever the generator now stands. Counting with sum(1 for _ in g) is also O(n) and uses the generator up, leaving nothing for the real work.

from itertools import islice

g = (x * x for x in range(10))
print(next(islice(g, 3, None)))
print(next(islice(g, 3, None)))

g = (x for x in range(5))
print(sum(1 for _ in g))
print(sum(1 for _ in g))

data = (n for n in [3, 1, 2])
print(sorted(data))

The same index gives a different answer the second time because the first call consumed items

The rule behind the table is simple: if the algorithm walks the data once, in order, stay lazy. If it needs the whole data set in hand at some point, laziness buys nothing there, and building the list explicitly is clearer.

The strongest case for generators is data that is large or never ends: a multi-gigabyte log file, a socket, a message queue, a paginated API. There the cost of a frame resume is invisible next to the cost of waiting for the disk or the network, and the alternative, loading everything first, may be impossible. A lazy pipeline holds only the current record, so memory stays constant no matter how much data flows through.

The example below processes an endless stream of records and keeps a running maximum. Nothing ever holds the whole stream, and islice bounds how much we look at. The same shape works with a file object (iterating a file yields one line at a time) or a network reader; just replace ticks() with the real source.

from itertools import islice

def ticks():
    n = 0
    while True:
        n += 1
        yield f'{n},{n % 7}'

def running_max(lines):
    best = 0
    for line in lines:
        _, v = line.split(',')
        best = max(best, int(v))
        yield best

print(list(islice(running_max(ticks()), 10)))

An endless source, constant state, results available immediately

Most iterator bugs in production share one trait: nothing crashes. The code runs, returns a plausible value, and the value is wrong. The cause is almost always that a generator is a single-use cursor, not a container. Once it has produced its last item, it stays empty for good, and asking it again is not an error.

An exhausted generator raises StopIteration on every next() call. A for loop or list() treats that as the normal end of the stream, so a second pass over the same generator simply does zero iterations. No exception, no warning. The classic form of this bug is calling list(g) to look at the data and then sum(g) to total it. The sum receives an empty stream and returns 0, which is a perfectly valid sum.

g = (x * x for x in range(4))
print(list(g))
print(sum(g))
print(bool(g))

The second pass sees an already-finished generator.

The last line is a second trap. The generator is spent, yet bool(g) is still True. A generator object defines neither __bool__ nor __len__, so Python falls back to the default: every object is truthy. That means if g: never tests whether the generator will yield anything, before or after it is consumed.

Decide whether the data is walked once or many times. If it is walked more than once, materialize it with data = list(g) and reuse the list. If it is too big for that, wrap the generator expression in a function and call the function each time you need a fresh pass. itertools.tee also works, but it buffers whatever the faster copy has read ahead, so it only helps when the copies stay close together.

Laziness defers work, and deferred work reads the world at the moment it finally runs, not the moment you wrote it. Two rules govern a generator expression. The outermost iterable is evaluated immediately, when the genexp is created. Everything else, including the element expression, any filters and any inner loops, runs on demand, one step at a time.

The eager part means the genexp holds an iterator over the original object. If you rebind the name (src = []) the genexp keeps the old list, because it never looks up src again. If you mutate the same list (src.append(4)), the iterator walks the live list and will see the new item. A bad outermost expression also fails at creation, not at first use, which is the one place a genexp fails early. Every other name used inside the expression is looked up late, at consumption time.

src = [1, 2, 3]
g = (x * 10 for x in src)
src.append(4)
print(list(g))

factor = 10
h = (x * factor for x in [1, 2])
factor = 100
print(list(h))

fs = list(lambda: i for i in range(3))
print([f() for f in fs])
fs = [lambda i=i: i for i in range(3)]
print([f() for f in fs])

Mutation after creation, a late-read name, and late-binding lambdas.

The first result shows the eager outermost iterable at work: the genexp captured the list object, so the appended 4 shows up. The second shows the lazy half: factor was 10 when the genexp was built but 100 when it ran. The last pair is the same effect in its famous form. Each lambda closes over the variable i, not over its value at that moment. By the time you call them, the loop is over and i is 2, so every lambda returns 2. A default argument is evaluated when the lambda is defined, so lambda i=i: i freezes the current value into each function.

Since Python 3.7 (PEP 479), a StopIteration that is raised inside a generator body and not caught there is converted into RuntimeError: generator raised StopIteration. Before this change, the exception leaked out and the consumer's loop treated it as a normal end, so a bug could silently truncate the output. Now the bug is loud.

The practical consequence is about the bare built-in next(). It is tempting to pull items from a second iterator inside a generator and let exhaustion end things naturally. That no longer works. Give next a sentinel default and decide explicitly what to do when the data runs out, or wrap the call in try/except StopIteration and return. The example below pairs up items and shows an odd-length input breaking the first version.

def pairs(it):
    inner = iter(it)
    while True:
        a = next(inner)
        b = next(inner)
        yield a, b

_end = object()

def pairs_ok(it):
    inner = iter(it)
    while True:
        a = next(inner, _end)
        if a is _end:
            return
        b = next(inner, _end)
        if b is _end:
            return
        yield a, b

try:
    print(list(pairs([1, 2, 3])))
except RuntimeError as e:
    print('RuntimeError:', e)
print(list(pairs_ok([1, 2, 3])))

The first version fails on the odd leftover item; the second ends cleanly.

Two more body-level surprises are about what a generator can accept and what it gives back. First, send(value) works by making the currently paused yield expression evaluate to value. A generator that has not started has no paused yield, so sending anything other than None raises TypeError. Advance it once with next(g) (priming it) before you send. Second, return x inside a generator does not produce x as an item. It finishes the generator, and x is stored on the StopIteration exception as its value attribute, where a for loop never looks.

def acc():
    total = 0
    while True:
        total += yield total

a = acc()
try:
    a.send(5)
except TypeError as e:
    print('TypeError:', e)
next(a)
print(a.send(5), a.send(7))

def count_to(n):
    for i in range(n):
        yield i
    return 'done'

print(list(count_to(2)))
g2 = count_to(1)
next(g2)
try:
    next(g2)
except StopIteration as e:
    print('return value:', e.value)

Priming a generator, and finding the returned value.

The list contains only the yielded 0 and 1; the string 'done' appears nowhere in it. To read a return value, either catch StopIteration as above or let another generator collect it with result = yield from count_to(2).

An iterator over a container keeps a position, and that position is meaningless if the container changes shape underneath it. A dict and a set notice: they remember their size when iteration starts, and the next step raises RuntimeError: dictionary changed size during iteration. A list does not notice. The iterator just keeps an index, so removing an element shifts everything left and the next element is jumped over, with no error. Iterate over a copy, such as list(d), or build a new collection with a comprehension.

d = {'a': 1, 'b': 2}
try:
    for k in d:
        d[k + 'x'] = 0
except RuntimeError as e:
    print(e)

d = {'a': 1, 'b': 2}
for k in list(d):
    d[k + 'x'] = 0
print(sorted(d))

nums = [1, 2, 2, 3]
for n in nums:
    if n == 2:
        nums.remove(n)
print(nums)

A loud failure for the dict, a silent one for the list.

The list result is wrong: the second 2 was skipped and survived the removal. The fix is nums = [n for n in nums if n != 2], which builds a new list instead of editing the one being walked.

A generator that opens a file or a connection holds it until its frame is closed. That happens when the generator finishes, when close() is called, or when it is garbage collected. If a consumer stops early, say with break, an exception or islice, the generator is left paused with the resource still open. In CPython, dropping the last reference usually closes it right away, but a reference cycle, a stored traceback or another interpreter can delay that indefinitely. Put the resource in a with or try/finally inside the generator, then make sure the consumer closes it with contextlib.closing.

from contextlib import closing

def lines():
    print('open')
    try:
        yield 'a'
        yield 'b'
    finally:
        print('closed')

with closing(lines()) as g:
    print(next(g))
print('after with')

Leaving the with block calls g.close(), which runs the finally.

A suspended generator is not on any call stack. The call that created it returned long ago, so when a bad value finally blows up, the traceback shows the consumer's line that pulled the item, not the code that built the pipeline or fed it bad data. Errors seem to appear far from their cause. Three things help: inspect the paused frame, validate arguments eagerly in an ordinary wrapper function that returns the generator, and use getgeneratorstate to see whether you are looking at a created, suspended or closed generator.

Python gives you six ways to produce a stream of values, and they overlap a lot. They differ on a few properties: when the work happens, how much memory it holds, whether you can walk the result twice, and whether you can ask for len or an index. The table lays them out so you can pick by property instead of by habit.

A list comprehension builds the whole result immediately. That costs O(n) memory, but you get a real list: you can loop over it again, index it, sort it and call len on it. A generator expression has almost the same syntax with parentheses, but it computes one item per request and holds one frame, so memory stays flat. The price is that it can only be walked once, and it has no index and no len.

A generator function is the step up when one expression no longer fits. It can branch, keep local state, wrap work in try/finally so cleanup runs, and receive values through send. An iterator class goes further: it is lazy and stateful like the generator, but it is an ordinary object. You can give it extra methods, inspect its fields, and add a reset.

An itertools primitive such as chain, islice or accumulate is lazy, written in C, and built to be composed with other iterators. Finally, range looks lazy but behaves like a sequence. It holds O(1) memory, yet you can reuse it, index it and call len on it. It only produces integers.

r = range(0, 100, 7)
print(len(r), r[3])
print(list(range(3)), list(range(3)))

range is lazy but still indexable, sized and reusable

The most practical row in the matrix is reuse. Lists, ranges and other containers produce a fresh iterator every time a for starts, so they survive any number of passes. Generators and generator expressions are themselves iterators. Once they are exhausted they stay exhausted, and a second loop quietly sees nothing.

nums = [3, 1, 2]
doubled = (n * 2 for n in nums)
print(list(doubled))
print(list(doubled))

The second pass raises no error, it just returns nothing

Four rules cover nearly every case. Apply them in order, because the first one that fires usually settles the question. Rule 1 is about how you will use the data: if you must iterate more than once, index into it, or sort it, materialize it into a list. Sorting and reversing need every item in hand anyway, so laziness cannot help there.

Rule 2 is about the size and source of the data. If it is unbounded, huge, or streamed from I/O such as a log file or a socket, use a generator so memory stays constant. Rule 3 then chooses the form. One expression means a generator expression. Branching, state or cleanup means a generator function. A rich object with its own methods means an iterator class. Rule 4 overrides hand-rolling: if the operation already has an itertools name, use it.

Rule 4 is the one people forget. A named combinator is already written in C, already tested, and composes with the rest of your pipeline. accumulate is a running fold, chain joins iterables end to end, and groupby collects consecutive items that share a key. Notice that groupby only groups neighbours, so sort by the same key first if the data is not already ordered.

from itertools import accumulate, chain, groupby

print(list(accumulate([3, 1, 4, 1, 5])))
print(list(chain('a', 'bc')))
rows = ['apple', 'avocado', 'banana', 'blueberry', 'cherry']
print({k: list(g) for k, g in groupby(rows, key=lambda s: s[0])})

The group key is the first letter, so the last key is 'c'

When no combinator fits and the logic has state and cleanup, write a generator function. This one keeps a running total, accepts values through send, and reports when it is closed. A try/finally inside a generator is something a generator expression cannot express.

def running_total():
    total = 0
    try:
        while True:
            total += yield total
    finally:
        print('closed')

g = running_total()
next(g)
print(g.send(5))
print(g.send(7))
g.close()

Prime with next(), send values, then close to trigger cleanup

Finally, when the iterator deserves to be an object in its own right, make it a class. It can carry extra methods that a generator cannot, such as a reset.

class Countdown:
    def __init__(self, n):
        self.start = n
        self.n = n
    def __iter__(self):
        return self
    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1
    def reset(self):
        self.n = self.start

c = Countdown(3)
print(list(c))
c.reset()
print(list(c))

A domain object: lazy, stateful, and resettable

Everything in this chapter hangs off three words. An iterable is anything that can hand you a fresh iterator. An iterator is the object that actually walks, and it can only go forward once. A generator is an iterator you did not have to write, because the compiler built it from a function containing yield.

The for statement is only sugar over that protocol. It asks the iterable for an iterator once, calls next() over and over, and treats StopIteration as the quiet signal to leave the loop. Unpacking, list(), sum(), in and every comprehension drive the same machinery.

Here is the same loop written out by hand. The try/except is exactly what the interpreter does for you.

A generator function adds one more rule. yield freezes the frame, with its locals and its position, and hands a value out. next() thaws the frame right after that yield. A return closes the frame for good, and its value is not yielded. It rides out inside StopIteration.value.

def gen():
    print('a')
    yield 1
    print('b')
    return 'done'

g = gen()
print(next(g))
try:
    next(g)
except StopIteration as e:
    print('value:', e.value)

Calling gen() runs nothing; the first next() runs up to the yield.

yield is an expression, so a value can travel back in. send(x) makes the paused yield evaluate to x and then runs to the next yield. throw(exc) raises an exception at the pause point, where the generator may catch it. close() raises GeneratorExit there, so cleanup in finally runs. A brand-new generator has no paused yield to receive a value, so prime it with next(g) first.

yield from sub hands control to a subgenerator and forwards send, throw and close straight through to it. When the subgenerator returns, the expression yield from evaluates to its return value.

The bracket choice decides when the work happens. Both forms evaluate the outermost iterable immediately, while everything else, the filter and the expression, is evaluated later by a genexp and read from the surrounding names at consume time.

data = [1, 2, 3]
g = (x * 2 for x in data)
data = [10]
print(list(g))

n = 5
h = (x + n for x in range(2))
n = 100
print(list(h))

The old list is captured, but n is looked up late.

The headline win of laziness is memory. A lazy pipeline holds one frame and one item, however long the stream is, while a list holds everything. The price is per-item overhead from resuming a frame, so a tight loop over a small list can be slightly faster than the generator.

import sys
big = [x * x for x in range(100_000)]
lazy = (x * x for x in range(100_000))
print(sys.getsizeof(big) > 100_000)
print(sys.getsizeof(lazy) < 1000)

getsizeof(lazy) counts the object only, which is the point.

Three bugs account for most production surprises with generators. Each one is silent or nearly so, which is why they are worth memorising.

Before you write a loop, check whether itertools already names it. Those primitives are implemented in C, are already tested, and stay lazy.

Stacked genexps make a constant-memory ETL pipeline. Each stage pulls exactly one item from the stage below, so the same shape works over a multi-gigabyte log file or an unbounded network stream.

import io
log = io.StringIO('ok 12\nerr 7\nok 30\nok x\n')
rows = (line.split() for line in log)
good = (r for r in rows if r[0] == 'ok' and r[1].isdigit())
total = sum(int(r[1]) for r in good)
print(total)

Swap the StringIO for open('big.log') and nothing else changes.

When a generator misbehaves, look at where it is paused. inspect.getgeneratorstate(g) reports GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED or GEN_CLOSED. While it is suspended, g.gi_frame.f_locals shows its live variables. A closed generator has no frame, so inspect it before you close it.

Tracebacks from a paused generator show the consumer, not the definition, so errors appear where the value was pulled. The state and locals above are the quickest way back to the cause.

Try to answer each question in your head before you open the answer. They test whether you can predict what the interpreter does, not whether you remember the wording.

• An iterable has __iter__, an iterator has __iter__ and __next__, and a generator is an iterator the compiler wrote for you.
• for is sugar for iter(), repeated next(), and stopping on StopIteration. Unpacking, list(), sum() and comprehensions use the same protocol.
• yield freezes a frame and next() thaws it. return closes the generator and parks its value in StopIteration.value.
• send, throw and close make a generator a two-way coroutine, prime it with next() first, and yield from forwards all three to a subgenerator.
• A genexp is lazy and one-shot, a list comprehension is eager and reusable, and in both the outermost iterable is evaluated immediately.
• Lazy code costs O(1) memory and gives a fast first result, but you pay per-item overhead, and you lose len, indexing and a second pass.
• The classic bugs are reusing an exhausted generator, late-binding closures, and StopIteration leaking out of a generator body (PEP 479).
• Name it before you build it: reach for itertools such as islice, chain, groupby, accumulate and batched before writing your own loop.