Iterators and Generators

7 pages · ~5 min✓ Reviewed

Builds on Errors and the Standard Library.

PythonPage 1

Iterables, Iterators and the Protocol

1. Iterable vs Iterator

	Iterable	Iterator
Has	`__iter__()`	`__next__()` and `__iter__()`
`__iter__()` returns	a new iterator	self
Examples	list, tuple, dict, str	file object, `iter([1,2])` (list_iterator)

2. What a for loop does

iter(obj)called once
next()repeated
StopIterationloop ends

3. Exhaustion and defaults

An iterator is single-pass and stateful
When exhausted, next() keeps raising StopIteration
next(it, default) returns default instead of raising

python

it = iter([1, 2])
print(type(it).__name__)
print(next(it), next(it))
print(next(it, "done"))
print(next(it, "done"))

output

list_iterator
1 2
done
done

Reusable vs one-shot

A list can be looped again; its iterator cannot. A for over an exhausted iterator runs zero times with no error.

PythonPage 2

Generators and yield

1. Generator basics

A function with yield returns a generator object when called
The body does not run until the first next()
yield pauses and keeps locals; the next next() resumes right after it

2. Finishing and delegating

Falling off the end or a bare return raises StopIteration
The return value becomes StopIteration.value
yield from sub delegates to a sub-iterable, passes its return value back

python

def gen():
    yield 1
    return "done"
g = gen()
print(next(g))
try:
    next(g)
except StopIteration as e: print(e.value)

return value rides on StopIteration

output

1
done

3. send and close

Call	Effect
`next(g)`	resumes; `yield` evaluates to None
`g.send(x)`	paused `yield` evaluates to x
`g.close()`	raises `GeneratorExit` inside

Prime first

Call next(g) once before send(x); a non-None send to a fresh generator raises TypeError.

PythonPage 3

Generator Expressions and Lazy Evaluation

1. Generator vs list

Round brackets build a generator, not a list
As the sole argument, drop the extra parentheses

	[x*x for ...]	(x*x for ...)
Builds	everything now	one item at a time
Memory	O(n)	O(1)
Reuse	any number of times	one pass

python

data = [3, -1, 4]
g = (x*x for x in data if x > 0)
print(type(g).__name__)
print(sum(x*x for x in data if x > 0))

output

generator
25

2. Lazy pipeline

Sourceyields one item
Filterkeeps or drops
Transformx*x
Consumerpulls the next

3. One-shot traps

Second loop over a spent generator yields nothing
len(g) and g[0] raise TypeError
Why a generator for a huge file? Items on demand, O(1) memory
Why empty the second time? Exhausted iterator: rebuild it or use a list

Remember

Need it twice, or need len() or indexing? Use a list.

PythonPage 4

itertools Essentials

1. Streams and slices

count(start, step): endless counter
cycle(it) repeats forever; repeat(x, n) gives x, n times
Stop them with islice or break
islice(it, stop) or islice(it, start, stop, step): lazy slice
No negative indices

python

from itertools import count, islice
print(list(islice(count(10, 5), 3)))

output

[10, 15, 20]

2. Join and group

chain(a, b) joins without copying
chain.from_iterable(lists) flattens one level
groupby groups only consecutive equal keys
Sort by the same key first

3. Combinatorics

Call	Order	Gives
`product(a, b)`	nested loops	every pair
`combinations(x, r)`	ignored	AB, AC, BC
`permutations(x, r)`	respected	AB, BA, AC...
`zip_longest(a, b)`	pads shorter	fillvalue

Gotcha

Unsorted input makes groupby repeat keys: one group per run.

Why sort before groupby? Only adjacent equal keys merge.
combinations vs permutations? Order ignored vs respected.

PythonPage 5

Context Managers and with

1. The with Protocol

with obj as x: calls enter() and binds its return value to x
Always calls exit(exc_type, exc, tb) on the way out
Truthy return from __exit__ suppresses the exception
None or False lets it propagate

2. @contextmanager

@contextlib.contextmanager turns a generator with one yield into a manager
Wrap yield in try/finally so cleanup runs on errors

python

from contextlib import contextmanager
@contextmanager
def cm():
    res = setup()
    try:
        yield res
    finally:
        cleanup()

the yielded value becomes the as target

No try/finally

The body's exception is raised at yield, so cleanup code after it is skipped.

3. Everyday Helpers

Helper	Does
`suppress(Err)`	ignores that exception
`closing(obj)`	calls `obj.close()` on exit
`ExitStack`	dynamic number of managers

with a, b: enters left to right, exits in reverse

PythonPage 6

Cheat Sheet: Iterators, Generators, itertools, with

1. Protocol and the Three Forms

iter(x) returns an iterator
next(it) advances one step
StopIteration marks the end; next(it, d) returns d instead

	Syntax	Evaluation	Reuse
Generator function	`yield` in a `def`	lazy	one-shot
Generator expr	`(x for x in d)`	lazy	one-shot
List comp	`[x for x in d]`	eager	reusable

2. itertools and with

Call	Gives
`islice(count(), 5)`	first 5 of an infinite stream
`chain(a, b)`	a then b, no copy
`groupby(sorted(d, key=k), key=k)`	runs of equal keys
`product(a, b)`	nested-loop pairs

python

@contextmanager
def cm():
    res = setup()
    try:
        yield res
    finally:
        cleanup()

bind res first, then yield inside try

3. Top Gotchas

Exhausted generator is empty on the second pass
groupby needs sorted input, by the same key
Bare yield without try/finally leaks cleanup

Cleanup

Put yield inside try and cleanup in finally, so it runs even when the with body raises.

PythonPage 7

Check yourself

What does this print?

6 0
A generator is one-shot; the second sum finds it exhausted

g = (x for x in [1, 2, 3])
print(sum(g), sum(g))

What does [k for k, _ in groupby('aabaa')] give?

['a', 'b', 'a']
groupby groups only consecutive keys; sort first to merge them

A @contextmanager runs print('closed') after a bare yield. The with body raises. Is 'closed' printed?

No, the exception is thrown at the yield and skips the cleanup
Wrap the yield in try/finally