| name | python-idioms |
| description | Teaches expert-level Python idiomatic patterns: comprehensions, generators, dataclasses, protocols, structural pattern matching, and context managers. Focuses on writing Pythonic code that leverages the language strengths rather than translating patterns from other languages.
Use when the user asks about Pythonic patterns, list comprehensions, generators, dataclasses vs named tuples, protocols, pattern matching, context managers, or idiomatic Python code style.
Do NOT use when the user asks about project setup (use `python-project-setup`), testing (use `python-testing-patterns`), async programming (use `python-async-patterns`), or type system features (use `python-type-system`).
|
| license | Apache-2.0 |
| metadata | {"author":"foundry-skills","version":"1.0.0","tags":"python best-practices clean-code","category":"software-engineering","subcategory":"languages-runtimes","depends":"","disclaimer":"none","difficulty":"intermediate"} |
Python Idioms
When to Use
Use this skill when:
- User asks how to write more "Pythonic" code or wants a code review focused on idiomatic style
- User is translating patterns from Java, C++, or JavaScript and the result feels wrong in Python (e.g., explicit index loops, abstract base class hierarchies for everything, factory class patterns)
- User asks specifically about comprehensions, generator expressions, or lazy evaluation
- User is choosing between
@dataclass, NamedTuple, TypedDict, or plain dicts for a data container
- User wants to define a structural interface using
Protocol instead of inheritance
- User asks about
match/case structural pattern matching (Python 3.10+)
- User wants to write or understand context managers,
contextlib utilities, or the with statement
- User asks about EAFP vs LBYL,
collections.defaultdict, itertools, or functools patterns
- User asks about the walrus operator (
:=), starred assignment, or other syntactic features
- User wants to understand when to use
__slots__, __repr__, __eq__, or other dunder methods idiomatically
Do NOT use this skill when:
- User wants to create a new Python project with packaging, virtual environments, or pyproject.toml -- use
python-project-setup
- User is asking about pytest fixtures, test doubles, or test organization -- use
python-testing-patterns
- User wants
asyncio, async/await, TaskGroup, or concurrent programming -- use python-async-patterns
- User is asking about Pydantic, marshmallow, attrs, or runtime data validation with serialization -- use
python-data-modeling
- User wants advanced type system features like
TypeVar, ParamSpec, Concatenate, TypeGuard, or variance -- use python-type-system
- User wants error handling architecture (custom exception hierarchies, error domains, retry patterns) -- use
python-error-handling
- User wants profiling, caching, or algorithmic performance optimization -- use
python-performance
- User is asking about Python packaging, distributions, or dependency management -- use
python-project-setup
Process
1. Identify the Pattern Category
Classify the user's request into one or more of these pattern domains. Each has a distinct decision framework:
- Collection transformation: comprehensions, generator expressions,
itertools, functools.reduce
- Data containers:
@dataclass, NamedTuple, TypedDict, enum, and when to use each
- Structural interfaces:
Protocol, duck typing, structural subtyping vs. inheritance
- Complex branching:
match/case structural pattern matching
- Resource management: context managers,
contextlib, __enter__/__exit__
- Attribute and key access: EAFP,
dict.get, getattr, collections.defaultdict, operator.attrgetter
- Iteration control:
itertools, enumerate, zip, zip_strict, chain, islice
- Callable patterns:
functools.partial, functools.cache, functools.wraps, closures
Ask yourself: is the user trying to produce data, shape data, define contracts, branch on structure, manage a lifecycle, or consume data? The answer determines which section of this skill to apply.
2. Apply the Right Comprehension or Generator Pattern
Comprehensions are for pure transformations without side effects. The moment a loop body has a side effect (logging, mutating external state, I/O), use a for loop instead.
- List comprehension -- use when the full result is needed immediately and fits in memory. Rule of thumb: < 10,000 elements or the result is indexed multiple times.
squares = [x * x for x in range(100)]
- Generator expression -- use when the result is consumed once, may be large, or feeds another iterator. Generators have O(1) memory regardless of input size.
total = sum(x * x for x in range(10_000_000))
- Dict comprehension -- when constructing a mapping from an iterable. Prefer over
dict(zip(...)) when transformation logic is involved.
word_lengths = {word: len(word) for word in corpus}
- Set comprehension -- when deduplication and membership testing are needed. Not order-preserving.
unique_domains = {email.split("@")[1] for email in emails}
- Nested comprehensions -- only one level of nesting is acceptable in idiomatic Python. Two-level nesting is the hard limit.
flat = [cell for row in matrix for cell in row]
- Generator function with
yield -- when the generation logic is too complex for an expression, involves try/except inside the loop, or needs to maintain state across yields.
def () -> Generator[, , ]:
file_path.() f:
chunk := f.read(size):
chunk
3. Select the Right Data Container
Apply this decision tree precisely -- the wrong container choice is one of the most common sources of non-Pythonic Python.
Step A -- Is the data mutable or immutable?
- Immutable with no behavior: consider
NamedTuple (tuple semantics, zero-overhead slots) or @dataclass(frozen=True) (hashable, usable as dict key or set member)
- Mutable with behavior: use
@dataclass
Step B -- Does the data need runtime validation?
- If yes, refer to
python-data-modeling for Pydantic -- do not add validation inside __post_init__ beyond simple invariant checks
Step C -- Will this be used as a dict (JSON-compatible) or typed positionally?
- If typed dict structure for JSON interop:
TypedDict -- allows optional keys via total=False or Required/NotRequired annotations
- If positional tuple access matters:
NamedTuple
Step D -- Is this an enumerated constant set?
- Integers with no string meaning:
enum.IntEnum
- String values for serialization:
enum.StrEnum (Python 3.11+) or str, enum.Enum mixin
- Auto-generated values:
enum.auto()
Full selection table:
| Requirement | Container | Key Feature |
|---|
| Mutable fields, type hints, defaults | @dataclass | Generated __init__, __repr__, __eq__ |
| Immutable, hashable value object | @dataclass(frozen=True) | Usable as dict key, set member |
| Lightweight tuple with named fields | NamedTuple | Tuple unpacking, indexing, zero overhead |
| JSON-like data, optional keys | TypedDict | total=False, structural dict typing |
| String constants for serialization | enum.StrEnum | str subclass, auto() gives lowercased name |
| Ordered integer constants | enum.IntEnum | Comparable to int literals |
| Namespace of related functions | types.SimpleNamespace or module | Rarely a class, avoid over-engineering |
Dataclass field patterns to know:
from dataclasses import dataclass, field
from typing import ClassVar
@dataclass
class OrderBatch:
orders: list["Order"] = field(default_factory=list)
max_size: ClassVar[int] = 500
_cache: dict = field(default_factory=dict, repr=False, compare=False)
4. Define Structural Interfaces with Protocol
Protocol is the idiomatic way to express "any object that supports these methods" without requiring inheritance. Use it when:
- You cannot modify the third-party class to inherit from your ABC
- You want structural (duck) typing rather than nominal typing
- You want a minimal interface -- define only what the consumer actually calls
from typing import Protocol, runtime_checkable
class Closeable(Protocol):
def close(self) -> None: ...
class Readable(Protocol):
def read(self, n: int = -1) -> bytes: ...
Key decisions:
Protocols vs ABCs:
- Use
Protocol for behavior-based contracts (what it can do)
- Use
ABC only when you want shared implementation via super() in subclasses, or when nominal inheritance is intentional (e.g., a plugin system where issubclass checks matter)
5. Apply Structural Pattern Matching
match/case (Python 3.10+) is not a switch statement -- it matches against the structure, type, and values of objects simultaneously. Use it when you have 3 or more branches that discriminate on type, shape, or a combination.
When to use match:
- Dispatching on command/event type with attribute destructuring
- Processing AST nodes or recursive data structures
- Implementing state machines where transitions depend on both state and event shape
- Handling API response variants (success/error with different payloads)
When NOT to use match:
- Simple if/elif on a single boolean or integer flag -- match adds syntax overhead without clarity gain
- When all cases do the same thing to different values -- use a dict dispatch instead
- Python 3.9 or earlier target environments
Pattern types:
match event:
case {"type": "login", "user_id": int(uid)}:
handle_login(uid)
case Order(status="pending", total=t) if t > 1000:
flag_for_review(event)
case [first, *rest] if len(rest) > 5:
batch_process(first, rest)
case {"status": "cancelled" | "refunded"}:
handle_terminal(event)
case Order(order_type=ot) if ot not in ALLOWED_TYPES:
raise ValueError(f"Unsupported order type: {ot}")
case _:
log_unhandled(event)
Class patterns require that the class declares __match_args__ (dataclasses do this automatically) or uses keyword patterns. Do NOT rely on positional class patterns for classes that do not define __match_args__.
6. Write Idiomatic Context Managers
Every resource with an explicit lifecycle (file handles, locks, network connections, database transactions, temporary directories) should be managed with a context manager. Never leave cleanup to the garbage collector.
contextlib.contextmanager -- the 90% solution:
from contextlib import contextmanager
from typing import Generator
@contextmanager
def managed_cursor(conn: Connection) -> Generator[Cursor, None, None]:
cursor = conn.cursor()
try:
yield cursor
conn.commit()
except Exception:
conn.rollback()
raise
finally:
cursor.close()
Rules for contextmanager:
- Code before
yield is __enter__ -- do NOT yield inside a bare try without finally
- Code after
yield in finally always runs -- this is your cleanup
- Re-raise exceptions after rollback -- do not swallow them silently
- Yield exactly one value (the resource) or
None
contextlib.suppress -- replace empty except blocks:
try:
os.remove(tmp_path)
except FileNotFoundError:
pass
from contextlib import suppress
with suppress(FileNotFoundError):
os.remove(tmp_path)
contextlib.ExitStack -- dynamic resource composition:
from contextlib import ExitStack
def process_files(paths: list[Path]) -> None:
with ExitStack() as stack:
handles = [stack.enter_context(p.open()) for p in paths]
process(handles)
Custom __enter__/__exit__ class -- use only when the context manager needs its own state, supports multiple entry, or has complex teardown that does not map cleanly to the linear try/yield/finally model:
class Timer:
def __enter__(self) -> "Timer":
self._start = time.perf_counter()
return self
def __exit__(self, *exc_info: object) -> bool:
self.elapsed = time.perf_counter() - self._start
return False
7. Apply EAFP and Idiomatic Attribute/Key Access
Python's EAFP (Easier to Ask Forgiveness than Permission) is idiomatic because exceptions in CPython are fast when not raised -- the try block has near-zero overhead in the happy path.
Key patterns:
if hasattr(obj, "process") and callable(obj.process):
obj.process(data)
try:
obj.process(data)
except AttributeError:
fallback(data)
if "key" in config and config["key"] is not None:
value = config["key"]
else:
value = default
value = config.get("key") or default
value = config.get("key", default)
from collections import defaultdict
groups: defaultdict[str, list[str]] = defaultdict(list)
for item in items:
groups[item.category].append(item.name)
host = getattr(config, "database", None) and getattr(config.database, "host", "localhost")
When LBYL IS appropriate:
- When the exception path would be extremely common (> ~10% of calls) and performance is measured to matter
- When the check is semantically meaningful ("does this file exist before I tell the user?")
- When the side effect of attempting the operation is irreversible
8. Apply itertools and functools Idioms
These are the standard library's idiomatic building blocks for iteration and callable manipulation. Do not reinvent them.
itertools patterns to know:
import itertools
flat = list(itertools.chain.from_iterable(nested))
def windows(seq, n):
iterators = itertools.tee(seq, n)
for i, it in enumerate(iterators):
next(itertools.islice(it, i, i), None)
return zip(*iterators)
for key, group in itertools.groupby(sorted_data, key=lambda x: x.category):
process_group(key, list(group))
roundrobin = itertools.cycle(["a", "b", "c"])
first_10 = list(itertools.islice(expensive_generator(), 10))
pairs = list(itertools.product(colors, sizes))
functools patterns:
import functools
@functools.cache
def fib(n: int) -> int:
return n if n < 2 else fib(n - 1) + fib(n - 2)
@functools.lru_cache(maxsize=256)
def fetch_config(env: str) -> Config: ...
log_error = functools.partial(log, level="ERROR", service="api")
def my_decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
combined = functools.reduce(lambda acc, x: acc | x, list_of_dicts, {})
Output Format
When delivering idiomatic refactoring, always present code in this structure:
For pattern selection decisions, use this decision table:
## Pattern Decision: [Category]
| Situation | Pattern | Why |
|-----------|---------|-----|
| [specific situation] | [specific pattern] | [specific reason] |
For multi-pattern refactors (the common case), group by layer:
## Refactoring Plan
### Layer 1 -- Data Modeling
[data container selection + code]
### Layer 2 -- Business Logic
[comprehensions, generators, pattern matching + code]
### Layer 3 -- Resource Management
[context managers + code]
### Layer 4 -- Interface Definition
[Protocol definitions + code]
Always include a "What Changed and Why" summary after the code block, with numbered points that reference specific Python semantics -- not just style preferences.
Rules
-
Never use a mutable default argument. def f(items=[]) shares the same list object across all calls. Use None with an explicit if items is None: items = [] body, or field(default_factory=list) in a dataclass. This is the most common Python bug in new code.
-
Never nest comprehensions more than two levels deep. Two-level nesting ([x for row in matrix for x in row]) is the absolute maximum. Three-level nesting is always unreadable and must be extracted to a named generator function.
-
Never use type(x) == SomeType for type checking. Use isinstance(x, SomeType) to respect subclasses, or structural Protocol checks. type() equality breaks with subclasses and is almost never what the author intended.
-
Never use a bare except: or except Exception: that swallows the exception silently. Empty except blocks hide bugs. Either re-raise, log and re-raise, or use contextlib.suppress with a specific exception type for intentional suppression.
-
Never use @dataclass when NamedTuple is sufficient. If the data is immutable, positionally meaningful, and needs no behavior beyond field access, NamedTuple provides tuple unpacking, is faster to construct, uses less memory, and requires no import from dataclasses. Only use @dataclass(frozen=True) when you need inheritance or non-positional defaults.
-
Never put business logic or I/O in __init__ or __post_init__. These methods are for field initialization and lightweight invariant checks only. Use a @classmethod factory (e.g., from_dict, from_file) for construction with side effects or complex parsing.
-
Always use str.join() for building strings from iterables. "".join([...]) is O(n). String concatenation in a loop is O(n^2) due to repeated allocation. This is one of the few Python idioms with a hard performance reason, not just style.
-
Always use pathlib.Path for all filesystem operations. , , and string-based path manipulation are legacy. objects compose with , expose , , , , , and cleanly. New code should never import .
Edge Cases
Legacy Codebase Without Type Hints or Tests
Do not introduce dataclasses, Protocol, or pattern matching until the changed code has test coverage. The safest migration order is:
- Add type hints to existing functions first (zero behavior change)
- Replace raw dict returns with
TypedDict (structural change only, fully backward compatible)
- Replace
dict-based objects with NamedTuple for read-only data (tuple indexing still works)
- Introduce
@dataclass only for new code or when a class is being rewritten anyway
Never rewrite an entire module of dict-based data structures to dataclasses in a single PR -- the diff is unreviable and the risk is high.
Python 3.9 or 3.10 Target Environment
Several idioms in this skill require specific Python versions:
match/case: Python 3.10+ only. For 3.9, use if/elif with isinstance chains
enum.StrEnum: Python 3.11+. For 3.10, use class Color(str, enum.Enum)
functools.cache: Python 3.9+. For 3.8, use functools.lru_cache(maxsize=None)
zip(..., strict=True): Python 3.10+
X | Y type union syntax in annotations: Python 3.10+. For 3.9, use from __future__ import annotations at the top of the file to defer evaluation, which makes the syntax valid even on 3.9
Always confirm the target Python version before recommending 3.10+ features. Check python_requires in pyproject.toml if the project has one.
Team Unfamiliar with These Patterns
When the user is introducing idioms to a team that primarily knows procedural Python:
- Introduce one pattern category per sprint, not all at once
- The order of easiest adoption: comprehensions →
enumerate/zip → pathlib → dataclasses → contextlib → Protocol → pattern matching
- Write a one-page team ADR (Architectural Decision Record) for each adopted pattern explaining why it was chosen
- Avoid walrus operator (
:=), starred assignment in complex positions, and chained itertools calls in shared code until the team is comfortable -- these have high cognitive load and fail code review silently when misread
match/case is often the most controversial -- demonstrate it on a known-complex if/elif chain before proposing it for new code
Interoperability with C Extensions and Cython
Certain idiomatic Python patterns prevent Cython optimization and native extension interop:
- Generators and generator expressions cannot be typed with Cython's
cdef -- use explicit typed loops in Cython source files
@dataclass with __eq__ and __hash__ prevents Cython's struct-like memory layout; prefer C structs via ctypes.Structure for performance-critical data exchange
Protocol isinstance checks work at Python level but have no meaning to C extensions -- use explicit type tags or discriminated unions for C FFI boundaries
- If a Python class will be subclassed in Cython, use
__slots__ to prevent arbitrary attribute addition, which breaks Cython's static analysis
__slots__ and Dataclass Interactions
__slots__ dramatically reduces memory per-instance (from ~200 bytes per dict to ~50 bytes per fixed slot) and is worth adding when creating millions of small objects. With dataclasses:
@dataclass
class Point:
__slots__ = ("x", "y")
x: float
y: float
@dataclass(slots=True)
class Point:
x: float
y: float
Slots interact badly with multiple inheritance -- two classes with disjoint __slots__ cannot be easily combined. Do not add slots to classes intended for mixin-heavy inheritance.
Generator Exhaustion Bugs
Generators are one-time iterators. The most common mistake is iterating a generator twice:
def even_numbers(n):
return (x for x in range(n) if x % 2 == 0)
evens = even_numbers(100)
print(list(evens))
print(list(evens))
Diagnose generator exhaustion bugs by checking if list(gen) on second call returns []. When a generator must be reusable, wrap it in a class with __iter__ returning a fresh generator, or document clearly that it is single-use.
Overuse of Pattern Matching
match/case is sometimes applied where a dict dispatch table is cleaner and faster:
match command:
case {"action": "create", "payload": dict(payload)}:
create(payload)
case {"action": "delete", "id": int(record_id)}:
delete(record_id)
HANDLERS = {
"create": handle_create,
"delete": handle_delete,
"update": handle_update,
}
handler = HANDLERS.get(command["action"])
if handler is None:
raise ValueError(f"Unknown action: {command['action']}")
handler(command)
Use pattern matching for structural discrimination (different shapes require different destructuring). Use dict dispatch for behavioral dispatch (same shape, different function to call).
Example
Input: "I have a function that reads transaction records from multiple CSV files, filters for transactions over $100, groups them by merchant category, and computes the total per category. The code uses nested for loops, lots of if key not in dict checks, and builds everything into plain dicts. How do I make it Pythonic?"
Full Worked Output:
import csv
import os
def process_transactions(file_paths):
all_transactions = []
for file_path in file_paths:
f = open(file_path, "r")
reader = csv.DictReader(f)
for row in reader:
all_transactions.append(row)
f.close()
filtered = []
for t in all_transactions:
if float(t["amount"]) > 100:
filtered.append(t)
grouped = {}
for t in filtered:
category = t["merchant_category"]
if category not in grouped:
grouped[category] = []
grouped[category].append(float(t["amount"]))
totals = {}
for category, amounts in grouped.items():
total = 0
for amount in amounts:
total = total + amount
totals[category] = total
return totals
from __future__ import annotations
import csv
import itertools
from collections import defaultdict
from contextlib import ExitStack
from dataclasses import dataclass
pathlib Path
typing Generator
:
merchant_category:
amount:
() -> Transaction:
cls(
merchant_category=row[].strip(),
amount=(row[]),
)
() -> Generator[Transaction, , ]:
ExitStack() stack:
handles = [
stack.enter_context(path.(newline=, encoding=))
path paths
]
rows = itertools.chain.from_iterable(
csv.DictReader(handle) handle handles
)
row rows:
:
Transaction.from_row(row)
(KeyError, ValueError):
() -> [, ]:
paths = [Path(p) p file_paths]
transactions = _read_transactions(paths)
qualifying = (t t transactions t.amount > min_amount)
totals: defaultdict[, ] = defaultdict()
t qualifying:
totals[t.merchant_category] += t.amount
(totals)
What Changed and Why:
-
@dataclass(frozen=True) replaces plain dict for transaction data. The fields merchant_category and amount are now typed, named, and immutable. frozen=True makes Transaction hashable, so it could be used in sets or as dict keys if needed. The from_row classmethod separates parsing from construction -- __init__ never reads I/O or raises ValueError directly.
-
ExitStack + list comprehension replaces open/close pairs. Every file handle opened in the list comprehension is registered with the stack, guaranteeing cleanup even if csv.DictReader construction raises on one file. The original code leaked the file handle if an exception occurred between open and close.
-
itertools.chain.from_iterable flattens the per-file row streams lazily. No intermediate list of all rows is built. For 50 files with 100,000 rows each, the original code held 5 million dicts in memory; this approach holds at most one row at a time.
-
Generator expression for filtering (qualifying) avoids a second pass and a second list. The pipeline is read → filter → accumulate in a single linear traversal.
-
defaultdict(float) eliminates the if key not in grouped pattern. defaultdict(float) initializes any missing key to 0.0 on first access, so totals[category] += amount works unconditionally. This replaces four lines with one.
-
sum() was implicit in defaultdict accumulation -- += amount is clearer than an inner loop and more efficient than calling sum(amounts) on a pre-built list.
-
pathlib.Path replaces bare string paths. path.open() is equivalent to open(str(path)) but integrates with the type system and avoids string manipulation for any path operations that might be added later (e.g., path.stem, ).