optimize-python

name

optimize-python

description

Python performance profiling and optimization. MUST be invoked when investigating CPU hotspots, memory usage, I/O slowness, async event loop blocking, or when optimizing slow Python code. Covers tools (cProfile, line_profiler, austin/speedscope, scalene), flamegraph generation, common optimization patterns (concurrent awaits, regex combining, recursive memoization, eliminating duplicate work, moving blocking I/O off the event loop), and profiling Python processes in Kubernetes pods.

Optimizing Python

The Profiling Loop

1. Pick a profiler based on what you're investigating: cProfile or austin for CPU hotspots, scalene when you also need memory, line_profiler when you've found the hot function and want line-level detail, PYTHONASYNCIODEBUG=1 for event loop blocking.
2. Profile a realistic workload, not a toy input if the real bottleneck only appears at scale.
3. Read the output (using the helper scripts if necessary) to get a ranked summary. Focus on self-time (where CPU actually burns), not inclusive time.
4. Pick a few things to fix: don't try to fix everything at once. Start with the highest self-time hotspot that looks addressable.
5. Fix and measure again: profile with the same workload to confirm the improvement. Speedups that don't show up in the profiler didn't happen. That said, it's fine to rework something into the clearly correct shape, even if it doesn't move the benchmark. Focus on the biggest things, but not at the total expense of the small ones.
6. Iterate until performance is acceptable, or the remaining hotspots are outside your control (C extensions, network, OS).

Tools

cProfile (built-in, call-level CPU)

python -m cProfile -o profile.out myscript.py
# sort by cumulative time:
python -c "import pstats; p = pstats.Stats('profile.out'); p.sort_stats('cumulative'); p.print_stats(30)"

For other ways to visualize .prof/.out files (GUI browsers, SnakeViz, gprof2dot, etc.), see the Python profiling docs and check what's available in the environment.

line_profiler (line-level CPU)

uv add --dev line-profiler

Decorate functions you want to drill into with @profile (no import needed when running via kernprof), then:

kernprof -l -v myscript.py
# -l: line-level profiling, -v: print results immediately
# results also saved to myscript.py.lprof; view later with:
python -m line_profiler myscript.py.lprof

austin + speedscope (sampling CPU → flamegraph)

austin is a statistical/sampling profiler that attaches to a running Python process or launches one. It produces output that can be converted to speedscope format for interactive flamegraphs. See the austin README for full details; run austin --help to see all flags.

uv add --dev austin-python  # installs austin-python converter tools
# also need austin itself: brew install austin / apt install austin / cargo install austin

Profile a script and produce a flamegraph:

The pipeline depends on the austin version, so check first:

austin --version

Austin >= 4: writes binary MOJO format by default, requires a two-step conversion:

austin -i 100 -o austin.mojo python myscript.py
uvx --from austin-python mojo2austin austin.mojo austin.collapsed
uvx --from austin-python austin2speedscope austin.collapsed austin.json

Austin < 4: writes collapsed stack text directly, one step:

austin -i 100 -o austin.collapsed python myscript.py
uvx --from austin-python austin2speedscope austin.collapsed austin.json

Then open austin.json in speedscope: load the file, or run npx speedscope austin.json.

Key austin flags:

• -i <microseconds>: sampling interval (default 100; lower = more detail, more overhead)
• -C: include child processes
• -t: terminate austin when the target exits (usually what you want for scripts)
• -p <pid>: attach to running process instead of launching one

Attaching to a running process (useful for servers/services):

austin -p $(pgrep -f myserver.py) -x 30 -o austin.out
# then apply the version-appropriate conversion pipeline above

scalene (CPU + memory + GPU, line-level)

scalene is a high-detail profiler that simultaneously tracks CPU time, memory allocation, and GPU (if available) at the line level. Good when you need both CPU and memory in one pass.

uv add --dev scalene
scalene --json --outfile profile.json myscript.py

Scalene distinguishes Python time vs. native/C time per line, which is useful for finding where numpy/pandas/etc. are spending time.

Profiling in Kubernetes

To profile a Python process running in a k8s pod, read references/k8s-pod-profiling.md for the full workflow: finding the pod, process discovery, running austin with kubectl exec, and retrieving the trace. The file transfer step has important gotchas (kubectl exec stdout has documented binary corruption; kubectl cp times out on larger files); the reference covers the recommended Python HTTP server approach and a text-only fallback.

Analyzing Profile Output Programmatically

Speedscope JSON and scalene JSON are large and dense: don't try to read them directly. Use the helper scripts in ${CLAUDE_SKILL_DIR}/scripts/ to extract actionable summaries. Always invoke them with uv run.

Speedscope summary (profile_speedscope.py)

Parses a speedscope JSON and reports self-time and inclusive-time per function, filtered to user code by default (stdlib/frozen frames hidden unless --all).

uv run ${CLAUDE_SKILL_DIR}/scripts/profile_speedscope.py austin.json
uv run ${CLAUDE_SKILL_DIR}/scripts/profile_speedscope.py austin.json --all        # include stdlib
uv run ${CLAUDE_SKILL_DIR}/scripts/profile_speedscope.py austin.json --top 20     # more entries
uv run ${CLAUDE_SKILL_DIR}/scripts/profile_speedscope.py austin.json --chains     # show top call chains

Output: ranked tables of self-time (where CPU actually burns) and inclusive-time (what called the hot code), with file:line references.

Scalene summary (profile_scalene.py)

Parses a scalene JSON and reports CPU breakdown (Python vs native/C) and memory per function and per line.

uv run ${CLAUDE_SKILL_DIR}/scripts/profile_scalene.py scalene-profile.json
uv run ${CLAUDE_SKILL_DIR}/scripts/profile_scalene.py scalene-profile.json --top 20
uv run ${CLAUDE_SKILL_DIR}/scripts/profile_scalene.py scalene-profile.json --memory  # sort by memory

Output: function-level and line-level tables with P/C bars (Python vs native CPU %), average memory footprint, and async await statistics if present.

Async / Event Loop Profiling

Detecting hidden blocking I/O

Blocking file I/O inside async code is a major source of event loop stalls, especially in cloud environments with network-backed volumes (e.g., NFS, EFS, GCS FUSE) where filesystem calls that are instant locally can take tens or hundreds of milliseconds. This is often invisible in local dev and only surfaces in production.

This matters most in servers, where blocking the event loop delays all other requests (responsiveness + throughput both suffer). In scripts, the event loop usually isn't handling concurrent requests, so blocking I/O hurts throughput but not responsiveness; it's still worth fixing, but the priority is different.

Detection options:

• Set PYTHONASYNCIODEBUG=1 before running; it logs a warning for any coroutine that blocks the loop for more than 100ms. Makes asyncio significantly slower, so only use it for debugging, not production or benchmarks. Equivalent in code: asyncio.get_event_loop().set_debug(True).
• aiomonitor or aiodebug for runtime loop inspection
• py-spy with --threads can show what threads are blocked on

Fix: move blocking work off the event loop with asyncio.to_thread:

import asyncio

# Before: blocks the event loop
def read_file(path):
    with open(path) as f:
        return f.read()

# After: runs in a thread pool, event loop stays free
async def read_file_async(path):
    return await asyncio.to_thread(read_file, path)

asyncio.to_thread helps when the blocking work can drop the GIL internally (file I/O, network calls via C extensions, etc.). It won't improve raw throughput if the work is CPU-bound Python, but it keeps the event loop responsive, which is the main win in servers. In scripts, prefer fixing throughput directly (concurrent awaits, batching) rather than reaching for asyncio.to_thread.

Common Optimization Patterns

A checklist of things to actively look for during a profiling pass. The fixes are standard; the value of this list is making sure each one gets checked.

General patterns

• Sequential awaits of independent coroutines: look for await inside a for loop where iterations don't depend on each other; run them concurrently with asyncio.gather or asyncio.TaskGroup instead.
• Multiple regex passes over the same string: combine patterns with |; when replacements differ, use one compiled pattern plus a dict-driven replacement function.
• Repeated work in recursion: hoist computations that are the same at every level into a public wrapper and pass results down as parameters; also watch for children re-doing work a parent already did (e.g. re-parsing a value parsed one level up).
• Re-parsing the same data: parse once and pass the parsed object around. Don't call response.json() or response.text repeatedly; some HTTP clients re-decode on every access.
• Look-before-you-leap double lookups: membership check followed by access does the lookup twice; do the access once and handle the miss (.get(), try/except).
• Repeated expensive calls with the same args: functools.cache / functools.lru_cache for pure functions.
• Reconstructing expensive objects per-request: common culprits are pydantic-settings BaseSettings subclasses (env var reads + validation on every instantiation), HTTP clients, connection pools, and compiled regexes. Preferred fix is manual dependency injection: take the object as an argument and construct it once at application startup (e.g. a FastAPI lifespan). If DI isn't practical, @cache on a no-arg factory is a reasonable fallback.

Hot-loop micro-optimizations

These matter only in genuinely hot inner loops; don't apply them indiscriminately.

• Membership tests on lists are O(n) per test; build a set once before the loop.
• String building with += is quadratic for large outputs; accumulate parts in a list and "".join once.
• Repeated attribute/global lookups resolve on every iteration; bind to a local before the loop.
• Raising exceptions is expensive (a no-exception try/except is cheap); don't signal a frequent/expected branch by raising in a hot loop.

Concurrency footguns

• Threads don't help CPU-bound Python (GIL): use ProcessPoolExecutor or multiprocessing. (I/O releases the GIL, which is exactly why asyncio.to_thread works for blocking I/O.)
• Unbounded asyncio.gather fan-out can exhaust connection pools, overwhelm downstream services, or spike memory; bound concurrency with an asyncio.Semaphore.

Library-specific footguns

• stdlib logging does per-record file I/O: FileHandler and its subclasses (RotatingFileHandler, TimedRotatingFileHandler) call flush(), and check for rotation (stat/tell the file) on every single emit(), with no batching across records. Locally these syscalls are cheap, but over network filesystems (NFS, EFS, GCS FUSE, mounted volumes in cloud environments) each one can cost milliseconds, and log-heavy code can spend most of its time in logging rather than real work. Quick fix: get the actual file I/O off the calling thread with logging.handlers.QueueHandler / QueueListener (see the logging cookbook's "Dealing with handlers that block"); the calling code just enqueues records (fast, non-blocking), and a separate listener thread performs the (still per-record) writes. This matters most for event-loop-driven code, where the calling thread is the loop and blocking it on slow syscalls stalls every other coroutine; it's also a win for the hot path more generally. For more throughput, pair the queue with a custom listener/handler that drains multiple queued records at once and batches them into a single write, rather than writing one-by-one.
• Pydantic re-validation of trusted data: constructing a model from already-valid internal data runs full validation unnecessarily. Use Model.model_construct(**data) to skip validation for trusted/internal data. Also avoid repeated model_dump() calls on the same instance: compute and reuse the dict.
• Prefer pydantic or orjson over stdlib json: use pydantic when you know the structure (parse into a model, serialize with model_dump_json()); use orjson when the structure is unknown or freeform. Both are substantially faster than stdlib json and should be the default choice rather than a late optimization.