| name | fastapi-ai-patterns |
| description | Production patterns and gotchas for building FastAPI services, especially AI/ML/LLM serving. Use when you build, review, or debug a FastAPI app: choosing def vs async def, loading models in lifespan, preventing event-loop blocking, setting response_model to stop data leaks, enforcing object-level authz in queries, wrapping LLMs with streaming + Pydantic validation loops, or designing RAG/embedding endpoints. |
FastAPI AI Patterns
Opinionated, production-calibrated guidance for FastAPI services — with the
focus on what makes AI/ML/LLM serving different from an ordinary CRUD API.
It is a knowledge skill: it teaches the decisions and the traps, then routes you
to a per-topic reference for depth.
Stock agents already write basic FastAPI well (routes, Pydantic models, simple
CRUD). This skill spends its tokens on the parts they get wrong by default:
the concurrency model, the lifespan/singleton lifecycle, the security boundaries,
and the discipline that turns a stochastic LLM into a typed component.
Inspired by FastAPI for AI Engineers: From First Endpoint to Production-Scale
AI Systems (AI Engineering Insider, 2026). All content here is re-expressed in
original wording; consult the book for its worked examples and case studies.
When to use
- Building a FastAPI service that serves ML models, embeddings, RAG, or wraps an LLM
- Reviewing or debugging a FastAPI app for production-readiness (latency, leaks, auth)
- Deciding
def vs async def for a handler, or diagnosing a "p99 exploded under load" incident
- Designing inference, streaming-chat, embedding, or document-Q&A endpoints
- Hardening an endpoint: response shape, authz, rate limiting, input/output guardrails
When NOT to use
- You want a full project skeleton generated → use
fastapi-ai-scaffold
- You are drilling interview questions on these topics → use
fastapi-ai-interview-prep
- Pure frontend/Next.js work, or a non-FastAPI Python web stack (Flask/Django) → different skills
Core mental model: the endpoint is a typed contract
A FastAPI endpoint signature is the API contract. Parameter annotations
declare what the request must contain; the return annotation or response_model
declares what the response will contain. The framework enforces both at runtime
and publishes them as OpenAPI. Everything else — validation, docs, the 422 error
body, SDK generation — is derived from that one declaration.
Two consequences worth internalizing:
- The annotation is the single source of truth. Don't duplicate validation
logic that a Pydantic field constraint already expresses.
- FastAPI is a thin layer over Starlette + Pydantic. When you debug, you are
often reading Starlette (routing, middleware,
BackgroundTasks) or Pydantic
(validation, coercion) source. Knowing which layer owns a behavior is the
difference between guessing and fixing.
The single most important decision: def vs async def
FastAPI runs async def handlers on one event loop; it dispatches plain
def handlers to a threadpool (~40 threads by default). Choose by the
workload, not by habit:
| Workload | Correct handler | Why |
|---|
Async-capable I/O (httpx, asyncpg, async redis) | async def + await | Loop interleaves thousands of in-flight waits |
Blocking-only library (requests, classic ORM, boto3) | plain def | Threadpool absorbs the block; loop stays free |
| Light CPU (< a few ms) | either | Negligible either way |
| Heavy CPU (model inference, parsing, crypto) | plain def, or offload (to_thread/process pool/queue) | Never run multi-ms CPU on the loop |
The deadliest FastAPI production bug: an async def handler that makes a
blocking call inside it (requests.get, time.sleep, a sync model
.predict(), a CPU-heavy parse). It passes every single-request test, then under
concurrency it freezes the entire loop — and p99 explodes for every endpoint
at once, including /health. See references/async-and-external.md.
Top gotchas (read before you write the handler)
These are the corrections an agent most often needs. They are environment facts
that defy reasonable assumptions, so they live here, not in a reference.
async def + blocking call = whole-process stall. One non-cooperating
coroutine starves every other request on the loop. If a call isn't truly
async, use plain def or offload it. Detect with a loop-lag watchdog.- Load ML models once, in the
lifespan handler — never per request, never at
import time. Per-request loading re-reads weights on every call (and OOMs
under concurrency); import-time loading makes tests, linters, and CLI tools pay
the cost (or crash without a GPU). Lifespan loading runs once per worker before
traffic and integrates with readiness probes. With N workers you hold N copies
— size worker count by memory, or front a dedicated inference server.
- Every response needs a
response_model (or explicit return type). Returning
an ORM object directly serializes all its columns — that is how
hashed_password ends up in a JSON response. Separate request / storage /
response schemas; the response schema is a structural leak-prevention boundary.
- Object-level authorization must be enforced in the query, not by review.
BOLA (broken object-level authorization) tops the OWASP API list because
GET /orders/{id} that trusts the path id leaks other users' data. Filter by
owner in the data layer (WHERE owner_id = current_user.id), so an unauthorized
fetch returns 404 by construction — don't rely on a reviewer remembering.
- Echo
model_version (and embedding model version) in every AI response.
When quality regresses the first question is "which model produced this?", and
canary rollouts serve multiple versions at once. Vectors from different
embedding models are incomparable — a model upgrade means re-embedding the corpus.
- Treat LLM output as untrusted input: generate → validate with Pydantic →
retry with the error appended → fall back. This loop is the single most
useful LLM-engineering pattern: it converts a stochastic text generator into a
typed component with an explicit failure mode. One retry fixes most cases;
on the second failure raise a typed error to a defined fallback (human review,
cheaper deterministic path, or 502) — never pass malformed data downstream.
- In RAG, enforce access control in retrieval (SQL/filter), never in the
prompt. Never let the model see chunks the user can't access and "ask it" to
respect permissions — prompt injection defeats that. The authorization boundary
lives outside the model.
- Per-process singletons (
httpx.AsyncClient, DB engine, Redis client, model
handle) belong in lifespan, injected via a dependency — not constructed per
request. A per-request httpx client pays TCP+TLS handshakes every call and
leaks connections; a lifespan-scoped client reuses keepalive connections and
enforces global limits.
- Wrap streaming generators in
try/finally. Clients disconnect mid-stream
constantly; token metering, logging, and upstream stream cancellation must run
even then, or you keep paying for abandoned generations and lose usage data.
- Keep
/health trivially cheap; use a separate /ready for dependency
readiness. A health check that does real work (or shares the event loop with
heavy handlers) will fail under load and trigger restart storms. Liveness and
readiness are different questions and need different probes.
- Retry only idempotent operations on transient errors, with backoff + jitter
and a total deadline below your caller's timeout. Naive retries amplify load
exactly when an upstream is weakest (retry storm); synchronized retries are a
self-inflicted thundering herd. Add a circuit breaker for repeated failure.
- Validate JWTs with a pinned algorithm; hash passwords with bcrypt/argon2, not
SHA-256. Accepting the token's own
alg header enables the alg=none forgery;
fast hashes make password cracking cheap. See references/security.md.
How to use the references
Load the one reference that matches the decision in front of you — don't preload
them all. Each is self-contained with patterns, gotchas, and capacity/cost rules
restated in original wording.
references/api-design.md — Read when designing routes, status codes,
pagination/versioning, or Pydantic v2 request/response schemas (Ch 1–3 scope:
ASGI vs WSGI, REST semantics, idempotency, validation, leak prevention).references/architecture-di.md — Read when structuring a project or wiring
Depends() (Ch 4: DI resolution + caching, yield deps for DB sessions,
router/service/repository layering, composition root, dependency_overrides).references/database.md — Read when adding persistence (Ch 5: session-per-
request, N+1 + selectinload/joinedload, SQLModel vs SQLAlchemy+Pydantic,
Alembic, pool sizing, PgBouncer, flush vs commit).references/security.md — Read when adding auth or hardening (Ch 6: password
hashing, JWT + revocation, API keys, CORS, rate limiting, OWASP API risks, BOLA).references/testing.md — Read when writing tests (Ch 7: TestClient,
dependency_overrides, SQLite-in-memory tiers, authz matrix, time injection,
coverage honesty, 422-body assertions).references/async-and-external.md — Read when doing concurrency or external
calls (Ch 8: event loop model, the full def/async table, httpx client +
retries + circuit breaker, BackgroundTasks vs Celery, webhooks, uploads,
streaming, WebSockets, backpressure).references/ai-ml-serving.md — Read when serving models or LLMs (Ch 9:
model loading, batch + dynamic micro-batching, LLM gateway + SSE streaming,
embeddings + vector DB + RAG, guardrails, validation loop, versioning/monitoring/
HITL, LLM cost model).references/deployment-observability.md — Read when deploying or operating
(Ch 10: Uvicorn/Gunicorn workers, Docker, K8s probes, logs/metrics/traces,
caching + stampede, load testing, graceful degradation, production checklist).
Authoritative sources (link, don't paraphrase from memory)
FastAPI's API surface and the surrounding ecosystem change between minor
versions. When unsure of exact syntax or current behavior, fetch the docs:
Gotchas
The top, cross-cutting gotchas are in the section above (they apply before you
even pick a topic). Topic-specific traps live in each reference's own ## Gotchas
section — for example, flush() vs commit() timing in database.md, the
alg=none attack in security.md, and SSE proxy-buffering in
ai-ml-serving.md. Load the reference for the area you're working in so you read
its gotchas before hitting the situation, not after.
