"Coverage isn't about bug prevention — it's about guaranteeing the agent has double-checked the behavior of every line of code it wrote." — Steve Krenzel
Tests FIRST, then code (TDD):
Write a failing test that defines expected behavior
Run it — verify it fails for the right reason
Write the minimum code to make it pass
Run it — verify it passes
Refactor if needed, re-run tests
Commit
100% meaningful coverage — every branch, every error path. At 100%, any uncovered line is an immediate signal of missing verification. The ratchet gate BLOCKS below 80%.
Only mock external boundaries: databases, third-party APIs, file I/O, clocks.
Never mock business logic — if you mock a service to test another service, you are hiding bugs.
Isolate tests from .env files: When testing settings/config that uses pydantic-settings or dotenv, pass _env_file=None (pydantic) or mock dotenv.load_dotenv to prevent the developer's .env from leaking into tests. Tests must be self-contained — they must pass regardless of what's in the local .env.
Use async-compatible connection strings: When using async frameworks (SQLAlchemy async, asyncpg), defaults must use the async driver scheme (e.g., postgresql+asyncpg:// not postgresql://). The sync scheme will fail at runtime with a cryptic driver error.
Realistic test data — use domain-representative values (real-looking emails, valid UUIDs, plausible amounts). Never "foo", 123, or "test".
Test names describe behavior: "returns 404 when order does not exist", not "test order".
Integration tests for multi-step flows: When a route triggers a background task or async flow (e.g., POST creates a record then starts processing), write a test that calls the endpoint and asserts the FINAL state — not just that each unit works alone. Assert exact record counts: assert db.query(Task).count() == 1 after one API call.
LLM Integration — Structured Output Mandatory
When generated code calls any LLM (Claude, GPT, or other), follow these rules:
1. Always Use Structured Output
Use tool_use / function_calling / response_format: { type: "json_schema", json_schema: ... } for every LLM call. Never parse free-text responses with regex or string splitting.
2. Define a Response Schema
Every LLM call must have a typed model for the expected response:
from pydantic import BaseModel
from typing importLiteralclassClassificationResult(BaseModel):
category: str
confidence: Literal["high", "medium", "low"]
reasoning: str
When generated code calls any external API (third-party services, partner APIs, cloud services), follow these rules. See .claude/skills/code-gen/references/api-integration-patterns.md for full templates.
Service Wrapper Pattern (Mandatory)
Every external API gets a dedicated wrapper class. This is the ONLY file that imports the SDK or makes HTTP calls to that service.
Business Logic (process_service.py)
↓ calls typed methods
API Wrapper (external_client.py) ← only file that imports SDK / makes HTTP calls
↓ calls
External API
Rules:
One wrapper class per external API
Wrapper exposes project-internal typed models, not SDK types
Business logic never sees SDK response objects — only your domain types
The wrapper is the mock boundary in tests
Error Taxonomy (Mandatory)
Every wrapper classifies errors into typed categories:
Missing raw response logging — Always log raw LLM response at DEBUG before parsing. This is the debugging ground truth.
Direct SDK imports outside wrapper — All SDK imports must be inside the wrapper class file. Business logic imports your wrapper, not the SDK.
Bare except on API calls — Catch ApiTransientError and ApiPermanentError specifically. Never except Exception.
Hardcoded retry config — Retry attempts, backoff, and timeout belong in config.yml, not in code.
Missing structured logging in API wrapper — Every request/response/error must be logged with structured fields (service, operation, attempt, latency_ms).
f-string log messages — Use extra dict for structured fields, not string interpolation. Structured logs are searchable; f-strings are not.
Missing logging at service boundaries — Every incoming request and outgoing call must be logged with timing and status.
Raw dict API responses — Always serialize through a response model. Raw dicts bypass validation and leak internal structure.
Magic numbers — All thresholds, limits, timeouts, and configuration belong in config.yml.
.env leaking into tests — Tests that validate "missing config raises error" will pass in CI but fail locally if .env has the value. Always pass _env_file=None in pydantic-settings tests.
Sync DB driver in async app — postgresql:// uses psycopg2 (sync). Async SQLAlchemy needs postgresql+asyncpg://. Always match the driver scheme to the engine type.
Duplicate record creation — Route creates a record, then calls a service that creates the same record again. Pass the ID, don't re-create. Test with assert count == 1 after one API call.
Manual session creation — Never create DB sessions manually per request. Use Depends(get_db) with async_sessionmaker. Manual sessions leak connections.
Fire-and-forget background tasks — Every background task must update a DB record on completion or failure. No background_tasks.add_task(fn) without status tracking.
CORS allow_origins=["*"] — Never use wildcard origins with allow_credentials=True. Read origins from env var, default to localhost.
Health check returns OK without checking DB — Health endpoint must SELECT 1 against the database. A healthy HTTP server with a dead DB is not healthy.
Engine not disposed on shutdown — Always await engine.dispose() in the lifespan's teardown. Leaked connections exhaust the pool.
No request ID tracing — Add middleware that generates a UUID per request, injects into logs and response headers. Without it, errors can't be traced to requests.
Deprecated startup/shutdown events — Use @asynccontextmanager lifespan, not @app.on_event("startup"). The event-based API is deprecated in FastAPI.
Thread pool exhaustion — asyncio.to_thread() uses a default pool of ~5 workers. Under concurrent load, blocking SDK calls exhaust the pool. Set loop.set_default_executor(ThreadPoolExecutor(max_workers=20)) or use async clients.
