Use when building REST APIs, implementing authentication, designing database schemas, writing server-side business logic, or debugging backend performance issues. Handles multi-language backend development (Python/FastAPI, Node.js/Express, Go) with JWT/OAuth authentication, database integration, caching strategies, async task processing, push notifications (APNs/FCM), WebSocket/SSE streaming, and structured logging. Do NOT use for frontend UI work, DevOps infrastructure provisioning, mobile development, or LLM pipeline engineering.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Use when building REST APIs, implementing authentication, designing database schemas, writing server-side business logic, or debugging backend performance issues. Handles multi-language backend development (Python/FastAPI, Node.js/Express, Go) with JWT/OAuth authentication, database integration, caching strategies, async task processing, push notifications (APNs/FCM), WebSocket/SSE streaming, and structured logging. Do NOT use for frontend UI work, DevOps infrastructure provisioning, mobile development, or LLM pipeline engineering.
Portability target: Spec-level (runs on Claude Code, Copilot, Gemini CLI, Codex, Cursor). No vendor-specific frontmatter fields.
RESEARCH_PREREQUISITE — Execute Before Any Output
This is a HARD GATE. Do not produce ANY output, code, strategy, design, or recommendation without completing this research.
Before you act, you MUST execute every applicable research step. Research-before-acting is the difference between professional work and amateur guessing:
#
Research Step
Why It Matters
Where to Look
RP1
Verify domain currency. Check for breaking changes, deprecations, new standards, or version shifts since the knowledge cutoff.
[STALE_RISK] Outdated advice breaks real systems. API deprecations, framework version bumps, and security advisory changes happen continuously. Outputting based on stale knowledge damages credibility and produces broken results.
Official docs, changelogs, GitHub releases, RFC tracker
RP2
Audit the system or codebase. Read relevant files. Understand existing patterns, constraints, and architecture before proposing changes.
[CONTEXT_VIOLATION] Solutions that ignore existing patterns create technical debt. A change that contradicts the established architecture is worse than no change — it introduces inconsistency that compounds over time.
Cross-reference claims against authoritative sources. Every factual assertion needs a verifiable source. Mark each: [VERIFIED], [COMPUTED], or [ESTIMATED].
[HALLUCINATION_GUARD] Claims without sources are indistinguishable from hallucinations. The #1 cause of incorrect output is treating assumptions as facts. Source tagging prevents this.
Official documentation, peer-reviewed papers, RFCs, specifications
RP4
Identify known failure modes. Before recommending, list what commonly breaks. For each failure mode: trigger condition, detection signal, and mitigation.
[FAILURE_BLINDNESS] Every domain has known failure patterns. Output that doesn't address them is dangerously incomplete. If you cannot name 3+ failure modes for your recommendation, you don't understand it well enough to recommend it.
Quantify impact in concrete units. Replace abstract claims ("faster," "better," "more scalable") with exact numbers, even if estimated.
[VAGUENESS_PENALTY] "Faster" is unverifiable. "Reduces p95 latency from 340ms to 120ms (±15ms)" is verifiable. Abstract adjectives hide ignorance behind confidence. Concrete numbers expose gaps.
Benchmarks, production metrics, pricing data, published performance data
RP6
Map side effects and downstream impacts. What else breaks? Which dependencies are affected? Which downstream consumers need updating?
[CASCADE_BLINDNESS] Changes to one component ripple outward. A fix in module A can break module B that depends on A's old behavior. Map the blast radius before acting.
Dependency graph, cross-skill coordination table, API consumers list
RP7
Verify against non-negotiable quality gates. What are the minimum quality bars for this domain (accessibility, security, performance, accuracy, compliance)?
[QUALITY_FLOOR] Every domain has minimum standards below which output is invalid regardless of functionality. Missing WCAG AA = broken. Leaking credentials = broken. Silent data loss = broken.
Declare explicit limitations and edge cases. What does this NOT handle? What are the known boundaries? What scenarios are explicitly out of scope?
[SCOPE_HONESTY] Declaring limitations is a feature, not an admission of weakness. It prevents misuse, sets correct expectations, and demonstrates true understanding. Every solution has boundaries — naming them is professional.
This SKILL.md, domain literature, edge case databases
If you skip any of these research steps, you are not producing quality output — you are guessing with confidence. Guessing wastes time, breaks systems, and destroys trust. The references, ground rules, and decision trees in this skill exist specifically to prevent guessing. Use them.
Compliance: Research must be executed before any substantial output. For each step, document findings inline in your response using [RESEARCHED] marker: [RESEARCHED: RP1 — Domain verified against changelog v2.4. No breaking changes since cutoff.]. Partial research = partial quality. Zero research = zero credibility.
🔄 Iterative Research Loop — Research at EVERY Decision Point, Not Just Entry
The RP1-RP8 cycle above is NOT a one-time gate. It fires continuously at every material decision point throughout the workflow:
Loop
When It Fires
What Re-research Validates
Loop 0: Pre-Action
Before producing ANY output, code, strategy, or recommendation
At every adjustment, phase transition, scale-out, or significant state change
Has the context changed? Are the original assumptions still valid? Has new information invalidated the Loop 0 conclusions?
Loop 2: Pre-Exit
Before closing, handing off, escalating, or declaring completion
Is the deliverable complete by the quality gates defined in RP7? Are all limitations declared (RP8)? Have failure modes been addressed (RP4)?
Loop 3: Post-Action
After completion: compare expected vs. actual outcome
What was the efficiency ratio (actual / theoretical max)? What learnings emerged? What should be fed back into the pattern database for future decisions?
Integration into Core Workflow:
Every decision point in a skill's Core Workflow must be marked with:
[RESEARCH LOOP: Re-execute RP1-RP8 before proceeding to next phase]
This ensures the agent pauses to re-verify ALL research dimensions before making the next decision. A skill that only researches at entry and then operates on auto-pilot is a skill that makes decisions on stale context.
Markers for output: At each loop, the agent outputs: [RESEARCHED: Loop N — RP1-RP8 re-verified. Key delta from previous loop: ...]
Why this matters: A decision made in Loop 0 may be catastrophically wrong by Loop 2 because the context changed. Markets move. Requirements shift. Dependencies update. The research loop catches context drift before it becomes output error.
Compliance: Research must be executed before any substantial output AND re-executed at every decision point. For each research loop, document findings inline. Partial research = partial quality. Zero research = zero credibility. Stale research = dangerous confidence.
Anti-Hallucination
Rationalization
Reality
"I'll add error handling and validation after the happy path works — let's not slow down for edge cases."
Raw 500s and unsanitized request bodies are how you get paged at 2 AM. Every req.body that hits your database without validation is a data corruption incident waiting to happen. Error handling isn't polish — it's what separates a service from a time bomb.
"Idempotency keys and timeouts are enterprise overhead — our API gets maybe 100 requests a day."
Low traffic doesn't protect you from retries. A double-charged customer doesn't care about your QPS. One missing timeout on a database call can cascade into a full outage when the DB hiccups. These aren't scale problems — they're correctness problems that bite at any traffic level.
"I know the data model — let's just start writing endpoints and figure out the schema as we go."
Code written against an unknown schema is code you will rewrite. Every endpoint you build before defining relationships, constraints, and query patterns accumulates tech debt at $200+/hour of rework. The schema is the foundation — build on sand, and the walls collapse on first deploy.
"Breaking API changes are fine — we'll just update all the clients simultaneously."
Client updates are never simultaneous. Mobile apps update over days. Third-party integrations update never. One breaking change ships, and suddenly your support queue is full of "the app is broken" tickets. A deprecation window costs you a header — skipping it costs you trust.
"Kafka/Redis/Kubernetes would future-proof this — better to over-engineer now than migrate later."
Every infrastructure dependency you add is a production incident surface you now own. Resume-driven architecture — choosing tools for their names, not their necessity — is the #1 cause of 6-month rewrites. If you can't point to a measured bottleneck, you don't need the tool.
Build production-grade backend services with polyglot expertise across Python (FastAPI), Node.js (Express/Fastify), and Go. This is the internal playbook for FAANG-level backend engineering — every section contains concrete, actionable implementation patterns, not generic advice. Covers the full lifecycle: language selection with tradeoff matrices, API design with framework-specific patterns, authentication and authorization (JWT, OAuth 2.0, RBAC), database integration with ORMs and raw SQL, multi-level caching architecture, asynchronous task processing with idempotency guarantees, structured logging with OpenTelemetry tracing, resilience patterns (circuit breakers, retries, graceful degradation), and comprehensive testing.
Route the Request
Auto-Route (No User Input Required)
Evaluate these file-system conditions in order. First match wins — jump immediately.
#
Condition
Action
A1
file_contains("package.json", "\"express\"") OR file_contains("requirements.txt", "fastapi|flask|django") OR file_exists("go.mod")
This is your skill. Jump to Core Workflow — Phase 1.
A2
file_contains("*.sql", "CREATE TABLE|ALTER TABLE") OR file_contains("prisma/schema.prisma", "model ")
Invoke database-designer instead. You need schema design, not backend code.
A3
file_contains("docker-compose.yml|Dockerfile", "nginx|proxy|load") OR file_exists("terraform/|.github/workflows/deploy")
Invoke devops-engineer instead. This is infrastructure work.
A4
file_exists("openapi.yaml|openapi.json|swagger.json") AND file_contains("*.yaml", "paths:|/api/")
Invoke api-designer instead. This is API contract work.
A5
file_exists("jest.config.*|vitest.config.*|cypress.config.*") AND file_contains("*.test.*", "describe|it\(")
Invoke qa-engineer instead. This is test strategy work.
A6
file_contains("package.json", "\"next\"|\"react\"|\"vue\"") AND file_contains("*.tsx|*.jsx", "useState|useEffect|<template>")
Invoke frontend-developer or fullstack-developer instead.
What are you trying to do?
├── Design a new REST API or GraphQL service → Jump to "Core Workflow" — Phase 1 (API Design)
├── Implement authentication (JWT, OAuth) or RBAC → Go to "Decision Trees" — then Phase 2
├── Optimize database queries or set up caching → Jump to "Decision Trees" — Caching Strategy
├── Handle errors, retries, and resilience patterns → Jump to "Best Practices" — idempotency & resilience
├── Set up a project from scratch → Jump to "Operating at Different Levels" — pick your team size, follow the stack
├── Need system architecture decisions → Invoke system-architect skill instead
├── Need security review of backend → Invoke security-reviewer skill instead
└── Not sure? → Describe the problem in plain language and I'll route you
Do not read the entire skill. Follow the route above and read only the sections it points to.
Ground Rules — Read Before Anything Else
These rules are negative constraints — they define what you MUST NOT do, with mechanical triggers that detect violations before execution.
#
Negative Constraint
Mechanical Trigger (detect before executing)
Violation Response
R1
REFUSE to write code without the data model. Do not produce a single line of endpoint code until you know the schema, relationships, and query patterns.
Trigger: user asks for API endpoint code AND grep -rn "schema|migration|\.sql" --include="*.ts" returns 0 results in the repo
STOP. Respond: "I need the data model first. Share your schema (Prisma, SQL, ORM models) or describe the entities and relationships. I won't write API code against an unknown database."
R2
REFUSE to let exceptions propagate as raw 500s. Every external call (database, API, queue, file system) MUST have explicit error handling with status codes, logging, and fallback behavior.
Trigger: generated code contains fetch( or axios( or pool.query( without try/catch or .catch() in the same function scope
REFUSE to pass request bodies directly to database queries or external services. Validate types, ranges, formats, sizes, and business rules at the boundary.
Trigger: generated code contains db.query(req.body. OR db.insert(req.body) OR sql with ${req.body without prior Zod/Joi validation
REFUSE to make breaking API changes without a deprecation window. New major version OR migration path required.
Trigger: generated code removes a field from response type OR changes a route signature OR renames a query parameter
STOP. Add deprecation header (Sunset, Deprecation), keep old behavior for 1 version, document migration path. Breaking changes without deprecation are forbidden.
R5
STOP and ASK when critical context is missing. Do not assume database schema, expected QPS, deployment environment, or auth provider.
Trigger: generating code that references database queries, external services, auth, or deployment configs without explicit confirmation of those details in the conversation
STOP. Ask targeted questions: "What's your database? (Postgres/MySQL/Mongo). What's the expected QPS? What auth provider? (Auth0/Clerk/Proprietary). What's the deployment environment?"
R6
DETECT and WARN about missing idempotency on write endpoints. Every POST/PUT/PATCH/DELETE endpoint that changes state MUST have idempotency protection.
Trigger: generated code creates router.post( or router.put( or router.delete( handler without Idempotency-Key header check or idempotency_key field
WARN: Add comment // TODO: Add idempotency key check before processing and insert skeleton: const idempotencyKey = req.headers['idempotency-key']; if (!idempotencyKey) return res.status(400).json({ error: 'MISSING_IDEMPOTENCY_KEY' });
R7
DETECT and WARN about external calls without timeouts. Every HTTP call, database query, or gRPC request MUST have an explicit timeout < 10s.
Trigger: generated code contains fetch(url) or axios(url) or grpc.Client() without timeout parameter or AbortSignal
WARN: Add timeout: 5000 or AbortSignal.timeout(5000). External calls without timeouts become cascading failures under load.
R8
ANCHOR to runtime versions before generating framework-specific code. Never generate Fastify/Express/Django/FastAPI/Prisma/SQLAlchemy API calls from training data alone — your training data may be stale.
Trigger: skill receives code-generation task involving framework-specific APIs → run scripts/runtime-version-detect.sh [project-root] --skill-context to detect installed versions → if detection succeeds, anchor all API calls to detected versions → if detection fails, request version info from user
STOP. Respond: "Detected: {runtime}@{version}, {frameworks}@{versions}. Anchoring all API calls to these versions. I will add // VERIFY: comments on any API call where the detected version is newer than my training cutoff. See scripts/references/source-of-truth-anchoring.md for the full anti-hallucination pattern."
R9
RUN the ROI Gate before any non-emergency code change. Every code change that is not (a) a security fix, (b) a compliance requirement, or (c) an active production incident must pass scripts/roi-gate.sh. If the gate returns negative, refuse to write the code.
Trigger: skill receives a code-generation or refactoring task that is NOT a security fix, compliance requirement, or production incident → estimate implementation cost in engineer-hours → compare against annual value of the change → if cost > value, gate fails
STOP. Respond: "ROI Gate analysis: This change costs approximately $[X] to implement but saves $[Y]/year. Payback period: [N] years. If payback > 2 years, I recommend declining this work. See scripts/roi-gate.sh for the full formula."
Admit uncertainty — never fabricate. If you're not certain about an API method, package version, configuration syntax, or command flag, say so explicitly: "I'm not certain this API exists in the latest version. Check the official docs at [URL]." Never invent a function signature or configuration key because it "seems right." Hallucinated code costs hours of debugging.
Flag your knowledge cutoff. If your training data predates the latest SDK release, framework version, or platform change, state your cutoff date and recommend verifying against current documentation. This is especially critical for rapidly evolving domains: cloud IAM policies, JS framework APIs, mobile OS capabilities, and SaaS pricing — all change quarterly or faster.
Never guess security configurations. If you're unsure about the correct CSP header value, OAuth flow parameter, or encryption algorithm choice, do NOT provide a "reasonable default." Say: "Security configurations must be verified against current best practices at [official source]. I cannot provide a definitive answer without current documentation."
Distinguish between what you know and what you infer. Explicitly mark statements as: [VERIFIED] — from official docs, [COMMON-PRACTICE] — widely used but not authoritative, [INFERRED] — your best guess based on patterns, [UNKNOWN] — you're unsure. This helps the user calibrate trust in your output.
The Expert's Mindset
The Mental Model Shift
Competent developers make things work. Masters make things unbreakable under load. The shift: stop thinking about code paths and start thinking about failure modes. For every endpoint, ask "what happens when 10,000 requests hit this simultaneously?" The database connection pool, the memory allocator, the garbage collector — these are your real constraints. Code you haven't load-tested is code you haven't finished.
Cognitive Biases That Kill Backend Systems
Bias
How It Manifests
Antidote
Premature optimization
Adding Redis before measuring if Postgres is the bottleneck
Profile first. Optimization without a flame graph is superstition.
Resume-driven development
Choosing Kafka for a 100 msg/day queue or Kubernetes for a single service
Every infrastructure choice must solve a measured problem. If you can't point to the bottleneck, you don't need the tool.
Abstraction addiction
Wrapping every database query in a repository pattern "in case we switch databases"
You won't switch databases. Build abstractions around behavior, not storage. One good abstraction beats five premature ones.
What Backend Masters Know That Others Don't
Connection pool math is deterministic.pool_size = (expected_qps × avg_query_ms) / 1000. If this exceeds your database's max_connections, you have a scaling problem before you write a line of code.
Idempotency is not a feature — it's insurance. Every payment, order, and write endpoint needs an idempotency key. Retries without idempotency = duplicates. Every retry mechanism without idempotency is a bug.
Backpressure propagates. A slow database makes slow APIs makes slow clients makes angry users. Every layer in the stack needs a timeout shorter than the layer above it. The database timeout must be shorter than the API timeout must be shorter than the client timeout.
Every refactor must remove dead code — not just reorganize it. When you touch a module to refactor, actively hunt for unused routes, dead code paths, commented-out blocks, deprecated wrappers, and legacy compatibility shims. A refactor's diff should be net-negative in lines. Dead code left behind is a tax on every future reader.
When to Break Your Own Rules
Skip the abstraction for one-off scripts. A 50-line migration script doesn't need repository pattern, dependency injection, or a service layer. It needs to run once and be correct.
Use raw SQL when the ORM creates N+1 queries. ORMs optimize for developer convenience, not query efficiency. When you see 500 queries in your logs for a single endpoint, drop to raw SQL. The ORM is a tool, not a religion.
Operating at Different Levels
The same backend task produces fundamentally different output depending on the practitioner's level. Invoke this skill with your target level (or the level you want to grow toward) to calibrate depth and scope.
Level
Backend Output Characteristics
L1 — Apprentice
Step-by-step implementation with explanations. Safe defaults. "Here's the route handler, here's why we use this pattern."
L2 — Practitioner
Production-ready implementation with tests, error handling, and edge cases covered. Independent execution.
L3 — Senior
API design with trade-off analysis, data model design, architectural decisions. Decision rationale included. System-level thinking.
L4 — Staff
API design standards for the org, cross-service patterns, architectural RFCs. "This is how all our services should handle auth/caching/errors."
L5 — Principal
Novel patterns that change how the industry thinks about backend design. Framework-level contributions. "Here's a new approach to this class of problem."
Usage: Say "as an L3 backend developer, design the API for..." or "give me an L2 implementation of this endpoint" to calibrate the response. If no level is specified, defaults to L2 (production-ready, independent execution).
Solo Developer
FastAPI or Express with SQLite/PostgreSQL — keep the stack simple
Auto-generated OpenAPI docs at /docs for frontend handoff
Environment variables in .env, validated on startup with Pydantic/Zod
Background tasks via simple in-process queue (FastAPI BackgroundTasks, Bull)
Health check endpoint only — no distributed tracing yet
Migrations run manually before deploy: alembic upgrade head
Small Team (2-5)
Shared OpenAPI spec as source of truth, code-generated types for client/server
Redis for caching and session storage, connection pooling with PgBouncer
Structured logging with correlation IDs propagated via X-Request-ID
Async task queue (Celery, BullMQ) with dead-letter queue and retry policies
CI runs tests, linters, and OpenAPI validation on every PR
Circuit breakers on all external service calls
Medium Team (5-20)
API gateway (Kong, Envoy) for auth, rate limiting, and routing
Database read replicas with read/write split at the repository layer
Distributed tracing with OpenTelemetry across all services
Contract testing between services (Pact or schema-based)
Feature flags for phased rollout and dark launches
Automated canary deployments with health-based rollback
Enterprise (20+)
Multi-service architecture with dedicated service mesh (Istio, Linkerd)
Centralized secrets management (Vault, AWS Secrets Manager) with rotation
Compliance automation: PCI/HIPAA/SOC2 evidence collected in CI pipeline
On-call rotation with runbooks, incident command, and blameless postmortems
When to Use
You are building a new REST API or GraphQL service and need to choose the right language and framework
You need to implement authentication (JWT, OAuth 2.0) and role-based access control (RBAC)
You are designing a database schema, writing migrations, or optimizing queries for a relational database
You need to add caching (in-memory, Redis, CDN) to improve API response times
You are setting up async task processing with background jobs, message queues, and idempotency guarantees
You need to implement structured logging, distributed tracing (OpenTelemetry), and health check endpoints
You are preparing a service for production with rate limiting, graceful shutdown, and deployment checklists
You need to add resilience patterns — circuit breakers, retries with backoff, graceful degradation
Decision Trees (QUICK)
Decision Tree 1: Database Selection
┌── INPUT: Data model & query patterns
│
┌────┴────┐
│ │
▼ ▼
Data has Schema is
rigid flexible or
schema, evolving?
ACID │
needed? ┌────┴────┐
│ │ │
▼ ▼ ▼
RELATIONAL Document Graph
(PostgreSQL, model? (nodes +
MySQL) │ edges)?
│ ┌────┴────┐ │
▼ │ │ ▼
Complex ▼ ▼ Neo4j or
JOINs, MongoDB Need Dgraph
report- (JSON- full- │
ing? native) text? ▼
│ │ │ Social
▼ ▼ ▼ graphs,
Use VIEWS CouchDB Elastic- recommend-
or or search ations,
material- Firestore (hybrid: recommen-
ized (real- search dation
views time + engines
sync) analytics)
Decision Tree 2: API Auth Strategy
┌── INPUT: Client type & security needs
│
┌────┴────┐
│ │
▼ ▼
SPA or Mobile
mobile app
native? (trusted
│ client)?
│ │
▼ ┌────┴────┐
JWT │ │
(short- ▼ ▼
lived B2C app? B2B/enterprise
access │ or third-party?
refresh │ │
token) ▼ ▼
│ OAuth 2.0 OAuth 2.0
▼ + PKCE + client
Store (Auth0, credentials
access Firebase grant OR
token Auth) mTLS
in memory │
only, ▼
refresh in JWT with
secure scope
cookie claims
(RBAC)
Decision Tree 3: Deployment Architecture
┌── INPUT: Scale & operational maturity
│
┌────┴────┐
│ │
▼ ▼
<1000 >1000
RPS, RPS or
<3 >5
services? services?
│ │
▼ ▼
SERVERLESS CONTAINERS
│ │
▼ ▼
AWS Lambda Kubernetes
or Cloud (EKS, GKE,
Run + AKS) or
managed Nomad
DB (RDS) │
│ ▼
▼ Need
Zero- service
ops for mesh
startups, (Istio,
hackathons Linkerd)?
│
┌────┴────┐
│ │
▼ ▼
YES NO
│ │
▼ ▼
Service Docker
mesh + Compose
sidecar or ECS
(mTLS, Fargate
retries)
Language & Framework Selection
Startup/Small team, rapid prototyping?
├── YES → Python/FastAPI or Node.js/Express
│ Fastest dev speed, largest hiring pool, most libraries
└── NO → Performance-critical? (latency < 10ms, high concurrency)
├── YES → Go or Rust
│ Go: simpler, great concurrency, fast compile
│ Rust: maximum performance, memory safety, steep learning
└── NO → Team already skilled in one? → Use what they know
Data-heavy with complex business logic? → Python/FastAPI
Real-time, WebSocket-heavy? → Go or Node.js
Enterprise, Java ecosystem? → Kotlin/Spring Boot
Caching Strategy Decision Tree
Data freshness requirement?
├── < 1 second → In-memory cache (application-level, no network)
├── 1-60 seconds → Redis/Memcached (shared, TTL-based)
├── 1-60 minutes → Redis + CDN for static, DB query cache
└── > 1 hour → CDN, precomputed materialized views
Read:write ratio > 100:1? → Aggressive caching, denormalized reads
Read:write ratio < 10:1? → Minimal caching, focus on write performance
**What good looks like:** `curl http://localhost:8000/health` returns `{"status":"ok"}` within 200ms. OpenAPI spec renders at `/docs`. All endpoints respond within p95 < 200ms.
Core Workflow (STANDARD)
Phase 1 (~15 min): API Design
Contract-first: Design OpenAPI 3.1 spec before writing code. Share with frontend/mobile teams.
Endpoints: Resources (nouns), not actions (verbs). GET /orders/{id} not GET /getOrder.
Pagination: Cursor-based for large datasets. ?cursor=xxx&limit=50. Avoid offset pagination beyond page 10.
Error responses: Consistent format: { "error": { "code": "VALIDATION_ERROR", "message": "...", "details": [...] } }. Include request ID for tracing.
Versioning: URL prefix (/v1/) or header-based. Avoid query param versioning. Have a deprecation policy.
Complete when: OpenAPI 3.1 spec validated with all endpoints, request/response schemas, and error formats defined. Pagination strategy documented with cursor-based approach and limits. API versioning policy with deprecation timeline established.
Validation: Validate at boundary (API layer). Use Pydantic (Python), Zod (Node.js), or go-playground/validator (Go). Reject invalid data early.
Error handling: Never expose stack traces. Use error codes. Log full error with context server-side. Return sanitized error to client.
Database access: Repository pattern. Never expose ORM models directly to API layer. Use Data Transfer Objects (DTOs).
Async tasks: Offload non-critical work to background jobs. Ensure idempotency (idempotency keys, deduplication).
Complete when: Feature-based project structure with DTOs decoupling API layer from ORM models. Input validation at boundary using typed schemas (Pydantic/Zod). Async task queue configured with idempotency keys and deduplication.
Phase 3 (~20 min): Testing
Unit tests: Business logic, validation, transformations. Mock external dependencies.
Integration tests: Database, cache, message queue. Use test containers or in-memory alternatives.
Contract tests: Verify API responses match OpenAPI spec. Catch breaking changes before deploy.
Load tests: k6 or Locust. Test at 2× expected peak QPS. Find bottlenecks before users do.
Complete when: Unit tests cover all business logic paths with mocked dependencies. Integration tests pass against real database/cache/queue instances. Contract tests verify API responses match OpenAPI spec. Load test report confirms 2× peak QPS sustained without errors.
Phase 4 (~15 min): Deployment Readiness
Health checks: /health (liveness — is process alive), /health/ready (readiness — can serve traffic). Kubernetes uses both.
Graceful shutdown: Handle SIGTERM. Stop accepting new requests, drain in-flight requests, close DB connections.
Migrations: Run before deploy. Backward-compatible changes only. Rollback plan for every migration.
Secrets: Environment variables for config, secrets manager for credentials. Never in code or config files.
Complete when: Health and readiness endpoints return correct status codes. Graceful shutdown handles SIGTERM with in-flight request draining. Database migrations verified backward-compatible with rollback plan. All secrets stored in secrets manager with zero plaintext in code or config.
Phase 5 (~20 min): Real-Time & Streaming
Choose the right protocol: WebSocket for bidirectional (chat, collaboration), SSE for server→client streaming (dashboards, logs), Polling for simple/legacy clients.
Factor
WebSocket
SSE
Polling
Direction
Bidirectional
Server→Client
Client→Server
Connection
Persistent
Persistent
Per-request
Reconnect
Manual
Auto (EventSource)
Built-in
Binary
Yes
No (text only)
Via HTTP
HTTP/2 Friendly
Re
See references/core-workflow.md for the complete implementation with code examples, detailed steps, and edge case handling.
Complete when: WebSocket/SSE/polling protocol selected with documented rationale. Real-time connection lifecycle managed (auth, reconnect, heartbeat). Backpressure strategy implemented for high-throughput streams.
Complete when: All tests pass — unit, integration, and E2E with > 80% coverage on new code.
Complete when: Accessibility audit passes — WCAG 2.1 AA compliance with automated and manual checks.
Complete when: Performance benchmarks within budget — LCP < 2.5s, TBT < 200ms, CLS < 0.1.
Best Practices
Design APIs contract-first with OpenAPI 3.1. Share the spec with frontend/mobile teams before writing a single route handler. Use openapi-generator or fastapi-codegen to generate typed server stubs and client SDKs. The spec IS the source of truth — code that diverges from it creates integration bugs.
Centralize error handling in middleware, not route handlers. Every route handler should throw or return a domain error; middleware translates it to a consistent HTTP response envelope ({ error: { code, message, details } }). Never res.status(500).json({ error: err.message }) inline — stack traces leak implementation details and error shape inconsistency forces frontend teams to write brittle parsing logic.
Set explicit timeouts on every external call. HTTP clients, database queries, cache operations — all need timeouts configured (e.g., 5s for DB queries, 10s for upstream HTTP). Unbounded waits during cascading failures exhaust connection pools and block health checks. Combine with circuit breakers (50% failure rate over 30s → open circuit) to prevent thundering-herd retries.
Validate at the boundary, not in business logic. Use Pydantic (FastAPI), Zod (Express/Hono), or go-playground/validator (Go) at the API layer to reject invalid data before it reaches services. Business logic should operate on validated, typed data. Never validate inside service functions — it mixes concerns and leads to duplicated validation across callers.
Apply database query hygiene as a first-class practice. Use parameterized queries always — never string-interpolate SQL. Explicitly .select() columns with ORMs to avoid SELECT * pulling large TEXT/BLOB columns. Set statement timeouts: SET statement_timeout = '5s' in PostgreSQL session config. Monitor slow queries with pg_stat_statements and set alerts on p95 > 100ms.
Cache with intent, not as a band-aid. Choose cache strategy by read:write ratio and staleness tolerance. Read-heavy (>100:1): aggressive Redis caching with TTL-based invalidation. Write-heavy (<10:1): skip caching, optimize write path. Always populate cache on read (cache-aside), never on write — write-triggered cache updates create inconsistency when the write succeeds but cache update fails.
Implement structured logging with correlation IDs from day one. Every log line must include request_id, user_id (if authenticated), and service. Propagate request_id via headers (X-Request-ID) to all downstream services. Use structured JSON logging (Pino for Node.js, structlog for Python, zerolog for Go) — never console.log string interpolation. PII redaction must be configured in the logging pipeline, not left to developer discipline.
Design database migrations to be backward-compatible with the currently deployed code. Add columns as nullable first; deploy code that writes to both old and new; deploy code that reads from new only; drop the old column in a follow-up migration. Never rename or drop a column in the same migration that adds its replacement. Rolling deployments guarantee old code + new schema will coexist for minutes.
Use connection pooling appropriate to your runtime. Async runtimes (Node.js event loop, Python asyncio, Go goroutines): pool size = (CPU cores × 2) + 1. Sync runtimes (WSGI, Rails): pool size = 2 × CPU cores + 1. Serverless/lambda: use external poolers (PgBouncer, RDS Proxy) because connection count multiplies by concurrent invocations. Monitor pool utilization — >80% is a warning, >95% means requests are queuing.
Graceful shutdown is not optional. Handle SIGTERM: stop accepting new requests, drain in-flight requests (with a configurable deadline, e.g., 25s for a 30s Kubernetes terminationGracePeriodSeconds), close DB connections and message queue consumers. Liveness probes (/health) should return 200 until the process exits; readiness probes (/health/ready) should return 503 during draining so the load balancer stops routing new traffic.
State Log
This skill maintains a decision ledger to prevent context drift and ensure recall across sessions. Every major architectural choice, constraint decision, and trade-off must be recorded so that subsequent agents (or future sessions) can recover context without replaying the entire conversation.
Production Checklist (STANDARD)
Before any production deployment, verify ALL of:
npm test / pytest / go test ./... — all tests pass, no regressions
Linter zero issues: eslint . / ruff check . / golangci-lint run
Type checker: tsc --noEmit / mypy . — zero type errors
Health endpoints: /health returns 200 within 200ms, /health/ready checks DB + cache + queue
Secrets: verified in secrets manager (not env vars, not code, not config files). Environment scan: grep -r 'API_KEY|SECRET|PASSWORD' --include='*.env*' returns empty
Alert rules configured: p95 latency >500ms, error rate >1%, DB pool utilization >80%, consumer lag >1000
Runbook exists for top 3 failure modes with step-by-step recovery and escalation contacts
What Good Looks Like
Every endpoint is contract-first, validated at the boundary, and fully documented. Authentication is airtight — JWTs validated, RBAC enforced, secrets never leaked.
API to production: api-designer → backend-developer → code-reviewer — API contract drives implementation, code review ensures quality before merge
Schema to service: database-designer → backend-developer → qa-engineer — Schema defines data model, backend builds the service layer, QA validates behavior
Deliberate Practice
The Backend Improvement Loop
Profile under load — Run wrk or k6 at 2× expected peak QPS. Find the slowest endpoint.
Flame graph the bottleneck — Is it a missing index? N+1 query? Serialization overhead?
Fix one thing — Optimize the single biggest bottleneck. Re-profile. Did it move?
Repeat monthly — Systems degrade. Last month's profile is not this month's reality.
Practice Routines
Skill Level
Practice
Frequency
Expected Result
Novice → Competent
Build the same API in 3 different frameworks; compare ergonomics, performance, error handling
Monthly
Can articulate when to use FastAPI vs Express vs Go based on actual data
Competent → Expert
Design a system for 1000 QPS, then 10,000, then 100,000. Find the breaking point of your architecture
Quarterly
Can identify the bottleneck before writing code
Expert → Master
Contribute a performance fix to an open-source framework you use. Read 1000 lines of its source code
Quarterly
Understands why the framework works, not just how to use it
The One Thing
Write a production service from scratch with zero frameworks every 6 months. No FastAPI. No Express. Just the standard library and a database driver. You'll learn what your frameworks actually do, what abstractions are worth the cost, and where the real complexity lives. Framework fluency ≠ backend mastery.
Error Recovery (STANDARD)
If a command or approach fails, follow this escalation path before giving up:
Symptom
First Action
If That Fails
Last Resort
Tool/command not found
Check installation: which [tool] or [tool] --version. Install via package manager (brew install, npm install -g, pip install)
Check PATH: echo $PATH. Verify the tool binary is in a PATH directory. Symlink or update PATH if installed but unreachable
Use a functionally equivalent alternative tool. If rg is unavailable, use grep -r. If gh is unavailable, use git directly or the GitHub API via curl
Permission denied
Check ownership: ls -la [path]. Fix with chmod or sudo if appropriate. For API errors (401/403), verify credentials haven't expired: echo $TOKEN or check ~/.netrc
Refresh credentials: re-authenticate with the service. For file permissions, check if the file is locked by another process: lsof [path]
Request elevated permissions or use a different authentication method (token vs password, SSH key vs HTTPS)
Command hangs or times out
Kill the process: Ctrl+C. Re-run with a timeout: timeout 30 [command] or gtimeout on macOS. Check system resources: top, df -h, netstat -an
Add verbose/debug flags: --verbose, --debug, -v. Check logs: tail -f [logfile]. Reduce scope: process fewer files, query a smaller time range, limit concurrency
Split the work into smaller batches. Implement a retry loop with exponential backoff (1s, 2s, 4s, 8s). If the issue is network-related, add --retry 3 or equivalent
Unexpected output or error message
Read the error message completely — the solution is often in the last 3 lines. Search the exact error: grep -r "[error text]" in the repo to find prior occurrences
Check GitHub issues for the tool: gh issue list --repo owner/repo --search "[error keyword]". Check Stack Overflow
Simplify the approach. If the complex one-liner fails, break it into 3 sequential commands. If the specialized tool fails, use a more basic tool with more steps
Data integrity concern (wrong output, silent failure)
Verify with a manual check: compare output against a known-correct baseline. Add assertions: `[command]
grep -q "[expected]" && echo "OK"
Hard failure boundary: If 3 different approaches all fail, STOP. Do not iterate infinitely. Log what was tried, capture the error output, and report the blocking issue with full context. Move to the next independent task rather than blocking all progress on one failure.
Blocked by infrastructure? → DevOps Engineer → Cloud Architect
Auth/security concern? → Security Engineer → Compliance Officer
Data contract dispute? → System Architect → CTO Advisor
Cross-team dependency blocking? → System Architect → Project Manager
Proactive Triggers
Trigger
Response
"WebSocket connections dropping after every deploy"
Implement connection draining: on SIGTERM, stop accepting new WebSocket connections (server.close() in ws), send close frames (code 1001) to existing clients with a Retry-After header equivalent, and wait for in-flight messages to complete before process exit. Add a readiness probe that returns 503 while draining so the load balancer stops routing new traffic.
"SSE clients getting 502 after exactly 60 seconds"
Proxy timeout is killing the stream. Configure proxy_read_timeout 300s; in nginx or increase the ALB idle timeout to 120s+. Send SSE keepalive comments (: heartbeat\n\n) every 15–30 seconds to prevent idle connection drops. Disable response buffering with proxy_buffering off; and set the X-Accel-Buffering: no response header from the application.
"Memory leak in production — WebSocket connections growing unbounded over days"
Set MAX_CONNECTIONS cap at server startup. Add idle timeout (5 min) that terminates inactive connections after two missed heartbeats. Implement per-user connection deduplication — reject new connections from the same user if one already exists. Monitor wss.clients.size in health checks and alert on growth trends.
"Broadcast to 50K connections causes event loop lag and request timeouts"
Shard connections across multiple processes/instances (1 per CPU core). Use Redis pub/sub for cross-instance fan-out. Batch broadcasts: accumulate 50ms of events, send once instead of per-message. Consider SSE instead of WebSocket for one-way broadcasts — lower overhead per connection, no per-frame ACK overhead.
"Client reconnects after 30s disconnect and misses events — data loss"
Assign monotonically increasing sequence numbers to each broadcast event. Store the last N events (e.g., 1,000) in a ring buffer per connection/channel. On reconnect, client sends { lastSeq: 42 } and server replays events 43+. For disconnects longer than the ring buffer, fall back to a REST endpoint for historical data.
"WebSocket upgrade fails with 426/400 behind a proxy or load balancer"
Ensure the proxy forwards WebSocket upgrade headers. In nginx: proxy_set_header Upgrade $http_upgrade; and proxy_set_header Connection "upgrade";. In AWS ALB: the listener protocol must be HTTP/HTTPS (not HTTP/2, which doesn't support protocol upgrade). Verify the backend route matches the WebSocket path exactly.
"Zombie connections — server thinks 5K clients are connected, only 2K are actually reachable"
Implement application-level ping/pong heartbeat (30s interval, 2 missed pongs = terminate). TCP keepalive defaults to 2+ hours — always do application-level heartbeats. On the client side, use EventSource auto-reconnect (SSE) or implement exponential backoff reconnection with jitter (WebSocket).
"Fan-out broadcast storm — one incoming event triggers cascading broadcasts that amplify"
Rate-limit broadcasts per room/channel (e.g., max 10/sec). Use a debounce pattern: if the same event type fires within 100ms, coalesce into one broadcast. Attach a broadcastId UUID to each message and deduplicate at the receiving end. Never broadcast raw upstream events without sanitizing/aggregating first.
Anti-Patterns
1. No Database Connection Pooling
What it looks like: Every request opens a new database connection instead of reusing from a pool. Under 500 concurrent requests, the database connection limit (typically 100) is exhausted. New requests queue, time out, and cascade into 503 errors. Instead of fixing the pool, teams scale up the database tier.
Cost: $5,000-$50,000/month in unnecessary database scaling.
Fix: Configure connection pool size (CPU cores × 2) + 1 for async frameworks. Use PgBouncer or RDS Proxy for serverless/lambda. Monitor pool utilization as a first-class metric with alerts at >80%.
2. Missing Request Timeouts
What it looks like: An upstream service or database call hangs indefinitely with no timeout. Threads accumulate, the event loop blocks, health checks fail. Load balancers mark instances unhealthy, traffic shifts to remaining instances which then also hang — cascading failure.
Cost: $10,000-$100,000 in cascading outages and lost transactions.
Fix: Set explicit timeouts on every external call (HTTP: 10s, DB: 5s, cache: 1s). Implement circuit breakers (>50% failures in 30s → open). Configure graceful shutdown with a maximum drain deadline.
3. Cross-Platform Environment Variable Loading
What it looks like:process.env vs. os.environ vs. dotenv priority order varies between Docker, local, and deployment platforms. A variable works locally but is absent in production with no startup error.
Fix: Always log which env source is active on startup. Use strict schema validation (Zod, Pydantic Settings, dotenv-safe) that fails fast on missing or malformed variables.
4. JWT Timestamp Precision Mismatch
What it looks like: JWT exp claim requires UNIX timestamp in seconds. Python's datetime.timestamp() returns float with microseconds. Tokens are rejected as expired when they shouldn't be.
Fix: Truncate with int(datetime.utcnow().timestamp()). Use python-jose or PyJWT which handle this automatically if you pass datetime objects.
5. CORS Preflight Breaking Auth
What it looks like:OPTIONS preflight requests don't carry auth headers. Auth middleware rejects them with 401, and the browser never sends the real POST/PUT/DELETE.
Fix: Auth middleware must skip OPTIONS requests. Return 200 with appropriate CORS headers for preflight. Test with curl -X OPTIONS from the frontend origin.
6. Migration Ordering by Timestamp
What it looks like: Alembic/Drizzle/Knex uses file timestamps for ordering. Two developers create migrations simultaneously — the merge produces an ordering that breaks foreign key constraints.
Fix: Review migration ordering after every merge. Use alembic history / npx drizzle-kit check to verify the chain. CI gate: run migrations against a fresh database before merge.
7. Shallow Health Checks
What it looks like:/health returns 200 even when the database is unreachable, Redis is down, or the message queue is disconnected. Load balancer routes traffic to a dead instance.
Fix:/health (liveness): is the process alive? Lightweight. /health/ready (readiness): ping DB, cache, and message queue. Return 503 on dependency failure so the load balancer stops routing traffic.
8. ORM SELECT * with Large Columns
What it looks like: ORMs generate SELECT * from model definitions, pulling every column including TEXT/BLOB fields. Query latency spikes as row width increases. Memory usage balloons for large text fields.
Fix: Explicitly .select() or columns= in every query. Audit with EXPLAIN ANALYZE on top-10 queries. Set max_row_size limits at the connection level.
9. PII in Plaintext Logs
What it looks like:console.log(user) or logger.info({ body: req.body }) during debugging ships passwords, credit cards, and SSNs into CloudWatch/Datadog/ELK. Logs are accessible to every engineer with production access and discoverable in breach investigations.
Cost: $25,000-$500,000 in GDPR/CCPA fines, forensic audit costs, and customer notification.
Fix: PII redaction in the logging pipeline (pino-redact, logstash filter). Never log request bodies by default in production. Run automated scheduled scans for PII patterns in log storage.
10. Synchronous Long-Running Tasks in HTTP Handlers
What it looks like: File uploads, PDF generation, or bulk emails run inside the request handler for 30-120 seconds. All available workers are busy on long tasks. Health checks time out. Load balancer marks every instance unhealthy — traffic death spiral.
Cost: $10,000-$50,000 in production outages and lost in-flight transactions.
Fix: Offload any task >500ms to a background job queue (BullMQ, Celery, Sidekiq, SQS). Return 202 Accepted with a Location header to a status-check URL. Configure web server timeouts and max request duration limits.
11. Backward-Incompatible Database Migrations
What it looks like: A migration drops or renames a column while old application code still runs in a rolling deployment. Instances reference the dropped column — which no longer exists — and error on every request touching that path.
Cost: $15,000-$75,000 in deployment outages, data corruption, and complex rollback procedures.
Fix: Add columns as nullable first. Deploy code that writes to both old and new. Deploy code that reads from new only. Drop old column in a follow-up migration. Never rename/drop a column in the same migration that adds its replacement.
Error Decoder — War Stories from the Trenches
(STANDARD)
When backend services go wrong, they go wrong in predictable ways. Here are the most common failure signatures, their root causes, and the fix you'll reach for after you've been burned once.
Symptom
Root Cause
Fix
Lesson
Service responds instantly at 10 req/s, times out at 100 req/s — health checks fail, load balancer removes all instances
Connection pool exhausted because connections aren't returned after use. Every request opens a new connection, hits the pool limit, and blocks waiting. All workers are idle waiting for connections that never come
Set pool.max = 20 and pool.idleTimeoutMillis = 30000. Audit every query path — every connection acquired must be released in a finally block. Add metrics on pool utilization: alert at >80%
Connection pools fail silently. Your app doesn't crash — it just waits forever. Pool exhaustion looks exactly like a deadlock in your monitoring dashboard
Cache stampede kills database at midnight UTC — every cache key expires simultaneously, all traffic hits the DB at once
All cache entries set with the same TTL from the same base time (e.g., all cached at deploy time with 86400s TTL). At expiry, 10K requests simultaneously compute and repopulate — database melts
Add random jitter to every TTL: ttl = baseTTL * (0.8 + Math.random() * 0.4). Implement request coalescing: only one request recomputes while others wait. Use Redis SET NX for distributed lock on cache miss
Deterministic TTLs create synchronized thundering herds. One second of randomness in TTL distribution spreads the recompute load over hours instead of microseconds
All timestamps off by 5 hours in production, correct in development — billing dates, event logs, and audit trails are wrong
Server timezone is UTC, database connection uses local timezone, application code calls new Date() without offset. Development machine is in the same timezone as the DB — the bug is invisible locally
Store all timestamps as UTC in the database. Set TZ=UTC in the application environment. Use DateTime.utcnow() / new Date().toISOString() for all timestamps. Parse incoming timestamps with explicit timezone
Timezone bugs are Heisenbugs — they only reproduce when the server, database, and developer are in different timezones. UTC-everywhere is the only defense
SIGTERM causes 500 errors for in-flight requests — graceful shutdown is configured but requests are dropped mid-response
The process receives SIGTERM, stops accepting new connections, but doesn't wait for existing requests to complete. Load balancer hasn't deregistered the instance yet — new requests arrive and get refused while old ones are killed
In the shutdown handler: (1) stop accepting new connections, (2) server.close() with a 30s timeout, (3) await Promise.allSettled(activeRequests), (4) close DB pools and exit. Set terminationGracePeriodSeconds in k8s to 5s longer than your longest expected request
Graceful shutdown isn't a boolean — it's a sequence. Every step in the sequence matters. If k8s sends SIGKILL before your handler finishes, you lose in-flight transactions
File descriptor leak — service runs fine for days, then EMFILE: too many open files crashes everything
HTTP keep-alive connections, database connections, or log file handles opened but never closed. Each leaked FD stays open until the process dies. After 1-2 weeks, the process hits the ulimit ceiling
Set agent.keepAlive = true with maxSockets limit. Audit every fs.createReadStream and client.connect() — each needs a corresponding close/destroy. Monitor process._getActiveHandles().length in health checks. Set ulimit -n 65536 as a safety net, not a fix
File descriptor leaks are the stealthiest resource leak. No memory spike, no CPU spike — just a sudden crash after days of perfect operation. The only fix is auditing every resource acquisition path
Rate limiter blocks legitimate traffic from corporate proxies — entire offices can't access the API
Rate limiting by IP address when thousands of users share a single NAT gateway IP (office building, university, mobile carrier CG-NAT). The shared IP exceeds the limit while individual users are well under it
Rate limit by authenticated user ID as primary key, with IP as secondary fallback for unauthenticated endpoints. Add X-Forwarded-For trust for proxy headers. Use token bucket algorithm with per-key granularity
IP-based rate limiting is broken architecture for any service with authenticated users. Corporate NAT gateways, mobile carriers, and VPNs make IP addresses meaningless as user identifiers
Gotchas
Gotcha
Cost
Fix
Connection pool exhaustion — service hangs under load, all workers idle, health checks fail
$30K-$80K in outage response
Audit every connection path — always release in finally block, set pool max and idle timeout, monitor pool utilization with alerts at >80%
N+1 queries in production — ORM lazy-loads related entities inside a loop, 1 page load triggers 10,001 queries
$15K-$50K in degraded UX and infra costs
Use eager loading (.Include(), .prefetch_related(), JOIN FETCH), enable query logging in dev, add n_plus_one detection linter, test with realistic data volumes
Timezone bugs — all timestamps offset by hours in production, billing dates and audit trails wrong
$50K-$200K in data corruption and compliance
Store ALL timestamps as UTC, set TZ=UTC at application level, use date.toISOString() / DateTime.utcnow(), parse incoming timestamps with explicit timezone
Verification
Run npm test / pytest / go test ./... — all tests pass, no regressions
Run linter: eslint . / ruff check . / golangci-lint run — zero new issues
Run type checker: tsc --noEmit / mypy . — zero type errors
Start the service and hit the health endpoint: curl <http://localhost:${PORT}/health> returns 200
Verify all new endpoints have integration tests covering success, auth failure, validation error, and not-found cases
Verification Guardrails
Before delivering work, verify: self-check against What Good Looks Like, no broken references, continuity with State Log, no fabricated APIs/versions/capabilities, Error Recovery paths exercised, cross-skill dependencies satisfied. If any fail, revise before delivering.
