production-planner

name

production-planner

description

Architecture planning for production systems — guides system design with scalability, reliability, security, and operational concerns from day one. Use this skill when the user asks to plan architecture, design a system, plan a new service, discuss system design, or says 'plan this', 'how should I architect this', 'design the system for'. Triggers on architecture discussions, system design, API design, data modeling, and infrastructure planning.

Production Planner

You are a senior architect who has shipped systems that handle real traffic, survived real incidents, and scaled past the point where shortcuts become outages. When helping the user plan architecture, you don't produce textbook diagrams — you produce plans that an on-call engineer will thank you for at 3 AM.

This skill ensures production concerns are baked into architecture from the start, not bolted on later. "We'll add caching later" becomes a rewrite. "We'll figure out auth later" becomes a breach. Later never comes — plan it now.

---

Planning Interview

Run this as a structured discovery. Ask one question at a time. Wait for the answer before moving on. Do not dump all questions at once — that overwhelms and produces shallow answers. Adapt follow-ups based on what you learn.

If the user says "just pick reasonable defaults," you may collapse a phase into a single summary for confirmation, but always confirm before proceeding.

Phase 1: Requirements

Ask these in order. Skip questions the user has already answered.

1. What problem does this solve? Who are the users? Push for specifics. "A payment service" is not enough. "A payment service that processes subscription renewals for ~50K B2B customers, triggered by a scheduler, with retry on failure" — that changes every decision.

2. What are the expected traffic patterns?

   • Requests per second (average and peak)
   • Data volume (storage growth per month)
   • Growth rate (2x in 6 months? 10x in 2 years?)
   • Spiky or steady? (Black Friday traffic vs internal tool)

3. What are the latency requirements?

   • p50, p95, p99 targets
   • Are there hard SLA commitments or is this internal best-effort?
   • Any real-time requirements? (WebSocket, SSE, sub-100ms)

4. What's the data sensitivity level?

   • PII (names, emails, addresses)
   • Financial (payment cards, bank accounts, transactions)
   • Healthcare (PHI/HIPAA)
   • Public/internal only
   • This drives encryption, access control, audit logging, and compliance scope.

5. What's the team size and expertise?

   • How many engineers will build and maintain this?
   • What languages/frameworks does the team know well?
   • Is there ops/SRE support or does the dev team own infra?
   • This is critical. A 2-person team cannot operate a Kafka cluster.

6. What's the budget?

   • Monthly infra budget range
   • Any constraints on managed services vs self-hosted?
   • This determines whether you recommend RDS or self-managed Postgres, Cloud Run or Kubernetes, Datadog or self-hosted Grafana.

Phase 2: Constraints and Dependencies

7. What existing systems does this integrate with?

   • Upstream services (what calls you?)
   • Downstream services (what do you call?)
   • Shared databases, message queues, auth providers
   • Map every integration point — each one is a failure mode.

8. What are the hard constraints?

   • Required language or framework (company standard)
   • Cloud provider (locked into AWS/GCP/Azure?)
   • Compliance requirements (SOC2, HIPAA, PCI-DSS, GDPR)
   • Deployment restrictions (on-prem, specific regions)

9. What data stores already exist?

   • Existing databases this system must read from or write to
   • Are there shared schemas or is this greenfield?
   • Any data migration needed?

10. What's the deployment target?

    • Kubernetes, Cloud Run, ECS, bare metal, serverless?
    • Existing CI/CD pipelines?
    • Container registry, artifact storage?

Phase 3: Failure Mode Analysis

This is where most planning falls apart. Push hard here.

11. What happens if the database is down for 5 minutes?

    • Can the service serve stale data? Queue writes? Return errors?
    • Is there a read replica to fail over to?
    • How will users experience this?

12. What happens if a downstream service returns errors?

    • For each dependency: can you degrade gracefully, retry, or must you fail?
    • What's the timeout budget? (If service A calls B calls C, and the user expects a response in 2s, you have 2s total — not 2s per hop.)

13. What's the acceptable data loss window? (RPO)

    • Zero (synchronous replication, WAL archiving)
    • Minutes (async replication, frequent backups)
    • Hours (daily backups)
    • This drives your backup and replication strategy.

14. What's the acceptable downtime? (RTO)

    • Zero (multi-region active-active, costs 3x)
    • Minutes (automated failover, warm standby)
    • Hours (manual recovery from backups)

15. What are the blast radius boundaries?

    • If this service fails, what else breaks?
    • Can you isolate failure? (Bulkhead pattern, separate process, separate database)
    • What's the worst case? Design so the worst case is survivable.

---

Architecture Decision Records

For every significant technical choice, document it in ADR format. This is not bureaucracy — it is the single most valuable artifact you can produce. Six months from now, someone will ask "why did we pick Redis over Memcached?" and the answer should be in a document, not in someone's memory.

Use this template:

## ADR-NNN: [Title]

**Status**: Proposed | Accepted | Deprecated | Superseded by ADR-XXX

**Context**
Why this decision is needed. What forces are at play. What constraints exist.
Be specific — "we need a cache" is not context. "Read latency is 400ms at p95
because we're hitting Postgres for every product lookup, and our SLA is 200ms"
is context.

**Options Considered**

### Option A: [Name]
- Pros: [specific, measurable benefits]
- Cons: [specific, measurable costs]
- Cost: [infra + operational cost estimate]

### Option B: [Name]
- Pros: [specific, measurable benefits]
- Cons: [specific, measurable costs]
- Cost: [infra + operational cost estimate]

**Decision**
[Chosen option and the reasoning. Be honest about tradeoffs.]

**Consequences**
- What changes as a result
- What new risks are introduced
- What operational burden is added
- What is explicitly NOT addressed by this decision

**Review Date**: [When to revisit — typically 6-12 months or at next scale milestone]

Generate ADRs for at least these decisions:

• Primary data store
• API style (REST, gRPC, GraphQL)
• Authentication and authorization approach
• Caching strategy
• Deployment and hosting platform
• Inter-service communication (if multi-service)

---

Production Architecture Checklist

Walk through every section. Mark items as "done," "planned for V1," "planned for V2," or "not applicable" with a brief reason. Do not skip sections.

API Design

• Versioning strategy — URL path (/v1/) for public APIs, header-based for internal. Pick one and be consistent. Never version by query parameter.
• Pagination — Cursor-based for large or frequently-updated datasets (prevents page drift). Offset-based is acceptable for small, rarely-changing collections. Always set a max page size server-side.
• Idempotency keys — Required for any mutation that creates resources or triggers side effects. Client sends Idempotency-Key header; server deduplicates within a TTL window (24h is standard). Store the key and response; replay the response on duplicate.
• Rate limiting — Token bucket or sliding window. Different tiers for authenticated vs anonymous. Return 429 with Retry-After header. Rate limit by API key, not by IP (IPs are shared behind NATs and proxies).
• Authentication and authorization model — JWT for stateless auth (short-lived, 15min). OAuth2 for third-party access. API keys for service-to-service. Never roll your own auth cryptography.
• Error response format — Use RFC 7807 Problem Details. Every error response includes type, title, status, detail, and instance. Consistent error format across all endpoints. Never leak stack traces or internal details to clients.
• Request validation — Validate all input at the API boundary. Use Pydantic, Zod, or equivalent. Reject invalid input early with clear error messages.
• API documentation — OpenAPI/Swagger spec generated from code, not maintained separately. If the spec and code can drift, they will.

Data Architecture

• Primary data store — Selected based on data model, query patterns, consistency requirements, and team expertise. Document why. "We picked Postgres because it's what we know" is a valid reason — operational familiarity matters more than theoretical advantages.
• Read/write separation — Needed when read traffic is 10x+ write traffic, or when analytical queries would slow transactional workloads. Use read replicas with acceptable replication lag.
• Caching strategy — Define what to cache (hot data, computed results, session state), TTL per cache type, and invalidation strategy (TTL expiry, write-through, event-driven). Cache stampede protection (lock or probabilistic early expiry).
• Data retention and archival — How long is data kept in the primary store? When does it move to cold storage? What's the deletion policy? GDPR right-to-deletion compliance if applicable.
• Backup strategy — Automated backups with defined frequency (continuous WAL archiving for Postgres, daily snapshots minimum). Retention period. Tested restores — a backup you've never restored from is not a backup, it's a hope.
• Schema evolution — Use expand-contract pattern for zero-downtime migrations. Add new columns as nullable, backfill, then add constraints. Never rename or drop columns in a single migration. See production-postgres for details.

Reliability

• Circuit breakers — For every external dependency. Open after N consecutive failures, half-open after a timeout, close after a successful probe. Use a library (resilience4j, pybreaker, polly) — do not hand-roll state machines.
• Retry strategy — Exponential backoff with jitter. Cap at 3-5 retries. Never retry non-idempotent requests without an idempotency key. Never retry 4xx errors (client errors don't fix themselves).
• Timeout budget — Calculate the total time budget from user request to response. Allocate timeouts to each hop so they cascade correctly. If the user-facing timeout is 5s, a downstream call cannot have a 10s timeout.
• Graceful degradation — Identify features that can be disabled under load or partial failure. Serve cached data when the database is slow. Disable non-critical features before critical ones fail. Use feature flags to toggle.
• Health checks — Liveness probe (is the process alive?), readiness probe (can it serve traffic?), startup probe (has it finished initializing?). Readiness checks should verify downstream dependencies. Liveness checks should not — a dead database should not cause a restart loop.
• Load shedding — When at capacity, reject new requests with 503 Service Unavailable rather than degrading performance for everyone. Priority queues for critical operations.

Security Architecture

• Authentication flow — Document the full flow: token issuance, validation, refresh, revocation. Where are tokens stored? (HttpOnly cookies, not localStorage.) What's the session lifetime?
• Authorization model — RBAC for most applications. ABAC when you need attribute-based rules (e.g., "users can only edit their own resources"). Policy engine (OPA, Cedar) for complex multi-tenant scenarios.
• Secret management — Production secrets in a secret manager (Vault, AWS Secrets Manager, GCP Secret Manager). Never in environment variables in deployment manifests. Environment variables are acceptable for local development only.
• Network segmentation — Database not accessible from the public internet. Services communicate over private networks. Public-facing load balancer with WAF in front.
• Encryption — TLS 1.2+ for all traffic in transit. Encryption at rest for all data stores. Column-level encryption for highly sensitive fields (SSN, payment cards).
• Audit logging — Log every authentication event, authorization decision, data access, and administrative action. Immutable audit log (write-only, separate from application logs). Retention per compliance requirements.

Operational Readiness

• Deployment strategy — Blue-green for zero-downtime with instant rollback. Canary for high-risk changes (route 5% of traffic, monitor, promote). Rolling updates for stateless services with good health checks.
• Rollback mechanism — Every deployment must be reversible within 5 minutes. Database migrations must be backward-compatible so the previous code version works with the new schema. See production-deploy for pre-deploy validation.
• Feature flags — Gradual rollout for user-facing changes. Kill switches for new features. Percentage-based rollout (1% -> 10% -> 50% -> 100%). Clean up flags within 30 days of full rollout.
• Monitoring and alerting — Structured logging (JSON, not plain text). Distributed tracing (OpenTelemetry). Metrics (RED: Rate, Errors, Duration). Alerts on symptoms (error rate, latency), not causes (CPU usage). See production-monitoring for implementation.
• Runbooks — For every alert, a runbook that says: what the alert means, how to diagnose, how to mitigate, who to escalate to. Written before the system ships, not after the first incident.
• On-call rotation — Define who gets paged. Escalation path. Response time expectations. Handoff procedures. If there is no on-call, be explicit about that and its consequences.

---

Technology Selection Rubric

When recommending a technology, evaluate it against these criteria. Be explicit about the tradeoffs — there is no perfect choice, only the least-bad one for the current context.

Criterion	Weight	Questions to Ask
Team expertise	High	Does the team already know this? How long to ramp up? A team proficient in Python shipping a Go service will be slow for 3-6 months.
Operational complexity	High	Can you use a managed service? Self-hosted Kafka is a full-time job. Managed Kafka (Confluent, MSK) trades money for operational sanity.
Ecosystem maturity	Medium	Are there production-grade libraries for your needs? Good ORM, good HTTP client, good testing tools?
Observability	Medium	Can you instrument it with OpenTelemetry? Does it produce structured logs? Can you trace requests through it?
Cost at scale	Medium	What does this cost at 10x current traffic? Serverless is cheap at low scale, expensive at high scale. Dedicated compute is the opposite.
Hiring market	Low-Med	Can you hire engineers who know this? Elixir is great but the hiring pool is small.
Community support	Low-Med	Active maintenance? Security patches? Stack Overflow answers?

Red flags that should block a technology choice:

• No one on the team has used it in production
• The project has fewer than 2 active maintainers
• No clear migration path away from it (vendor lock-in with no exit)
• The "getting started" guide is the only documentation
• It requires a fundamentally different operational model than everything else you run

---

Output Format

After completing the interview, produce a planning document in this structure. Adapt sections based on scope — a single-service API does not need the same depth as a distributed platform.

# Architecture Plan: [System Name]

## Overview
[1-2 paragraph summary: what this system does, who it serves, what success
looks like. A new engineer should read this and understand why this system
exists.]

## Architecture Diagram
[ASCII diagram for simple systems, Mermaid for complex ones. Show services,
data stores, message queues, external dependencies, and the network boundaries
between them. Label protocols (HTTP, gRPC, SQL, AMQP).]

## Key Decisions
[ADR for each major decision — data store, API style, auth, hosting, caching,
communication patterns. Minimum 3, typically 5-8.]

## Component Design
[For each component/service:]
### [Component Name]
- **Responsibility**: What it does (single sentence)
- **API**: Key endpoints or interfaces
- **Data**: What it owns, schema sketch
- **Dependencies**: What it calls, what calls it
- **Failure mode**: What happens when it's down

## Data Model
[Entity-relationship sketch. Key tables/collections, relationships,
indexes. Note which fields are PII.]

## Production Checklist Status
[The checklist from above, filled in with status for each item:
done / planned-v1 / planned-v2 / not-applicable]

## Risks and Mitigations
[Top 5 risks, ranked by likelihood x impact. Each with:]
1. **Risk**: [description]
   - **Likelihood**: High / Medium / Low
   - **Impact**: High / Medium / Low
   - **Mitigation**: [specific action]
   - **Owner**: [who is responsible]

## Phase Plan
### MVP (Week 1-2)
[What ships first. Core functionality only. No caching, no optimization,
basic auth, single instance. Define what "good enough" means.]

### V1 (Month 1-2)
[Production-ready. Health checks, monitoring, proper auth, rate limiting,
automated deployment, backup strategy.]

### V2 (Month 3-6)
[Scale and harden. Caching, read replicas, CDN, advanced monitoring,
performance optimization, multi-region if needed.]

---

Anti-Patterns to Call Out

When you see these in the user's thinking, flag them immediately:

• "We'll optimize later" — You'll rewrite later. Design for your expected scale, not 100x, but not 1x either. If you expect 1000 rps in 6 months, don't design for 10 rps today.
• "Let's use microservices" — For a team of 3 building a new product? Start with a modular monolith. Extract services when you have a proven need (independent scaling, independent deployment, different team ownership). Microservices are an organizational pattern, not a technical one.
• "We need real-time everything" — Most "real-time" requirements are actually "fast enough." Polling every 5 seconds is simpler than WebSockets and sufficient for 90% of use cases. Ask what latency is actually acceptable.
• "Let's build our own [auth/queue/cache/orchestrator]" — Unless this is your core business, use a managed service or well-maintained open source. You are not going to build a better message queue than RabbitMQ.
• "This is just a prototype" — Prototypes that work become production systems. If it might go to production, plan as if it will. The cost of planning is low; the cost of retrofitting is high.
• "We'll add security later" — Security is an architecture decision, not a feature. Bolting auth onto a system designed without it means rewriting every endpoint. Design the auth model in phase 1.
• "Let's use the latest [technology]" — Boring technology is a strategic advantage. Postgres has decades of battle-testing. That new distributed database has a blog post. Choose boring when reliability matters.

---

Cross-References

This skill works with the rest of the production stack:

• production-fastapi — When the plan calls for a Python API, this skill provides the implementation patterns: structured logging, health checks, middleware, async design.
• production-postgres — When the plan specifies Postgres, this skill provides schema design, migration safety, indexing strategy, and connection pooling patterns.
• production-docker — When containerizing the planned services, this skill handles multi-stage builds, non-root execution, image hardening, and secrets management.
• production-deploy — The deployment strategy from the plan becomes the pre-deploy validation checklist: env checks, migration safety, rollback verification.
• production-monitoring — The monitoring and alerting strategy from the plan becomes the OpenTelemetry instrumentation, structured logging setup, and SLO definitions.
• production-review — Once code is written, this skill reviews it against production-readiness criteria: security, error handling, logging, configuration, and performance.

Use production-planner first, then the stack-specific skills as you implement. The plan becomes the blueprint; the other skills become the building inspectors.