| name | production-readiness |
| description | Audit an agentic product against the 24-point Definition of Done before launch. Covers context, tools, permissions, reliability, evals, observability, security, cost, the Loop License, and measurement science (judge calibration, retrieval metrics, ground-truth provenance, drift, human oversight) — the minimum bar for production. Use whenever the user is preparing to launch / ship / deploy an agentic product, asks "is this production-ready," wants a pre-launch checklist, or is doing a code review before going live. |
Production Readiness — 24-Point Definition of Done
An agentic product is not production-ready until all 24 points pass. Each point catches a class of failures that has hit real products.
This is an audit checklist, not a feature list. Walk through it with the user; mark each as pass, gap, or N/A with explicit reasoning. Gaps must be closed or accepted with eyes open.
The paved road. Many of these points come satisfied out of the box if you run the AgenticProduct family reference stack — memory (AgenticMind), runtime & fleet ops (AgenticOps), evals & observability (AgenticPerformance), the model & cost plane (AgenticGateway), and Layer-8 red-teaming (AgenticAssurance). It's the fastest way to green, not a requirement — satisfy any point your own way (Principle 2). See the reference-stack skill.
The 24 points
Context and state
1. Context utilization < 40% in typical turn
Why: past 40% utilization, model recall degrades nonlinearly. The "dumb zone" begins.
Common gap: dumping conversation history into every turn instead of using selection / compaction.
2. State externalized (not living only in context window)
Why: in-context state evaporates on session boundary, restart, or compaction.
Common gap: "the agent will remember from the conversation" — it won't, reliably.
3. Compaction pipeline tested on long sessions
Why: compaction that drops critical state silently is worse than no compaction.
Common gap: built compaction, never tested on a session long enough to need it.
Tools and permissions
4. Destructive actions require explicit human approval
Why: Replit incident — agent wiped 1,200+ companies' data despite a "code freeze" prompt. Prompts don't enforce.
Common gap: "the prompt tells the agent not to delete production data" — insufficient.
5. Permissions enforced by code, not by prompt
Why: prompt injection is a real threat; LLM can be coerced; code cannot.
Common gap: OAuth scopes too broad; agent runs as superuser internally.
6. Tool execution sandboxed
Why: tool execution is an attack surface; treat it like any RPC exposed to untrusted input.
Common gap: agent has shell access with no sandbox.
Tenant isolation (if multi-tenant)
Skip this whole section if the product is single-tenant or deployed per customer. Otherwise every box must be checked — there is no partial cross-tenant isolation.
Why: an agent is a confused deputy — prompt-level isolation leaks. The first cross-tenant leak is a churn-and-lawsuit event, and retrofitting isolation is the migration nobody budgets for. See the tenant-isolation skill.
Common gap: a tenant-agnostic answer cache serving tenant A's response to tenant B; tenant_id passed as a tool argument the model can be talked into changing.
Reliability
7. Durable execution: pause/resume/retry works on killed process
Why: any agent running > 60s will hit a crash, restart, or wait — without durability, work is lost.
Common gap: "we tested the happy path" — production isn't the happy path.
8. Structured outputs validated by schema; assertions on critical path
Why: unvalidated outputs cause silent corruption; assertions catch it early.
Common gap: regex-parsing LLM responses; works in dev, fails in production edge cases.
9. Guardrails on input and output (minimum: PII, jailbreak, schema validation)
Why: defense in depth; multiple cheap guardrails beat one perfect one.
Common gap: no input guardrails; trusting the model to refuse.
Evals and observability
10. Eval set ≥ 50 examples per top-priority failure mode
Why: generic evals don't catch product-specific failures; 50 cases give enough signal to detect regression.
Common gap: 20 cases of "helpfulness" — not enough, not the right thing to measure.
11. LLM-judges calibrated against human labels (TPR/TNR tracked)
Why: uncalibrated judges produce meaningless scores; teams stop trusting evals and revert to vibe.
Common gap: built a judge; never measured if it agrees with humans.
12. CI blocks deploys on eval regression; 100% production traces logged
Why: evals without enforcement are theater; traces are the only way to debug failures after launch.
Common gap: evals exist but don't gate merges; tracing is sampled, missing the failures.
Security and identity
13. Lethal-trifecta check performed and documented
Why: private data × untrusted content × external comms is an exfiltration channel — injected content reads a secret and ships it out. Simon Willison's lethal trifecta: the deployment check every agent must pass.
Common gap: all three legs live and unmitigated because no one drew the diagram; "the model won't do that" stands in for a mitigation.
14. MCP tool definitions pinned; servers allow-listed; OAuth 2.1 scoped tokens
Why: an approved tool description can mutate after you approve it; a forwarded or over-scoped token turns the agent into a confused deputy. The supply chain is part of the attack surface.
Common gap: installing a community MCP server by URL and trusting its description forever; minting broad OAuth scopes "to be safe."
Cost
15. Per-run token / cost ceiling enforced in code
Why: without a code-level ceiling, one bad loop is an unbounded invoice. Cost is a reliability property, not just a finance report.
Common gap: watching cost in a dashboard after the fact instead of capping it in the request path.
Unattended operation (L3+) — the Loop License
These four bind only when the agent runs unattended at L3+ (finds its own work, loops without a human in each turn). Full treatment: STANDARD.md Part IV + templates/loop-license/CHECKLIST.md.
16. Loop License held (all six gates)
Why: an unattended loop with no license is a factory with no quality control — it ships whatever it produces at machine speed.
Common gap: having evals and a cost cap but no kill switch or declared blast radius, so nothing can stop or bound a run in flight.
17. Stop conditions & fail paths declared and enforced
Why: "no declared way to stop" is the defining L4 failure.
Common gap: budgets exist but there is no escalation-after-N — the loop burns the whole budget retrying a doomed step.
18. Independent verification
Why: self-verification shares the writer's blind spots — a pass tells you nothing new.
Common gap: a single agent that writes and then "reviews" its own output in the same context.
19. Loop economics
Why: a loop that is cheap per call but rarely produces a verified result is expensive — only cost-per-verified-outcome shows it.
Common gap: measuring raw spend but never dividing by outcomes that actually passed verification.
Measurement science & human oversight
Items 20–23 deepen the eval bar wherever the relevant component exists; the oversight item binds at L3+. Full treatment: STANDARD.md Part V (measurement science).
20. Judge calibration for gating verdicts
Why: verbalized LLM confidence is systematically overconfident; an uncalibrated judge in a gate is a check that isn't one — it invalidates the Loop License for the levels it gates.
Common gap: trusting a judge's "95% confident" verbatim, with no anchored accuracy behind it.
21. Retrieval evaluated on its own terms
Why: retrieval and reasoning fail differently; an end-to-end number that conflates them cannot direct a fix.
Common gap: shipping an embedding-model swap because task evals "looked fine," silently dropping Recall@5.
22. Ground-truth provenance
Why: a golden set without provenance is unanchored — you don't know what its pass rate means.
Common gap: a "golden" set nobody can trace to a rubric version or a labeler.
23. Drift monitoring
Why: drift answers "when did my evals stop representing production?" — without it a green suite can be measuring the past.
Common gap: a golden set refreshed on a calendar, not when production actually moved.
(L3+) Human oversight as a program
Why: oversight is an operated program, not a checkbox; graduation must be reversible on regression.
Common gap: "human review" that only sees escalated hard cases, skewing the review-derived golden data.
Gate integrity
24. No safety gate silenced to pass CI
Why: a gate is trust-bearing only if green means the property holds, not that the check was silenced; disabling it removes the exact protection at the moment it fired. tsc passing is not a substitute for the no-unsafe-* family — any is assignable to everything by design, so the compiler waves it through (Canon 5, gate-integrity invariant).
Common gap: a flaky lint rule disabled repo-wide to unblock CI, silently blinding every file instead of the one that misfired.
Audit posture
When running this audit with the user:
- Walk through each point sequentially. Don't jump around.
- For each: pass / gap / N/A with reason. "N/A because we don't have destructive actions" is fine; "N/A because we don't think it matters" is not.
- Estimate effort to close each gap. Rank them by risk-adjusted cost.
- Make the explicit launch decision. "Launch with these N gaps accepted, address in week 1" is a valid choice. "Launch and hope" is not.
Post-launch hardening (after the 24 points)
Once the 24 are met, the next tier of investments:
- A/B testing infrastructure — compare new prompts/models/tools against current production
- Cost telemetry per request, per user, per agent type — find the expensive calls
- Failure runbooks — what to do when each named failure mode fires in production
- Eval set growth from production — sample weekly, label, add to eval set
- Model swap exercise — verify you can swap the model without breaking; trains the muscle
- Multi-region deployment — if availability matters
- Privacy controls and audit log — GDPR/CCPA compliance if you serve regulated users
Common "almost ready" patterns
Teams often have 21 of 23 covered. The common gaps are:
| Gap | Frequency | Severity |
|---|
| #11 (judge calibration) | Very common | High — invalidates eval scores |
| #5 (code-enforced permissions) | Very common | Critical — Replit-class incident risk |
| #2 (state externalized) | Common | High — first restart loses work |
| #7 (durable execution tested) | Common | High — failure on first real crash |
| #12 (CI gating) | Common | Medium — slow degradation over time |
If the user is short on time, prioritize closing these.
Output of this skill
When the audit completes, the user should have:
- A pass/gap/N/A scorecard across all 24 points
- Effort estimate to close each gap
- A risk-adjusted prioritization
- An explicit launch decision with accepted risks documented
- A 30-day hardening roadmap for post-launch
