| name | agentic-engineering-audit |
| description | Audit an AI agent or LLM-powered system against the agentic-engineering discipline. Scores Harness, Context, Tool Design, Skills, Memory, Evals, Model Selection, Failure Modes, and Adversarial Collaborator patterns, then prescribes three highest-ROI fixes. Use when a user wants to review an agent architecture, diagnose why an agent is unreliable or expensive, prepare a production launch, critique an LLM system design doc, or ask whether an agent is production-ready. |
Agentic Engineering Audit
Score an AI agent across nine pillars, then prescribe three fixes with concrete impact and effort estimates.
Workflow
- Ingest the target:
- GitHub repo URL: clone, then read entry points, agent loop, tool definitions, prompts, and any eval files.
- Pasted code: read into a working directory.
- Live URL: interact in 3–5 turns covering happy path plus one edge case.
- Design doc: read and quote it directly in the scorecard.
- Confirm the agent's one-sentence job and (if known) volume, latency target, monthly cost budget, known failure modes, team size.
- Score each of the nine pillars 0–5 with concrete evidence (file paths, line numbers, transcript quotes).
- Compute a weighted average — Harness, Context, Tool Design, and Evals weighted ×1.5 — and map to a letter grade.
- Pick exactly three fixes maximizing (impact × feasibility) / cost. Never list more.
- Produce the audit document below and share it.
Pillar Rubric
| # | Pillar | "5" looks like | Common 0–2 failure |
|---|
| 1 | Harness | Explicit step caps, budget caps, retry policy, structured logging, replay capability | Naked while True with no cap |
| 2 | Context | Deliberate per-turn assembly from named sources with token budgets | Whole conversation history every call |
| 3 | Tool Design | Crisp names, typed args, idempotent semantics, clear error returns, docstrings the model can use | Free-form string returns, silent failures |
| 4 | Skills | Reusable capabilities packaged as discoverable, composable units with their own prompts and tests | Logic inlined in one mega-prompt |
| 5 | Memory | Explicit memory layers (session / project / user / org) with read/write rules and eviction | "Memory" = stuffing prior turns into context |
| 6 | Evals | Versioned eval set covering happy path, edges, regressions; runs in CI | Vibes-based testing in a notebook |
| 7 | Model Selection | Right model per step with explicit fallbacks | One frontier model for every call |
| 8 | Failure Modes | Handles tool timeout, model refusal, hallucinated args, infinite loops, partial completion | Single try/except that swallows everything |
| 9 | Adversarial Collaborator | A critic/judge step pushes back on the primary model | Model marks its own homework |
Grade Mapping
- 4.5+ → A — production-ready
- 3.5–4.4 → B — ship-with-caveats
- 2.5–3.4 → C — beta, expect incidents
- 1.5–2.4 → D — prototype only
- <1.5 → F — pre-prototype
Output Format
Produce audit-<target>-<date>.md:
# Agentic Engineering Audit — <Target>
## Summary
<One paragraph + letter grade + headline weakness.>
## Scorecard
| # | Pillar | Score | Evidence |
|---|---|---:|---|
## Three Prescribed Fixes
1. **What:** <action sentence>
**Why:** lifts <pillar> from X to Y (~+Z points)
**How:** <concrete steps, code snippet if useful>
**Effort:** <hours>
2. ...
3. ...
## Evidence Appendix
<File paths, line numbers, transcript quotes that support each score.>
Quality Bar
- Specificity over diplomacy: cite file:line, not "context management could be improved."
- Recommendations are imperative: "Add a step cap of 25," not "consider limits."
- One example per pattern, not five.
- Exactly three fixes. The constraint is the point.
- Never grade above C without a passing eval suite in evidence.
