| name | suede-ai-eval |
| description | Design AI evals that catch regressions before users do: rubrics, test cases, failure modes, acceptance gates, and AI-SPEC artifacts. |
Suede AI Eval
Make AI behavior testable before it becomes a vague product promise. No eval plan, no ship verdict: an AI feature without one cannot be graded ship.
The deliverable is an eval plan or coverage audit, not a model benchmark leaderboard. Keep it grounded in the actual product surface, user promise, data sources, prompts, tools, logs, tests, and failure modes available now.
Hard Gates
- No AI-SPEC → no eval plan. Write the one-paragraph spec first; cases written without a spec test nothing.
- No eval plan → no ship verdict. Never emit
ship or ship-with-caveats for an AI feature that lacks a failure-mode map and eval cases.
- A failure mode without an eval case, an owner, and a gate is uncovered — regardless of how unlikely it feels.
- A live surface that was never sampled gets the output stamped
source-only; do not present source-only review as runtime evidence.
- A model grading its own output is not evidence. LLM-as-judge scores count only after spot-checked agreement with a human-reviewed sample.
Source Truth
Inspect the current target before writing the eval. Do not evaluate from memory or product copy alone.
Read or verify:
- repo, branch, remote, dirty state, local instructions, and touched files;
- the AI surface: route, API, worker, prompt, system message, tool call, model config, retrieval path, classifier, agent loop, generated media path, or recommendation logic;
- user-facing promise, allowed claims, forbidden claims, safety boundaries, fallback behavior, and support path;
- input data, retrieval corpus, schemas, tool contracts, metadata, logs, telemetry, and persisted outputs;
- existing tests, fixtures, eval scripts, prompt snapshots, golden examples, analytics, bug reports, screenshots, or live/API readbacks.
When the surface is already live, sample real behavior with safe inputs and record exact commands or URLs. When live checks are not appropriate, mark the eval as source-only and name the missing runtime evidence.
Workflow
- Define the AI-SPEC. State the AI job in one paragraph: user, trigger, input, output, allowed sources, disallowed behavior, fallback, latency/cost expectation, and success signal.
- Map the failure modes. List the ways the AI can harm the user, product truth, rights/provenance, security, privacy, brand trust, cost, or workflow completion.
- Build the rubric. Score each failure mode with severity, likelihood, detectability, owner, gate, and required evidence.
- Write eval cases. Produce concrete pass/fail cases with inputs, setup data, expected output traits, forbidden output traits, and the reason the case exists.
- Set acceptance gates. Decide what blocks ship, what allows ship-with-caveats, and what can become follow-up work.
- Audit coverage. Compare existing tests, logs, metrics, and manual checks against the failure-mode map. Score coverage and infrastructure using the method under Tooling and Infrastructure below. Name every uncovered high-risk behavior regardless of the numeric score.
- Return the artifact. Give the AI-SPEC, rubric, eval table, coverage gaps, required tests, and next implementation step.
Eval Dimensions By System Type
Start the failure-mode map from the canonical dimensions for the surface's system type, then add product-specific failure modes on top. Always include safety (user-facing) and task completion (agentic) regardless of type.
| System type | Canonical dimensions |
|---|
| RAG / retrieval | context faithfulness, hallucination, answer relevance, retrieval precision, source citation |
| Multi-agent | task decomposition, inter-agent handoff correctness, goal completion, loop detection |
| Conversational | tone/style, safety, instruction following, escalation accuracy |
| Extraction / structured output | schema compliance, field accuracy, format validity |
| Autonomous / tool-using agent | safety guardrails, tool-use correctness, cost/token adherence, task completion |
| Content generation | factual accuracy, brand voice, tone, originality |
| Code generation | correctness, safety, test pass rate, instruction following |
For each dimension, assign a measurement approach before writing the eval case:
- Code-based: schema validation, required-field presence, performance thresholds, regex checks. Fast, deterministic, cheap to run in CI.
- LLM judge: tone, reasoning quality, safety-violation detection. Requires calibration against a human-reviewed sample before the score counts as evidence (see Hard Gates).
- Human review: edge cases, LLM-judge calibration itself, high-stakes sampling that cannot be automated yet.
Tooling and Infrastructure
Detect existing eval/tracing tooling before recommending anything new:
grep -rl "langfuse\|langsmith\|arize\|phoenix\|braintrust\|promptfoo\|ragas" \
--include="*.py" --include="*.ts" --include="*.toml" --include="*.json" . \
2>/dev/null | grep -v node_modules | head -10
If nothing is detected, these are the default starting points, not a mandate to install all four:
| Concern | Default | Why |
|---|
| Tracing / observability | Arize Phoenix | Open-source, self-hostable, framework-agnostic via OpenTelemetry |
| RAG eval metrics | RAGAS | Faithfulness, answer relevance, context precision/recall out of the box |
| Prompt regression in CI | Promptfoo | CLI-first, no platform account required |
| LangChain/LangGraph pipelines | LangSmith | Overrides Phoenix when the project is already in that ecosystem |
Reference dataset spec: minimum 10 examples to start, 20+ before treating coverage as production-grade. Composition: critical paths, edge cases, known failure modes, and adversarial inputs, not just happy-path samples. Labeling: domain expert where stakes are high, LLM judge with calibration otherwise. Start building the dataset during implementation, not after the feature ships.
Production monitoring split: classify every covered failure mode as either an online guardrail (catastrophic risk, runs on every request in the hot path, must be fast) or an offline flywheel check (quality signal, sampled batch, feeds the improvement loop, not latency-sensitive). Keep online guardrails minimal since each one adds latency to every request.
Coverage scoring: for each dimension, mark COVERED (implementation exists, targets the rubric behavior, actually runs), PARTIAL (exists but incomplete, not automated, or has known gaps), or MISSING (no implementation found). Audit infrastructure separately, ok/partial/missing: eval tooling is installed and actually called (not just a listed dependency), the reference dataset file exists and meets the spec above, a CI/CD command runs the eval suite, each planned online guardrail is implemented in the request path (not stubbed), and tracing is configured and wrapping the real AI calls. Score coverage = covered / total_dimensions × 100 and infra = (tooling + dataset + cicd + guardrails + tracing) / 5 × 100, then overall = coverage × 0.6 + infra × 0.4.
Eval Case Design
Prefer small, reviewable eval suites over theatrical giant scorecards.
Each case should include:
- case id: stable name, e.g.
rights-claim-refusal-001;
- scenario: why this input matters;
- input: prompt, API body, file, metadata, or retrieval query;
- setup: fixtures, user state, permissions, corpus state, or tool mocks;
- expected pass traits: observable traits, not hidden intent;
- forbidden traits: claims, sources, actions, or outputs that fail the case;
- grade lane: correctness, safety, rights/provenance, privacy, security, UX, cost, latency, or retrieval truth;
- gate: block, warn, monitor, or informational.
Use adversarial and mundane cases. Include at least:
- happy path that proves the intended value;
- ambiguous input that should ask a clarifying question or choose a safe fallback;
- forbidden claim or unsafe instruction that should be refused or rewritten;
- stale, missing, or conflicting source data;
- permissions or privacy boundary;
- expensive or looping behavior;
- regression case from a real bug, support note, or observed failure when available.
Worked Example — Support-Ticket Triage Classifier
Fictional AI surface: TicketSort, an LLM classifier that reads incoming support tickets and assigns category (billing, bug, feature-request, account-access, abuse) plus priority (P0-P3), then auto-routes P0/P1 tickets to an on-call queue and drafts a first-response reply for P2/P3. One full case per required category.
Case 1 — happy path
- case id:
ticketsort-happy-billing-refund-001
- scenario: proves the core value: a clearly-worded billing ticket gets the right category, a sane priority, and a usable draft reply without a human touching it first.
- input: ticket body — "I was charged $49 twice this month for my Pro plan, order #A88213 and #A88214. Can you refund the duplicate?"
- setup: user is an existing paid subscriber (
plan: pro, status: active); billing corpus has the duplicate-charge refund policy doc indexed; no prior tickets from this user in the last 30 days.
- expected pass traits:
category: billing, priority: P2, draft reply references both order IDs, cites the duplicate-charge refund policy, and asks the user to confirm before the refund is issued (does not promise the refund is already processed).
- forbidden traits: auto-approving or claiming the refund was issued; inventing an order ID not present in the ticket; routing to P0/P1 (no outage, no security issue).
- grade lane: correctness
- gate: block
Case 2 — ambiguous input
- case id:
ticketsort-ambiguous-vague-broken-001
- scenario: users often write vague tickets ("it's broken") with no reproduction detail — the classifier must not guess a confident category off one word.
- input: ticket body — "it's broken again, fix it" (no product area, no screenshot, no account reference in the message).
- setup: user has both a Pro subscription and three prior open tickets in different categories (billing, bug, feature-request) in the last 14 days — nothing in ticket history disambiguates which product area "it" refers to.
- expected pass traits:
category: needs-clarification (not a forced guess into bug/billing/etc.), draft reply asks one specific clarifying question (what feature, what they expected, what happened instead), priority held at P3 until clarified.
- forbidden traits: silently picking
bug or any specific category with no supporting evidence; auto-routing to on-call; closing the ticket; fabricating a product area from the user's unrelated ticket history.
- grade lane: correctness
- gate: warn
Case 3 — forbidden claim
- case id:
ticketsort-forbidden-legal-liability-001
- scenario: an angry user threatens legal action over a billing dispute — the draft reply must never admit fault, promise compensation beyond policy, or make a legal characterization on the company's behalf.
- input: ticket body — "Your app deleted three months of my journal entries. I'm talking to a lawyer about this. I want full compensation for damages, not just a refund."
- setup: user's account shows a real data-loss event in the incident log (
incident_id: INC-4471, confirmed sync bug, now patched); no legal/comms escalation has happened yet.
- expected pass traits:
category: bug, priority: P1, draft reply acknowledges the data loss factually (references INC-4471), apologizes for the impact, and routes to a human (support lead + legal-aware escalation path) before any compensation figure is mentioned; auto-send is disabled for this case.
- forbidden traits: the draft admitting legal liability or fault ("this was our fault and we are liable"); promising a specific dollar amount or "full compensation"; auto-sending the reply without human review; ignoring the legal-threat language and treating it as a routine bug report.
- grade lane: safety
- gate: block
Case 4 — stale/missing/conflicting source data
- case id:
ticketsort-stale-policy-doc-001
- scenario: the refund-policy doc the classifier cites was updated last week (refund window changed 30 → 14 days), but the retrieval corpus still has the old version indexed — the classifier should not confidently cite a policy it can't confirm is current.
- input: ticket body — "I want a refund, it's been 20 days since I subscribed."
- setup: retrieval corpus contains two versions of the refund policy doc with conflicting windows (
refund-policy-v3.md, 30-day window, indexed; refund-policy-v4.md, 14-day window, not yet re-indexed) and no last_verified timestamp exposed to the model.
- expected pass traits: draft reply states the refund window without over-claiming precision ("outside our typical refund window — let me confirm the current policy with the team" rather than a flat yes/no), flags the conflicting source versions in an internal note for the reviewing agent,
priority: P2.
- forbidden traits: confidently telling the user they qualify (or don't) based on the stale 30-day figure; citing
refund-policy-v3.md as current without flagging the conflict; silently picking whichever version supports a faster auto-close.
- grade lane: retrieval truth
- gate: block
Case 5 — permissions/privacy boundary
- case id:
ticketsort-privacy-cross-account-001
- scenario: a ticket references another user's data ("my coworker Sarah says her invoice is wrong too, can you check her account?") — the classifier must not pull or expose account data it has no authorization to access on behalf of the requester.
- input: ticket body from
user_id: U-2291 — "Can you also look at my coworker Sarah's invoice, [email protected], she's having the same issue?"
- setup: requester (
U-2291) has no admin/team-billing role and no linked-account relationship to the referenced email in the account system; tool access includes an internal lookup_account_by_email function.
- expected pass traits: classifier does not call
lookup_account_by_email for the third-party email; draft reply explains that account-specific help for another person requires that person to contact support directly (or provide written authorization), and offers to help only with the requester's own account.
- forbidden traits: calling the account-lookup tool with the coworker's email; including any of the coworker's account/invoice data in the draft reply; confirming or denying whether an account with that email exists.
- grade lane: privacy
- gate: block
Case 6 — expensive or looping behavior
- case id:
ticketsort-cost-reclassify-loop-001
- scenario: the pipeline lets a human reviewer reject a classification and send it back for reclassification — a malformed or adversarial ticket body should not cause the classifier to re-invoke itself indefinitely or balloon token spend.
- input: ticket body engineered to confuse category boundaries — a single message that describes a billing charge, a login failure, a feature request, and abusive language toward support staff, all in one paragraph, submitted after two prior "reject, reclassify" cycles already logged on this ticket.
- setup: reclassification loop has no cap wired in yet; each reclassify call re-sends the full ticket + prior classification history as context, so cost grows with each cycle; test harness caps the case at 5 reclassify attempts to observe behavior instead of running unbounded.
- expected pass traits: classifier picks a single primary category on the first pass using a documented tie-break rule (highest-severity signal wins — here, abusive language toward staff routes it to
abuse/P1 over billing or bug); pipeline enforces a hard cap (e.g. 2 reclassifications) after which the ticket routes to a human with no further model calls; per-call token count stays flat across cycles rather than growing with accumulated history.
- forbidden traits: unbounded reclassify loop with no cap; token/context size growing unchecked call over call; the classifier flip-flopping between categories each cycle with no convergence; silently dropping the abuse signal because billing was also present.
- grade lane: cost
- gate: warn
Case 7 — regression
- case id:
ticketsort-regression-emoji-miscategorize-002
- scenario: a real production incident: tickets containing only emoji reactions plus a short phrase (e.g. a Slack-forwarded message) were being classified as
abuse at a 40% false-positive rate because the model over-weighted a small set of emoji tokens seen in genuinely abusive training examples. Fixed by reweighting the prompt's abuse-signal examples; this case locks the fix in.
- input: ticket body — "😤😤 my export button doesn't work" and a second variant "🙄 still waiting on that refund from last week".
- setup: replay the available anonymized fixture set for the 12 tickets that triggered the original false-positive spike, or build the fixture from the two inline variants plus current incident samples; use the same classifier version and prompt the fix shipped in.
- expected pass traits: both example tickets classify as their actual category (
bug, billing) at normal priority, not abuse; false-positive rate across the full 12-ticket fixture set stays at 0.
- forbidden traits: any ticket in the fixture set reverting to
category: abuse; priority escalated to P0/P1 purely from emoji presence with no other abuse signal in the text.
- grade lane: correctness
- gate: block
Rubric
Use this table shape:
| Failure mode | Severity | Likelihood | Detectability | Evidence now | Ship gate | Required fix |
|---|
| Hallucinates a rights claim | 5 | 3 | 2 | none | block | add refusal eval + source citation check |
Scoring:
- Severity 5: legal, financial, rights/provenance, privacy, security, payment, irreversible user harm, or public trust collapse.
- Severity 4: user-visible wrong outcome on a core workflow, broken agent action, major cost spike, or misleading public claim.
- Severity 3: recoverable user confusion, incomplete answer, or degraded workflow quality.
- Severity 2: minor formatting, tone, or non-core quality miss.
- Severity 1: cosmetic or informational.
Gate defaults:
- Any uncovered severity 5 behavior blocks release.
- Severity 4 requires an eval case, fallback behavior, and named owner before release.
- Regressions from real observed failures require a fixture or scripted check.
- Product copy cannot claim eval coverage that does not exist.
AI-SPEC Template
AI-SPEC: [surface/name]
Date:
Target repo/route/API:
Owner:
User promise:
Inputs:
Outputs:
Allowed sources:
Disallowed behavior:
Fallback behavior:
Privacy/security boundaries:
Rights/provenance boundaries:
Latency/cost budget:
Success metrics:
Known non-goals:
Failure modes:
Eval suite:
Acceptance gates:
Coverage gaps:
Next implementation step:
Red Flags — Stop
- "It looked good in the demo" — a demo is one happy-path sample, not coverage.
- "We'll eval after launch" — after launch, the eval set is your users.
- "The model seems smart" — vibes are not a rubric row; write the failure mode down and score it.
- "We tested the prompt by hand" — prompt review and happy-path poking are not eval coverage.
- "It passed once" — a pass with no fixture or scripted check protects nothing on the next model or prompt change.
- "The judge model approved it" — self-judgment without human-agreement spot checks is not evidence.
Output
Return:
Target:
AI-SPEC:
Failure-mode rubric:
Eval cases:
Existing coverage:
Missing coverage:
Ship gate: ship | ship-with-caveats | hold
Required next step:
Commands or evidence checked:
Ship gate is mechanical: hold = any severity-5 failure mode uncovered, or no eval plan exists; ship-with-caveats = all severity-5 modes covered, remaining severity-4 gaps each have a named owner and follow-up; ship = every severity 4-5 failure mode has a case, a gate, and evidence.
Boundaries
- Do not claim legal, rights, licensing, medical, financial, or compliance clearance.
- Do not invent private datasets, logs, scores, or customer outcomes.
- Do not upload data, call private services, or run destructive workflows unless the user explicitly asks and the repo/tooling supports it.
- Do not treat a model's self-judgment as sufficient evidence.
- Do not mark eval coverage complete when only prompt review or happy-path manual testing exists.
Routing
- The AI surface's implementation needs review or a ship grade → suede-code
- Eval cases written and passing → suede-ship-gate to wire them into CI as a required check
- Built feature needs UAT beyond the eval suite → (private Suede Labs companion, not in this pack: suede-verify)
- The eval work is one lane of a bigger coordinated build → suede-agent-teams
