| name | trace-triage |
| description | Diagnose agent misbehavior by reading its full execution trace (Langfuse, LangSmith, Braintrust, OTel, or any event log) - finds structural defects invisible from outputs alone. Use when an agent's plausible-looking result is suspected wrong, a user reports bad results from an agent, two agents or runs need comparing on the same task, or a recurring trace-quality bench is being stood up. Requires access to a trace carrying tool calls, reasoning, and intermediate results. Triggers - "why did the agent do that", "read the trace", "triage this run", "the output looks fine but", "compare these two runs", "the user says the agent's results are bad", "derive an eval rubric from my agent's pipeline". |
Trace Triage
Outputs tell you what an agent produced. Traces tell you what it considered and threw away — and that is where most real defects live.
Agent-agnostic. Requires a trace carrying tool calls, model reasoning, and intermediate results.
The method: the rubric is the pipeline, staged
Do not start from a generic checklist. Generic checklists find generic bugs.
Write down the agent's pipeline stage by stage. For each stage, ask:
- What can this stage get silently wrong? — wrong in a way that still produces a well-formed, plausible result.
- What in the trace would prove it? — the specific field, argument, or branch.
- What would the output look like if it did? — usually completely normal. That is the point, and it is the whole reason output-only review misses these.
Each stage becomes one lens.
If the stages aren't known — triaging a vendor's agent, a teammate's, or an unfamiliar codebase — recover them from the trace first: cluster tool calls by purpose and read the reasoning between clusters. Each cluster boundary is a stage boundary. Do this before applying any lens.
The worked example below is a production retrieval-and-rank agent. If the target agent is a different shape, use the example as a pattern reference only and derive the target's own stages first. Stages 1, 2, 6, 7 and 8 transfer to almost any agent — spec formation, tool-argument construction, selection, delivery, and self-claims. Stages 3, 4, 9 and 10 are retrieval-specific.
Worked example — a retrieval-and-rank agent
1 · Criteria generation (check this first — everything downstream inherits its poverty)
The agent converts intent into a structured spec.
- Does every requirement the user stated map to a field that exists in the schema? A requirement with no slot cannot be scored on, cannot be ranked on, and nothing downstream will protect it — it fails silently forever, however many times the user repeats it.
- Are constraints derived from evidence, or only from prose? Currency, locale, language and address on a page are all constraints. A site whose pricing and language switcher point at one market should not yield a spec with no geography rule.
- Rule richness. Count the criteria. Thin specs cannot discriminate, and every downstream quality signal will flatten.
2 · Tool-argument construction ← the sneakiest stage
Do the emitted arguments match what the agent's own preceding reasoning said it was about to do?
This is where the reasoning is right and the argument is backwards:
- Role inversion — an agent sourcing buyers for outsourced CFO services filtered on
current_titles=[CFO, Controller], and returned the companies that already had the function in-house.
- Scope collapse — a multi-company scope where the results cluster onto one member of the list instead of spreading across it.
The tool did exactly as instructed. The output list looks fine.
3 · Retrieval breadth
Several strategies usually run (structured filter, semantic, hybrid). Which one is actually carrying the segment? Sparse or unusual segments are often served by exactly one strategy — if that one is ever dropped, coverage collapses silently.
4 · Entity resolution and recency
- Is the entity attached to the record the current one? Shipping someone's former employer is a classic.
- Did entity resolution fail silently? When no company entity resolves, every company-level rule (size, industry, still-operating) becomes unenforceable — and the record still scores.
- Are there states the data structurally cannot see? Name them as blind spots rather than pretending they are covered.
5 · Scoring calibration
Pull the distribution over all scored candidates, not the delivered top-N.
- Flat or bimodal (everything top-scored, or top-and-bottom only) means no discrimination among passers.
- Before blaming the judge, check stage 1. Compare the same judge against specs of differing rule richness. If spread tracks rule count, the constraint is the criteria generator.
- In the source engagement, three natural cases in one day — 29 leads all top-scored, 8 all top-scored, then 26/11 with real gradation — pointed a long-running argument decisively at the criteria generator rather than the scorer. Three observational cases with different candidate pools, so not a controlled experiment; but the direction was unambiguous.
6 · Selection, merge and discard ← highest yield
Compare what was generated to what was delivered.
- Read the branch that lost. Was it actually worse — and was it dropped for the reason that made it worse?
- Winner-take-all selection is the highest-severity pattern here. An agent that retrieves correctly and then discards the right answer is indistinguishable, from outside, from one that failed to retrieve.
- Tie-breaks on a flat score are near-random. In one run, six good items were dropped to keep five, on an exact tie.
7 · Delivery integrity
What was persisted, what the artifact carries, and what the user actually saw are three different numbers. Reconcile all three.
- Prose stating one count, the artifact holding another, the database a third.
- Empty ID lists shipped alongside a populated artifact.
- Any query reading the union of all branches over-states what the user received. Always split validated from delivered.
8 · Claim ↔ evidence integrity
Take the final user-facing message and check every claim it makes about its own process against structured state.
If the message says a criterion was prioritised, a rule for that criterion must exist. Prose asserting work the system never did is worse than a plain miss — it removes the user's ability to detect the failure.
9 · Distribution hygiene
- Unprompted concentration on a dimension nobody asked to concentrate on.
- Constraint leakage — a hard geography filter admitting neighbours.
- Deduplication across name variants, transliterations and diminutives. Before calling two records duplicates, compare underlying IDs; before calling a name fake, check whether it is a real account with an unusual display name.
10 · Actionability
Correct and useless are different failures. Is the delivered item reachable — contact method present, channel viable, subject actually active? A perfectly on-target record with no route to act on it has failed the user even though every field is right.
Execution lenses — run these too
Correctness lenses find wrongness. These find waste and fragility.
- Evidence acquisition — did the fetches feeding the judgment actually succeed? Scrapers and enrichment calls fail selectively (auth-walled pages, PDFs, rate-limited hosts), and agents routinely proceed on whatever came back. Never let a confident verdict ship on a record whose primary source failed to load; emit an evidence-coverage ratio per decision. In the source engagement this was found late and was as severe as anything the correctness lenses produced: a qualification agent whose scraper failed on every profile and PDF it was handed still returned a verdict, and every affected run reported success.
- Tool-call economy — duplicate calls, N+1 fan-out, calls whose results are never read, sequential work that could be parallel.
- Context and cache discipline — is the stable prefix actually stable? Cache hit rate, context re-sent per step, prompt growth per turn. Establish whether a zero cache figure means no caching or no instrumentation before treating it as either.
- Latency distribution — the tail, not the mean, and always on a filtered denominator. Non-production and internal traffic will dominate a raw percentile and make it meaningless.
- Failure and degraded modes — hard caps hit, interrupts, reconnects, silent fallbacks. Verifier or grader iteration count above one means something churned — worth reading even when the run passed.
- Cost per unit of delivered value — not cost per run. A cheap run that delivers nothing is not cheap.
A rubric that never looks somewhere will never report a defect there, and that silence reads as health. So keep correctness and execution lenses in the same routine.
In the source engagement the two were run separately: correctness nightly, and latency as its own dedicated census — a filtered-denominator trace study across 309 production traces that surfaced real problems, including runs where users waited minutes for zero results. Both halves earned their keep. Wiring the execution lenses into the nightly gate is the obvious next step, and the reason to do it is compounding: a census finds problems once, a schedule finds them continuously.
Cross-cutting rules
Ground truth from outside. If the thing grading the output is the thing that produced it, that is a vibe, not an eval. Resolve factual claims against an independent source. A second arm need not be a competitor — the previous version of the same agent works. One number is a fact; two are a gradient.
Disconfirm before filing. For subjective findings (score quality, severity, "it always does X") and for anything derived from a cheap numeric proxy (duplicate-call counts, cache ratios, latency outliers), spawn a fresh reviewer with no prior context — trace plus claim only — and let it reach its own conclusion. Metric-derived findings are the ones most likely to be confidently wrong: in the source engagement, "nine duplicate calls" turned out to be nine distinct inputs, because the comparison hashed only the first 4000 characters of each. Compare full inputs before calling two calls identical. Only findings checkable by direct inspection of a single artifact — a row count, a URL mismatch, a missing file — skip this pass.
Convergence is the strongest evidence available. When a finding claims the discard threw away the right answer, an independent agent surfacing the same discarded item turns an opinion into corroboration.
Never triage a partial trace. Large runs can exceed what a trace endpoint will return. Confirm the trace is complete before reasoning over it, and reconstruct from persisted artifacts when it is not — a partial triage reported as a triage is worse than none.
Finding format
A finding that cannot be closed is not a finding.
ID · SEVERITY · one-line defect statement
Evidence: the exact trace excerpt, quoted
Consequence: what the user experienced
Fix: what changes, in which component
Acceptance: the assertion that proves it fixed (write it as an inverse predicate)
Telemetry: the counter that would have caught this automatically
Prompt fixes do not stick. Findings need an owner and a regression re-test on a later run. Treat anything marked resolved without a passing re-test as still open — in the source engagement a scoring finding was closed on one day and had reoccurred by the next.
Procedure
- Pull the complete trace.
- Establish the pipeline's stages — from the code if available, otherwise recovered from the trace as above.
- Derive one lens per stage using the three questions.
- Walk the stages in order. If time-boxed: criteria generation and selection/discard first — empirically they carry the most severity, ahead of the retrieval and scoring stages most people check first.
- Reason exhaustively rather than scan-and-summarise; a tidy summary glosses exactly the weak spots being hunted.
- Verify factual claims against an outside source.
- Disconfirm subjective and metric-derived findings with a context-free reviewer.
- File in the format above; route each finding to whoever owns that component.
- Close only on a passing regression re-test.
Running it on a schedule
The gate matters more than the cadence: define what counts as a genuinely new case, or the loudest user gets re-measured every night and it gets called coverage. Most nights the correct action is to spend nothing.
Aim for every lens the rubric carries to run inside the scheduled job. A one-off investigation produces findings once; the schedule produces them continuously.
The reading is automatable. The judgment is not — a human decides what is real before it becomes work.
