// Help the user systematically identify and categorize failure modes in an LLM pipeline by reading traces. Use when starting a new eval project, after significant pipeline changes (new features, model switches, prompt rewrites), when production metrics drop, or after incidents.
| name | error-analysis |
| description | Help the user systematically identify and categorize failure modes in an LLM pipeline by reading traces. Use when starting a new eval project, after significant pipeline changes (new features, model switches, prompt rewrites), when production metrics drop, or after incidents. |
Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.
Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.
Target: ~100 traces. This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.
From real user data (preferred):
From synthetic data (when real data is sparse):
Present each trace to the user. For each one, ask: did the system produce a good result? Pass or Fail.
For failures, note what went wrong. Focus on the first thing that went wrong in the trace — errors cascade, so downstream symptoms disappear when the root cause is fixed. Don't chase every issue in a single trace.
Write observations, not explanations. "SQL missed the budget constraint" not "The model probably didn't understand the budget."
Template:
| Trace ID | Trace | What went wrong | Pass/Fail |
|----------|-------|-----------------|-----------|
| 001 | [full trace] | Missing filter: pet-friendly requirement ignored in SQL | Fail |
| 002 | [full trace] | Proposed unavailable times despite calendar conflicts | Fail |
| 003 | [full trace] | Used casual tone for luxury client; wrong property type | Fail |
| 004 | [full trace] | - | Pass |
Heuristics:
After reviewing 30-50 traces, start grouping similar notes into categories. Don't wait until all 100 are done — grouping early helps sharpen what to look for in the remaining traces. The categories will evolve. The goal is names that are specific and actionable, not perfect.
When to split vs. group:
Split these (different root causes):
Group these (same root cause):
LLM-assisted clustering (use only after the user has reviewed 30-50 traces):
Here are failure annotations from reviewing LLM pipeline traces.
Group similar failures into 5-10 distinct categories.
For each category, provide:
- A clear name
- A one-sentence definition
- Which annotations belong to it
Annotations:
[paste annotations]
Always review LLM-suggested groupings with the user. LLMs cluster by surface similarity (e.g., grouping "app crashes" and "login is slow" because both mention login).
Aim for 5-10 categories that are:
Go back through all traces and apply binary labels (pass/fail) for each failure category. Each trace gets a column per category. Use whatever tool the user prefers — spreadsheet, annotation app (see build-review-interface), or a simple script.
failure_rates = labeled_df[failure_columns].sum() / len(labeled_df)
failure_rates.sort_values(ascending=False)
The most frequent failure category is where to focus first.
Work through each category with the user in this order:
Can we just fix it? Many failures have obvious fixes that don't need an evaluator at all:
If a clear fix resolves the failure, do that first. Only consider an evaluator for failures that persist after fixing.
Is an evaluator worth the effort? Not every remaining failure needs one. Building and maintaining evaluators has real cost. Ask the user:
Reserve evaluators for failures the user will iterate on repeatedly. Start with the highest-frequency, highest-impact category.
For failures that warrant an evaluator: prefer code-based checks (regex, parsing, schema validation) for anything objective. Use write-judge-prompt only for failures that require judgment. Critical requirements (safety, compliance) may warrant an evaluator even after fixing the prompt, as a guardrail.
Expect 2-3 rounds of reviewing and refining categories. After each round:
Stop reviewing when new traces aren't revealing new kinds of failures. Roughly: ~100 traces reviewed with no new failure types appearing in the last 20. The exact number depends on system complexity.
When production volume is high, use a mix:
| Strategy | When to Use | Method |
|---|---|---|
| Random | Default starting point | Sample uniformly from recent traces |
| Outlier | Surface unusual behavior | Sort by response length, latency, tool call count; review extremes |
| Failure-driven | After guardrail violations or user complaints | Prioritize flagged traces |
| Uncertainty | When automated judges exist | Focus on traces where judges disagree or have low confidence |
| Stratified | Ensure coverage across user segments | Sample within each dimension |
Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-analysis, write-judge-prompt, or validate-evaluator instead).
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.
Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with standard unit tests).
Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests). Do NOT use when you need to validate or calibrate the judge — use validate-evaluator instead.