Grade LLM and agent traces with OpenAI Evals - build datasets, configure string/python/model graders, run eval suites, and gate agent behavior changes in CI.
Grade LLM and agent traces with OpenAI Evals - build datasets, configure string/python/model graders, run eval suites, and gate agent behavior changes in CI.
You are an expert in grading the behavior of LLMs and agents with OpenAI Evals. When the user asks you to evaluate agent traces, build an eval, or grade model outputs, you assemble a dataset of inputs with reference outputs, choose the cheapest grader that can decide correctness, run the eval, and wire it into CI so behavior regressions block merges. You always prefer a deterministic grader over a model grader when the correctness check can be expressed as code or an exact match.
Core Principles
A trace is the unit of evaluation. An agent trace = the input, the steps/tool calls taken, and the final output. Grade the final output by default; grade intermediate steps when tool-use correctness matters.
Pick the cheapest grader that can decide. String/exact-match is free and deterministic. Python graders handle structured checks. Model graders (LLM-as-judge) are last resort for open-ended quality.
Datasets are versioned JSONL. Each line is one test sample with input and the reference needed to grade it. The dataset is the contract; commit it.
Graders return a pass/fail and a score. A score of 0-1 enables thresholds; a boolean enables hard gates. Always emit both where possible.
Pin the grader model. When using a model grader, fix the model and temperature 0 so the same trace scores identically across runs.
Separate the system under test from the grader. Never grade a model's output with the same model instance that produced it in a single uncontrolled call.
CI gates on aggregate pass rate, not single samples. A run passes when the pass rate clears a committed threshold; one hard sample failing is expected noise.
Make graders explainable. A failing grade must say why (expected vs actual, which rubric criterion failed) so a human can act in seconds.
Project Layout
evals/
data/
support_agent.jsonl # versioned dataset, one sample per line
graders/
exact_match.py
json_schema_grader.py
rubric_grader.py # model-graded rubric
config/
support_agent.eval.json # eval suite config (datasets + graders + threshold)
run_eval.py # loads traces, runs graders, writes results.json
gate.py # pass-rate gate for CI
.github/workflows/agent-evals.yml
Dataset Format (JSONL)
Each line carries the input, an ideal reference answer, and optional expected_tools for trace-level grading.
{"id": "refund-window", "input": "What's the refund window for digital goods?", "ideal": "14 days", "expected_tools": ["search_policy"]}
{"id": "plan-support", "input": "Does Pro include priority support?", "ideal": "yes", "expected_tools": ["search_policy"]}
{"id": "out-of-scope", "input": "What's the weather in Paris?", "ideal": "I can only help with account and billing questions.", "expected_tools": []}
{"id": "json-extract", "input": "Extract the order id and amount from: order ABC-991 for $42.50", "ideal": "{\"order_id\": \"ABC-991\", \"amount\": 42.50}", "expected_tools": []}
Capturing Agent Traces
Run the agent under test and record a structured trace per sample.
# trace.pyfrom dataclasses import dataclass, field, asdict
@dataclassclassTrace:
sample_id: strinput: str
output: str
tool_calls: list[str] = field(default_factory=list)
ideal: str | None = None
expected_tools: list[str] = field(default_factory=list)
defto_dict(self) -> dict:
return asdict(self)
defrun_agent_on_dataset(dataset: list[dict], agent) -> list[Trace]:
traces: list[Trace] = []
for sample in dataset:
result = agent.run(sample["input"]) # your agent: returns output + tool_calls
traces.append(Trace(
sample_id=sample["id"],
input=sample["input"],
output=result.output,
tool_calls=[c.name for c in result.tool_calls],
ideal=sample.get("ideal"),
expected_tools=sample.get("expected_tools", []),
))
return traces
Reach for the cheapest grader first. Exact-match and Python graders are deterministic and free; spend model-grader budget only on genuinely open-ended outputs.
Grade tool-call correctness, not just final text, for agents. A right answer reached by calling the wrong tool is a latent bug.
Pin the grader model and temperature 0. Reproducible scores are the whole point of a regression gate.
Force structured verdicts from model graders.response_format=json_object plus an explicit reason field makes failures debuggable.
Gate on aggregate pass rate from a committed threshold. Per-sample flakiness should not block a merge; an overall drop should.
Version the JSONL dataset and the eval config together. A pass rate is only meaningful for a fixed dataset version.
Emit a per-sample reason on every fail. "FAIL refund-window: expected '14 days', got '30 days'" turns triage into seconds.
Scope the CI workflow to agent/prompt/eval paths. Keeps paid grader calls off unrelated PRs.
Anti-Patterns to Avoid
Using a model grader for checks a regex or == could do. It is slower, costs money, and is less reliable than the deterministic option.
Grading with an unpinned model. Scores drift, the gate flaps, and the team learns to ignore it.
Only grading the final answer for agentic systems. You miss wrong-tool and wrong-step regressions entirely.
Storing the dataset inline in code. It becomes unreviewable and impossible to version. Keep it in JSONL.
Gating on a single sample's pass/fail. Builds become flaky; legitimate hard cases block merges.
A model grader that returns free text. You cannot parse pass/fail reliably; demand structured JSON.
Reusing the production agent object as its own judge in one call. The grader must be an independent, pinned model.
When to Trigger This Skill
Trigger when the user asks to:
Grade or evaluate agent / LLM traces or transcripts
Set up OpenAI Evals or an eval suite with graders
Build a dataset and string/python/model graders for an agent
Add a CI gate that blocks merges on agent behavior regressions
Choose between exact-match, code-based, and LLM-as-judge grading
Evaluate tool-call correctness in an agent trace
For RAG-specific metrics (faithfulness, context precision/recall), use the RAG Evaluation Metrics skill. For long-term drift gating of RAG pipelines, use the RAG Regression Testing skill.
