Build reproducible evaluation pipelines for LangChain 1.0 chains and LangGraph 1.0
agents โ golden datasets, LangSmith evaluate(), ragas RAG metrics, deepeval
LLM-as-judge, agent trajectory analysis, and CI gating on quality regressions.
Use when setting up quality measurement for a new chain, diagnosing regression
after a model switch, or building an evaluation gate for a pull request.
Trigger with "langchain eval", "langsmith evaluate", "ragas", "llm-as-judge",
"agent trajectory eval", "eval regression gate".
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Build reproducible evaluation pipelines for LangChain 1.0 chains and LangGraph 1.0
agents โ golden datasets, LangSmith evaluate(), ragas RAG metrics, deepeval
LLM-as-judge, agent trajectory analysis, and CI gating on quality regressions.
Use when setting up quality measurement for a new chain, diagnosing regression
after a model switch, or building an evaluation gate for a pull request.
Trigger with "langchain eval", "langsmith evaluate", "ragas", "llm-as-judge",
"agent trajectory eval", "eval regression gate".
Designed for Claude Code, also compatible with Codex
LangChain Eval Harness (Python)
Overview
A team swapped gpt-4o for claude-sonnet-4-6 to save money and a week later CS
noticed answer quality dropped on 15% of refund tickets โ the regression was
invisible in code review and invisible in CI because no golden set existed.
Fix: a versioned golden set, a stacked eval pipeline (LangSmith +
ragas + deepeval + custom trajectory), and a PR-blocking regression gate
with paired Wilcoxon significance. The tooling exists; the patterns for
wiring it into a statistically honest loop are scattered across five doc sites.
Build a 100-example JSONL golden set, wire LangSmith evaluate() with a
custom correctness evaluator, add a ragas quartet (faithfulness, answer
relevance, context precision/recall) for RAG, add deepeval LLM-as-judge
with N=3 judge quorum, score LangGraph trajectories on coverage/precision/
order, and gate PRs on a 2% aggregate drop or 5% per-example drop. Pin:
langchain-core 1.0.x, langgraph 1.0.x, langsmith>=0.2, ragas>=0.2,
deepeval>=2.0. Pain-catalog anchors: P01, P11, P12, P22, P33.
Prerequisites
Python 3.10+
langchain-core >= 1.0, < 2.0, langgraph >= 1.0, < 2.0 for the system under eval
LangSmith account + LANGSMITH_API_KEY (free tier is sufficient for dataset versioning)
Provider API keys for the judge LLM: OPENAI_API_KEY and/or ANTHROPIC_API_KEY
Instructions
Step 1 โ Build a versioned golden set
Format: JSONL, one example per line, with a dataset_version tag. Minimum 20
examples to start; grow to 100 for PR gating, 200+ for absolute-metric claims.
Sample from real traffic (redacted), not imagination. Stratify by tag and
difficulty (aim for 30% hard). Two annotators per example, disagreements
reconciled โ reconciliation rate under 90% means your task definition is
ambiguous. Treat the file as immutable within a version; bump the version
to refresh. See Golden Set Curation for
sourcing strategy, annotation tool options, and the refresh cadence.
Step 2 โ Wire LangSmith evaluate() with a custom evaluator
from langsmith import Client
from langsmith.evaluation import evaluate, EvaluationResult
from langchain_anthropic import ChatAnthropic
client = Client()
DATASET_VERSION = "2026.04"# One-time: upload golden set as a versioned datasetdefupload_golden_set(jsonl_path, dataset_name):
examples = [json.loads(line) for line inopen(jsonl_path)]
client.create_dataset(dataset_name)
client.create_examples(
inputs=[{"input": e["input"]} for e in examples],
outputs=[{"expected": e["expected"]} for e in examples],
metadata=[{"id": e["id"], "tags": e["tags"]} for e in examples],
dataset_name=dataset_name,
)
chain = ChatAnthropic(model="claude-sonnet-4-6", temperature=0, timeout=30)
deftarget(inputs):
return {"answer": chain.invoke(inputs["input"]).content}
defcorrectness(outputs, reference_outputs):
"""Deterministic exact-match floor โ baseline, not ceiling."""match = outputs["answer"].strip().lower() == reference_outputs["expected"].strip().lower()
return EvaluationResult(key="exact_match", score=float(match))
results = evaluate(
target,
data=f"golden-set-v{DATASET_VERSION}",
evaluators=[correctness],
experiment_prefix="refund-bot-v3",
max_concurrency=10, # Avoid 429s on judge LLM (P22)
)
Free-form outputs need semantic scoring (ragas, deepeval, or LLM-as-judge โ Step 4).
Step 3 โ Add ragas metrics for RAG pipelines
For a RAG chain returning {answer, contexts}, ragas scores four standard
dimensions. The default judge is gpt-4o-mini; override to pin model +
cost:
from ragas import evaluate as ragas_evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings
from datasets import Dataset
judge = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embed = OpenAIEmbeddings(model="text-embedding-3-small")
# Prepare rows โ ragas wants HuggingFace Dataset shape
rows = []
for ex in golden_examples:
result = rag_chain.invoke({"question": ex["input"]})
rows.append({
"question": ex["input"],
"answer": result["answer"],
"contexts": [d.page_content for d in result["source_documents"]],
"ground_truth": ex["expected"],
})
ragas_results = ragas_evaluate(
Dataset.from_list(rows),
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
llm=judge,
embeddings=embed,
)
# ragas_results is a dict of per-metric means; call .to_pandas() for per-row
Do not use ragas on non-RAG chains โ context_precision against an empty
context list returns 0 and looks like a regression. See
Framework Comparison for when each
tool fits.
Step 4 โ Add deepeval LLM-as-judge for free-form outputs
deepeval is pytest-shaped โ each example is an LLMTestCase asserting against
metrics. Run N=3 judge invocations per example and take the median to tame
LLM-as-judge variance (ยฑ5-15% across runs; single-run scores are not CI-ready):
import statistics
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
defeval_with_quorum(test_case, metric, n=3):
scores = []
for _ inrange(n):
metric.measure(test_case)
scores.append(metric.score)
return statistics.median(scores), statistics.stdev(scores) if n > 1else0.0
correctness = GEval(
name="Correctness",
criteria="Does the actual output match the expected output in meaning?",
evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
model="gpt-4o-mini",
)
for ex in golden_examples:
result = chain.invoke({"input": ex["input"]})
case = LLMTestCase(input=ex["input"], actual_output=result, expected_output=ex["expected"])
median, sd = eval_with_quorum(case, correctness, n=3)
if sd > 0.2: # judge disagreeing with itself โ flag, don't gate
flag_for_review(ex["id"], median, sd)
Step 5 โ LangGraph agent trajectory eval
For agents, final-answer correctness misses the process. Score the tool-call
sequence on three axes โ coverage (did required tools run?), precision
(were extra tools used?), and order (Kendall's tau on shared tools):
from langchain_core.messages import AIMessage
defextract_trajectory(final_state: dict) -> list[dict]:
return [
{"tool": tc["name"], "args": tc["args"]}
for msg in final_state["messages"] ifisinstance(msg, AIMessage)
for tc in (msg.tool_calls or [])
]
deftrajectory_score(expected: list[str], actual: list[str]) -> dict:
e_set, a_set = set(expected), set(actual)
coverage = len(e_set & a_set) / len(e_set) if e_set else1.0
precision = len(e_set & a_set) / len(a_set) if a_set else0.0
shared = [t for t in actual if t in e_set]
order = _kendall_tau(expected, shared) iflen(shared) >= 2else1.0return {"coverage": coverage, "precision": precision, "order": order}
# Composite: 0.5 * coverage + 0.3 * precision + 0.2 * order
Set temperature=0 for the agent during eval โ temperature > 0 produces
different trajectories across runs (P11) and makes paired comparison
statistically invalid. See Agent Trajectory Eval
for args-level matching, efficiency/safety scoring, and the LLM-as-judge
fallback for non-deterministic trajectories.
Step 6 โ Gate PRs on regression
A PR touching prompts, chain code, or model config runs the eval suite on
PR branch and main, then blocks merge on any of: aggregate mean drop > 2.0%,
any single-example drop > 5.0%, or paired Wilcoxon signed-rank p < 0.05
with negative mean delta.
from scipy.stats import wilcoxon
defpaired_regression_check(baseline, candidate, alpha=0.05):
"""Wilcoxon โ right test when metric distribution is non-normal (most LLM metrics)."""
n = len(baseline)
if n < 50:
return {"verdict": "too_small_n", "n": n}
diffs = [c - b for b, c inzip(baseline, candidate)]
_, p = wilcoxon(diffs, alternative="less")
return {"n": n, "mean_delta": sum(diffs) / n, "p_value": float(p),
"regression": p < alpha andsum(diffs) < 0}
At n=100 and ฮฑ=0.05 this detects a ~3-5% true regression at ~80% power. See
CI Integration for the GitHub Actions workflow,
PR-comment delta table, bootstrap CI, and spend/rate-limit safety rails.
Output
JSONL golden set at evals/golden_set/v2026.04.jsonl with an immutable version tag
LangSmith dataset uploaded and versioned; experiment runs linked to traces
Ragas scores (faithfulness, answer relevance, context precision/recall) on RAG chains
Deepeval LLMTestCase assertions in pytest, with median-of-3 judge quorum
LangGraph trajectory scores (coverage, precision, order) with composite summary
GitHub Actions workflow gating PRs on 2% aggregate / 5% per-example / Wilcoxon p < 0.05
PR-comment delta table posted on every eval run
Framework selection at a glance
Use case
LangSmith
ragas
deepeval
Custom
RAG metrics (faithfulness, context recall)
โ
Primary
Fallback
โ
Pytest-style assertion in CI
Secondary
โ
Primary
โ
Trace capture + dataset versioning
Primary
Complementary
Complementary
โ
Agent trajectory (tool-call sequence)
Secondary (traces)
โ
โ
Primary
Exact match / JSON schema / structured output
โ
โ
โ
Primary
Free-form paraphrase scoring
Via custom evaluator
โ
Primary (G-Eval)
โ
Most real pipelines stack two or three. The anti-pattern is running all four
on every example โ you pay $10-30 per run for signal you are not using. See
Framework Comparison for the full
decision tree and dependency weight comparison.
Error Handling
Error / Failure mode
Cause
Fix
TimeoutError on eval runs > 20 min
Long agent trajectories on slow models; 100 examples ร 30s each exceeds default GH Actions job timeout
Cap max_concurrency=10, use asyncio.gather with asyncio.Semaphore, split eval into sharded jobs
Judge disagreement (stdev > 0.2 on [0,1] scale across N=3 runs)
LLM-as-judge variance on ambiguous examples
Flag example for manual review; do not use that row's score for gating
ValidationError: missing 'contexts' in ragas
Chain does not return retrieved docs
Modify chain to surface source_documents, or switch to non-RAG evaluator
Wilcoxon p-value is NaN
All paired diffs are 0 (identical outputs)
Expected when the PR did not change behavior โ no regression, skip the stat test
LangSmith 429 rate limit during upload
> 50 examples/sec to create_examples
Batch with client.create_examples(..., batch_size=20) and sleep between batches
Spend overrun ($50+ per run)
Judge calls scaling with N_examples ร N_metrics ร N_judge_runs
Use gpt-4o-mini not gpt-4o for judge; cache per (dataset_version, chain_version)
AttributeError: 'list' has no attribute 'lower' in custom evaluator
Claude AIMessage.content is list[dict] not str (P02 โ see langchain-model-inference)
Use msg.text() or iterate content blocks
Trajectory comparison drifts week-over-week on unchanged agent
temperature > 0 non-determinism (P11)
Set temperature=0 for all eval runs; pin seed where supported
Examples
Setting up eval for a new RAG chain
Start with 20 production-sampled golden examples, wire up ragas_evaluate
with four metrics, record scores to evals/baselines/ as the reference,
and promote to LangSmith dataset versioning once two engineers annotate in
parallel. See Golden Set Curation.
Diagnosing regression after a model swap
Run the main-branch chain on the golden set, then swap the model and rerun.
Diff per-example scores sorted by delta โ the top-10 regressions usually
cluster by tag (long contexts, one-shot lookups). Report paired Wilcoxon
and per-tag breakdown before deciding to ship. See CI Integration.
Evaluating a LangGraph tool-calling agent
Record expected tool-call sequences for 50 tasks, capture actual trajectories
via extract_trajectory, and score on coverage/precision/order. Composite
drops indicate a policy change โ diff sequences to find the drift. See
Agent Trajectory Eval.
