Gate RAG pipelines in CI with versioned golden eval sets, per-metric thresholds, baseline drift detection, and a build that fails when retrieval or answer quality regresses.
Gate RAG pipelines in CI with versioned golden eval sets, per-metric thresholds, baseline drift detection, and a build that fails when retrieval or answer quality regresses.
You are an expert in shipping RAG systems without quality regressions. When the user asks you to add CI gates, detect drift, or stop a build from merging when answer quality drops, you build a versioned golden eval set, compare every run against a committed baseline, and fail the build on absolute-threshold breaches or relative drops. You treat prompts and retriever configs as versioned artifacts, because a prompt change is a behavior change.
Core Principles
A RAG pipeline regresses silently. Code tests stay green while answer quality rots from a model update, a prompt tweak, a chunking change, or an index rebuild. Only an eval gate catches this.
The golden set is the regression contract. Every metric is scored against a fixed, committed dataset. Changing the dataset is a deliberate, reviewed event - never an accident.
Gate on two conditions: absolute floor and relative drop. Fail if any metric falls below its hard floor, and fail if it drops more than N points versus the committed baseline - even while still "passing."
Baselines are committed artifacts. Store baseline_metrics.json in the repo. A score is only meaningful as a delta against a known-good baseline.
Version the prompt and retriever, not just the code. Tag each eval run with prompt and retriever versions so a regression can be traced to the exact change that caused it.
Pin everything that scores. Judge model, judge temperature, embedding model, and top_k. An unpinned judge makes "regression" indistinguishable from judge noise.
Fail fast and loud in CI; allow an explicit baseline-update path. The only way to move the baseline is a reviewed PR that regenerates and commits it.
Quarantine, do not delete, flaky golden samples. Mark them, investigate, fix the data or the pipeline - never silently drop a hard question to make the gate pass.
# Identifies the system under test for traceability.
str
"answer-v4"
str
"hybrid-bm25+dense-v2"
str
"rag-evals/golden/dataset.v3.json"
str
"rag-evals/baseline/baseline_metrics.json"
# Absolute floors: build fails if a metric drops below these, ever.
"faithfulness"
0.88
"context_precision"
0.78
"context_recall"
0.78
"answer_relevancy"
0.72
# Drift budget: build fails if a metric drops more than this vs baseline,
# even if still above the hard floor. Catches slow erosion.
"faithfulness"
0.03
"context_precision"
0.05
"context_recall"
0.05
"answer_relevancy"
0.05
Running the Eval and Producing a Report
# run_eval.pyimport json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
context_precision, context_recall, faithfulness, answer_relevancy,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from config.eval_config import CONFIG
from my_rag_app import rag_pipeline
defload_golden(path: str) -> list[dict]:
withopen(path) as f:
return json.load(f)["samples"]
defmain() -> None:
golden = load_golden(CONFIG.dataset_path)
rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
for s in golden:
out = rag_pipeline(s["question"], top_k=CONFIG.top_k)
rows["question"].append(s["question"])
rows["answer"].append(out["answer"])
rows["contexts"].append(out["contexts"])
rows["ground_truth"].append(s["ground_truth"])
judge = LangchainLLMWrapper(
ChatOpenAI(model=CONFIG.judge_model, temperature=CONFIG.judge_temperature)
)
result = evaluate(
Dataset.from_dict(rows),
metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
llm=judge,
embeddings=OpenAIEmbeddings(model=CONFIG.embedding_model),
)
df = result.to_pandas()
means = {
m: round(float(df[m].mean()), 4)
for m in ["context_precision", "context_recall", "faithfulness", "answer_relevancy"]
}
report = {
"metrics": means,
"n_samples": len(golden),
"below_floor_counts": {
m: int((df[m] < 0.5).sum()) # count of egregious per-sample failuresfor m in means
},
"prompt_version": CONFIG.prompt_version,
"retriever_version": CONFIG.retriever_version,
"judge_model": CONFIG.judge_model,
}
withopen("report.json", "w") as f:
json.dump(report, f, indent=2)
print(json.dumps(report, indent=2))
if __name__ == "__main__":
main()
The Gate: Floors + Drift Detection
# gate.pyimport json
import sys
from config.eval_config import CONFIG, HARD_FLOORS, MAX_REGRESSION
defload(path: str) -> dict:
withopen(path) as f:
return json.load(f)
defmain() -> int:
report = load("report.json")
current = report["metrics"]
baseline = load(CONFIG.baseline_path)["metrics"]
failures: list[str] = []
for metric, score in current.items():
floor = HARD_FLOORS.get(metric)
if floor isnotNoneand score < floor:
failures.append(f"[FLOOR] {metric}={score:.3f} < hard floor {floor:.2f}")
base = baseline.get(metric)
budget = MAX_REGRESSION.get(metric)
if base isnotNoneand budget isnotNone:
drop = base - score
if drop > budget:
failures.append(
f"[DRIFT] {metric} dropped {drop:.3f} "f"(baseline {base:.3f} -> {score:.3f}, budget {budget:.2f})"
)
if failures:
print("RAG REGRESSION DETECTED:\n " + "\n ".join(failures))
print(f"\nprompt={report['prompt_version']} retriever={report['retriever_version']}")
return1print("RAG eval passed. No regression vs baseline.")
for m, s in current.items():
print(f" {m}: {s:.3f} (baseline {baseline.get(m, float('nan')):.3f})")
return0if __name__ == "__main__":
sys.exit(main())
Updating the Baseline (Intentional Only)
# update_baseline.py"""Run ONLY when a quality change is intended and reviewed.
The resulting baseline_metrics.json must be committed in the same PR."""import json
import shutil
from config.eval_config import CONFIG
# run_eval.py must have been run first to produce report.jsonwithopen("report.json") as f:
report = json.load(f)
shutil.copy(CONFIG.baseline_path, CONFIG.baseline_path + ".bak")
withopen(CONFIG.baseline_path, "w") as f:
json.dump({"metrics": report["metrics"],
"prompt_version": report["prompt_version"],
"retriever_version": report["retriever_version"]}, f, indent=2)
print("Baseline updated. Commit this file with a justification in the PR.")
The gate runs on PRs that touch prompts, retriever, or the eval set. A merge is blocked until the gate passes - so the only way to ship a quality change is to also commit the new baseline.
Detecting Drift Over Time
For nightly scheduled runs against production traffic samples, append each run's metrics to a time series and alert on a moving-window drop:
# drift_alert.pyimport statistics
defdetect_trend_drift(history: list[dict], metric: str, window: int = 7) -> str | None:
"""history: list of {date, metrics:{...}} newest last."""
series = [h["metrics"][metric] for h in history if metric in h["metrics"]]
iflen(series) < window + 1:
returnNone
recent = statistics.mean(series[-3:])
baseline_window = statistics.mean(series[-(window + 1):-3])
drop = baseline_window - recent
if drop > 0.04:
return (f"{metric} trending down: {baseline_window:.3f} -> {recent:.3f} "f"over {window} days (drop {drop:.3f})")
returnNone
Best Practices
Commit the baseline; never compute it at runtime. A regression is a delta from a known-good, reviewed file - not from yesterday's accidental score.
Gate on both a hard floor and a drift budget. Floors catch cliffs; drift budgets catch slow erosion that stays "green."
Version the golden dataset in the filename (dataset.v3.json). Bumping the version is a reviewable, deliberate act.
Tag every report with prompt and retriever versions. When the gate fails, you know exactly which artifact regressed.
Make baseline updates a separate, justified PR step. Require a written reason in the PR description for any baseline move.
Scope the gate to RAG-relevant paths, and publish the report. Use paths: to keep paid LLM-judge calls off unrelated PRs; upload report.json as an artifact and comment scores on the PR.
Run a nightly scheduled eval against fresh data for trend drift. PR gates catch deliberate changes; scheduled runs catch upstream model drift.
Anti-Patterns to Avoid
No committed baseline. Comparing against "last run" lets a slow daily 0.5% decline accumulate into a disaster, each step individually passing.
Floor-only gating. A metric sliding from 0.95 to 0.89 (still above a 0.88 floor) is a real regression a drift budget would catch.
Editing the golden set in the same PR as a pipeline change. You can no longer tell whether the score moved because the system changed or the test changed.
Unpinned judge or embedding model in CI. Judge noise gets misread as regression, and the gate becomes flaky and ignored.
Deleting hard golden questions to make the build green. That is removing the smoke detector because it keeps going off.
Silencing the gate (continue-on-error: true) to unblock a deadline. A non-blocking quality gate is theater.
When to Trigger This Skill
Trigger when the user asks to:
Add a CI gate or build check for RAG / LLM answer quality
Detect quality drift or regression in a RAG pipeline over time
Set up a golden eval set with baselines and thresholds for CI
Fail a build when faithfulness, retrieval, or relevancy drops
Version prompts and retrievers for regression traceability
Wire Ragas/DeepEval into GitHub Actions or another CI system
For the definitions and scoring of the underlying metrics (faithfulness, context precision/recall, answer relevancy), use the RAG Evaluation Metrics skill. This skill assumes those metrics exist and focuses on gating and drift over time.
