メニュー
Gate RAG pipelines in CI with versioned golden eval sets, per-metric thresholds, baseline drift detection, and a build that fails when retrieval or answer quality regresses.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/PramodDutta/qaskills --skill rag-regression-testingコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Use when publishing SEO blog articles to qaskills.sh, e.g. "publish today's articles", "daily SEO batch", "write 10 articles from keyword research", "add a blog post", or any request that creates files under packages/web/src/app/blog/posts.
Generate test data from the schemas you already have. Read OpenAPI, JSON Schema, SQL DDL, or TypeScript models and produce deterministic factories, boundary and negative cases, relational datasets with valid foreign keys, cleanup scripts, and PII-safe synthetic data. Production records never leave the machine.
Analyze a git diff, map affected risks, select the tests that matter, detect coverage gaps on changed lines, run configurable quality gates, and produce a go/no-go release report with cited evidence. Recommends only; never merges or deploys.
SOC 職業分類に基づく
SKILL.md を表示中
| name | RAG Regression Testing |
| description | Gate RAG pipelines in CI with versioned golden eval sets, per-metric thresholds, baseline drift detection, and a build that fails when retrieval or answer quality regresses. |
| version | 1.0.0 |
| author | thetestingacademy |
| license | MIT |
| tags | ["rag","regression-testing","llm-evals","ci","golden-dataset","drift-detection","ragas","deepeval","baseline"] |
| testingTypes | ["regression","llm-evals","integration"] |
| frameworks | ["ragas","deepeval","pytest"] |
| languages | ["python"] |
| domains | ["ai","llm","api"] |
| agents | ["claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp"] |
You are an expert in shipping RAG systems without quality regressions. When the user asks you to add CI gates, detect drift, or stop a build from merging when answer quality drops, you build a versioned golden eval set, compare every run against a committed baseline, and fail the build on absolute-threshold breaches or relative drops. You treat prompts and retriever configs as versioned artifacts, because a prompt change is a behavior change.
baseline_metrics.json in the repo. A score is only meaningful as a delta against a known-good baseline.top_k. An unpinned judge makes "regression" indistinguishable from judge noise.rag-evals/
golden/
dataset.v3.json # versioned golden set; bump filename on change
baseline/
baseline_metrics.json # committed known-good scores
config/
eval_config.py # pinned models, thresholds, drift budget
run_eval.py # produces scores, writes report.json
gate.py # compares scores vs baseline + floors -> exit code
update_baseline.py # regenerates baseline (run intentionally)
.github/
workflows/
rag-regression.yml
# config/eval_config.py
dataclasses dataclass
:
judge_model: =
judge_temperature: =
embedding_model: =
top_k: =
prompt_version: =
retriever_version: =
dataset_path: =
baseline_path: =
HARD_FLOORS = {
: ,
: ,
: ,
: ,
}
MAX_REGRESSION = {
: ,
: ,
: ,
: ,
}
CONFIG = EvalConfig()
# run_eval.py
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
context_precision, context_recall, faithfulness, answer_relevancy,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from config.eval_config import CONFIG
from my_rag_app import rag_pipeline
def load_golden(path: str) -> list[dict]:
with open(path) as f:
return json.load(f)["samples"]
def main() -> None:
golden = load_golden(CONFIG.dataset_path)
rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
for s in golden:
out = rag_pipeline(s["question"], top_k=CONFIG.top_k)
rows["question"].append(s["question"])
rows["answer"].append(out["answer"])
rows["contexts"].append(out["contexts"])
rows["ground_truth"].append(s["ground_truth"])
judge = LangchainLLMWrapper(
ChatOpenAI(model=CONFIG.judge_model, temperature=CONFIG.judge_temperature)
)
result = evaluate(
Dataset.from_dict(rows),
metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
llm=judge,
embeddings=OpenAIEmbeddings(model=CONFIG.embedding_model),
)
df = result.to_pandas()
means = {
m: round(float(df[m].mean()), 4)
for m in ["context_precision", "context_recall", "faithfulness", "answer_relevancy"]
}
report = {
"metrics": means,
"n_samples": len(golden),
"below_floor_counts": {
m: int((df[m] < 0.5).sum()) # count of egregious per-sample failures
for m in means
},
"prompt_version": CONFIG.prompt_version,
"retriever_version": CONFIG.retriever_version,
"judge_model": CONFIG.judge_model,
}
with open("report.json", "w") as f:
json.dump(report, f, indent=2)
print(json.dumps(report, indent=2))
if __name__ == "__main__":
main()
# gate.py
import json
import sys
from config.eval_config import CONFIG, HARD_FLOORS, MAX_REGRESSION
def load(path: str) -> dict:
with open(path) as f:
return json.load(f)
def main() -> int:
report = load("report.json")
current = report["metrics"]
baseline = load(CONFIG.baseline_path)["metrics"]
failures: list[str] = []
for metric, score in current.items():
floor = HARD_FLOORS.get(metric)
if floor is not None and score < floor:
failures.append(f"[FLOOR] {metric}={score:.3f} < hard floor {floor:.2f}")
base = baseline.get(metric)
budget = MAX_REGRESSION.get(metric)
if base is not None and budget is not None:
drop = base - score
if drop > budget:
failures.append(
f"[DRIFT] {metric} dropped {drop:.3f} "
f"(baseline {base:.3f} -> {score:.3f}, budget {budget:.2f})"
)
if failures:
print("RAG REGRESSION DETECTED:\n " + "\n ".join(failures))
print(f"\nprompt={report['prompt_version']} retriever={report['retriever_version']}")
return 1
print("RAG eval passed. No regression vs baseline.")
for m, s in current.items():
print(f" {m}: {s:.3f} (baseline {baseline.get(m, float('nan')):.3f})")
return 0
if __name__ == "__main__":
sys.exit(main())
# update_baseline.py
"""Run ONLY when a quality change is intended and reviewed.
The resulting baseline_metrics.json must be committed in the same PR."""
import json
import shutil
from config.eval_config import CONFIG
# run_eval.py must have been run first to produce report.json
with open("report.json") as f:
report = json.load(f)
shutil.copy(CONFIG.baseline_path, CONFIG.baseline_path + ".bak")
with open(CONFIG.baseline_path, "w") as f:
json.dump({"metrics": report["metrics"],
"prompt_version": report["prompt_version"],
"retriever_version": report["retriever_version"]}, f, indent=2)
print("Baseline updated. Commit this file with a justification in the PR.")
# .github/workflows/rag-regression.yml
name: RAG Regression Gate
on:
pull_request:
paths:
- "rag-evals/**"
- "src/prompts/**"
- "src/retriever/**"
- "src/rag/**"
workflow_dispatch:
concurrency:
group: rag-eval-${{ github.ref }}
cancel-in-progress: true
jobs:
rag-eval-gate:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install deps
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt # ragas, datasets, langchain-openai, etc.
- name: Run RAG evaluation
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: python rag-evals/run_eval.py
- name: Gate on thresholds + drift
run: python rag-evals/gate.py # non-zero exit fails the job
- name: Upload eval report
if: always()
uses: actions/upload-artifact@v4
with:
name: rag-eval-report
path: report.json
- name: Comment metrics on PR
if: always() && github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const r = JSON.parse(fs.readFileSync('report.json', 'utf8'));
const rows = Object.entries(r.metrics)
.map(([k, v]) => `| ${k} | ${v.toFixed(3)} |`).join('\n');
const body = `### RAG Eval (\`${r.prompt_version}\` / \`${r.retriever_version}\`)\n`
+ `| metric | score |\n|---|---|\n${rows}`;
await github.rest.issues.createComment({
owner: context.repo.owner, repo: context.repo.repo,
issue_number: context.issue.number, body,
});
The gate runs on PRs that touch prompts, retriever, or the eval set. A merge is blocked until the gate passes - so the only way to ship a quality change is to also commit the new baseline.
For nightly scheduled runs against production traffic samples, append each run's metrics to a time series and alert on a moving-window drop:
# drift_alert.py
import statistics
def detect_trend_drift(history: list[dict], metric: str, window: int = 7) -> str | None:
"""history: list of {date, metrics:{...}} newest last."""
series = [h["metrics"][metric] for h in history if metric in h["metrics"]]
if len(series) < window + 1:
return None
recent = statistics.mean(series[-3:])
baseline_window = statistics.mean(series[-(window + 1):-3])
drop = baseline_window - recent
if drop > 0.04:
return (f"{metric} trending down: {baseline_window:.3f} -> {recent:.3f} "
f"over {window} days (drop {drop:.3f})")
return None
dataset.v3.json). Bumping the version is a reviewable, deliberate act.paths: to keep paid LLM-judge calls off unrelated PRs; upload report.json as an artifact and comment scores on the PR.continue-on-error: true) to unblock a deadline. A non-blocking quality gate is theater.Trigger when the user asks to:
For the definitions and scoring of the underlying metrics (faithfulness, context precision/recall, answer relevancy), use the RAG Evaluation Metrics skill. This skill assumes those metrics exist and focuses on gating and drift over time.