ragas-rag-evaluation

name	Ragas RAG Evaluation
description	Evaluate RAG pipelines with Ragas, measuring faithfulness, answer relevancy, context precision and recall, building golden datasets, and wiring threshold gates into CI for retrieval regressions.
version	1.0.0
author	thetestingacademy
license	MIT
tags	["ragas","rag","llm-evals","faithfulness","context-precision","context-recall","retrieval","python","ci-gates"]
testingTypes	["llm-evals","integration","regression"]
frameworks	["ragas","pytest"]
languages	["python"]
domains	["ai","llm","api"]
agents	["claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp"]

Ragas RAG Evaluation Skill

You are an expert AI quality engineer specializing in Ragas. When the user asks you to evaluate, debug, or regression-test a RAG (retrieval-augmented generation) pipeline, follow these instructions.

Core Principles

1. Separate retrieval failures from generation failures. Ragas metrics split cleanly: context precision/recall judge the retriever, faithfulness/answer relevancy judge the generator. Diagnose before tuning.
2. A RAG eval needs four fields. question, answer, contexts, ground_truth. Build your harness to capture all four; missing ground_truth kills recall metrics.
3. Golden datasets are the asset. The pipeline changes weekly; the dataset is what makes change measurable.
4. Thresholds gate, trends inform. Hard floors in CI, plus week-over-week trend tracking for slow degradation.
5. Judge cost is a design constraint. Sample for PR checks, full-set nightly.

Setup

pip install ragas datasets
export OPENAI_API_KEY=sk-...   # judge + embeddings (other providers configurable)

The Core Metrics

Metric	Judges	Question it answers
faithfulness	Generator	Is every claim in the answer supported by the retrieved contexts?
answer_relevancy	Generator	Does the answer actually address the question?
context_precision	Retriever	Are the relevant chunks ranked above irrelevant ones?
context_recall	Retriever	Did retrieval fetch everything needed to answer?
answer_correctness	End to end	Does the answer match ground truth (factually + semantically)?

Diagnosis table: low faithfulness with high context_recall means the generator ignores or contradicts good context (fix prompting). Low context_recall means retrieval misses content (fix chunking, embeddings, top_k). Low context_precision with high recall means noisy retrieval (fix reranking).

Evaluating a Pipeline

from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    Faithfulness, AnswerRelevancy, LLMContextPrecisionWithReference, LLMContextRecall,
)


rows = []
 item  load_golden():
    result = rag_pipeline.query(item[])
    rows.append({
        : item[],
        : result.answer,
        : [c.text  c  result.chunks],
        : item[],
    })

dataset = EvaluationDataset.from_list(rows)


report = evaluate(
    dataset,
    metrics=[Faithfulness(), AnswerRelevancy(), LLMContextPrecisionWithReference(), LLMContextRecall()],
)
(report)          
df = report.to_pandas()  
df[df[] < ].to_json(, orient=)