evaluating-with-ragas

name	evaluating-with-ragas
description	Evaluating RAG pipelines using RAGAS metrics (context_recall, context_precision, faithfulness, answer_relevancy). Use when measuring retrieval quality, comparing RAG configurations, or running LLM-as-judge evaluation. Triggers on "evaluate", "RAGAS", "measure recall", "context precision".

Evaluating with RAGAS

Evaluate RAG systems using RAGAS framework with gpt-4o-mini.

When to Use

• User says "evaluate results" or "measure quality"
• Need context_recall, context_precision scores
• Comparing RAG configurations objectively

Progress Checklist

[ ] 1. Validate input data format
[ ] 2. Filter out invalid results (empty contexts/answers)
[ ] 3. Run RAGAS evaluation
[ ] 4. Handle NaN values in results
[ ] 5. Return cleaned metrics dict

Core Rules

YOU MUST:

• Filter results with empty contexts or blank answer BEFORE calling RAGAS
• Convert NaN values to 0.0 in final output
• Use gpt-4o-mini for evaluation (cost-effective)

YOU MUST NOT:

• Pass empty contexts to RAGAS (causes NaN)
• Ignore the ground_truth field (required for context_recall)

Data Format

RAGAS requires exactly this structure:

{
    "question": ["Q1", "Q2"],
    "answer": ["A1", "A2"],
    "contexts": [["chunk1", "chunk2"], ["chunkA"]],
    "ground_truth": ["GT1", "GT2"],
}

Quick Implementation

from ragas import evaluate
from ragas.metrics import context_recall, context_precision, faithfulness, answer_relevancy
from datasets import Dataset

def evaluate_rag(results: list[dict]) -> dict:
    # Filter invalid
    valid = [r for r in results if r.get("contexts") and r.get("answer", "").strip()]
    
    dataset = Dataset.from_dict({
        "question": [r["question"] for r in valid],
        "answer": [r["answer"] for r in valid],
        "contexts": [r["contexts"] for r in valid],
        "ground_truth": [r["ground_truth"] for r in valid],
    })
    
    scores = evaluate(dataset, metrics=[
        context_recall, context_precision, faithfulness, answer_relevancy
    ])
    
    # Clean NaN
    return {k: 0.0 if math.isnan(v) else round(v, 4) for k, v in scores.items()}

Metric Interpretation

Metric	Good	Meaning
context_recall	>0.7	Retrieved the right chunks
context_precision	>0.7	Retrieved chunks are relevant
faithfulness	>0.8	Answer sticks to context
answer_relevancy	>0.8	Answer addresses question

Primary metric for experiments: context_recall

References

• See references/ragas_setup.md for detailed setup
• See references/handling_errors.md for error patterns