| name | rag-evaluation |
| description | Evaluate and improve RAG (Retrieval Augmented Generation) pipeline quality. Outputs retrieval evaluation metrics, answer quality benchmarks, end-to-end evaluation framework, and improvement roadmap. |
| argument-hint | ["use case","document corpus","evaluation dataset size","latency requirements"] |
| allowed-tools | Read, Write, Bash |
RAG Evaluation
RAG has two failure points: retrieval (wrong chunks fetched) and generation (wrong answer given right chunks). Evaluation must cover both. Without systematic evaluation, RAG improvements are guesswork and regressions are invisible.
RAG Evaluation Framework
┌──────────────────────────────────────┐
│ RAG PIPELINE │
│ │
Query → Retrieval → Chunks → Generation → Answer │
│ ↑ ↑ │
│ EVAL 1: EVAL 2: │
│ Retrieval Generation │
│ quality quality │
└──────────────────────────────────────┘
EVAL 1 — Retrieval:
Context Precision: Are retrieved chunks relevant?
Context Recall: Are all relevant chunks retrieved?
EVAL 2 — Generation:
Answer Faithfulness: Is the answer grounded in the retrieved context?
Answer Relevance: Does the answer address the question?
END-TO-END:
Correctness: Is the answer factually correct?
Completeness: Does the answer cover all aspects?
Evaluation Dataset Construction
anthropic Anthropic
json
client = Anthropic()
() -> []:
eval_pairs = []
doc documents[:n_questions]:
response = client.messages.create(
model=,
max_tokens=,
messages=[{: , : }]
)
:
pairs = json.loads(response.content[].text)
pair pairs:
pair[] = doc
eval_pairs.append(pair)
json.JSONDecodeError:
eval_pairs
