Menu
Score RAG answers for faithfulness, answer relevancy, and correctness using LLM-as-judge methods.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Use this skill when building, debugging, or improving Retrieval-Augmented Generation systems, including chunking, vector database selection, hybrid search, reranking, multimodal RAG, code documentation RAG, retrieval latency, and production RAG architecture.
Chunk nested documents into parent-child levels so retrieval can move from broad sections to fine-grained passages.
Use semantic boundaries and embedding similarity to chunk text for higher-relevance retrieval.
Route RAG chunking decisions across semantic, hierarchical, sliding-window, contextual-header, and framework-selection strategies.
Use overlapping windows to preserve context across chunk boundaries while controlling retrieval size.
Reduce retrieval latency with caching, batching, and index-level optimization.
| name | generation-evaluation-metrics |
| title | Generation Evaluation Metrics |
| description | Score RAG answers for faithfulness, answer relevancy, and correctness using LLM-as-judge methods. |
| allowed-tools | ["Read","Grep","Glob","Bash"] |
| category | evaluation-metrics |
| tags | ["generation","faithfulness","llm-as-judge","hallucination"] |
Generation evaluation measures the answer itself: whether it is grounded in the retrieved context, addresses the question, and is factually correct. These metrics catch hallucination and drift that retrieval metrics cannot see.
A pipeline can retrieve perfect context and still produce a wrong, unsupported, or off-topic answer. Exact-match and BLEU-style scores fail on open-ended answers, so teams need semantic and grounding-aware evaluation instead.
Score faithfulness against the retrieved context and correctness against a reference answer as two different numbers.
Why: An answer can be faithful to bad context yet wrong, or correct yet unsupported; conflating them hides both failure modes.
Define an explicit rubric and score with a strong judge model, requesting a short justification alongside each score.
Why: A fixed rubric and recorded rationale make LLM judgments repeatable and auditable instead of arbitrary.
Spot-check a sample of judge scores against human labels and adjust the rubric until they agree.
Why: An uncalibrated judge produces confident but misleading numbers; calibration keeps automated scores trustworthy.
For rubrics and implementations, see RAGAS generation metrics, DeepEval metrics, and the G-Eval paper.
