메뉴
Score RAG answers for faithfulness, answer relevancy, and correctness using LLM-as-judge methods.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Use this skill when building, debugging, or improving Retrieval-Augmented Generation systems, including chunking, vector database selection, hybrid search, reranking, multimodal RAG, code documentation RAG, retrieval latency, and production RAG architecture.
Chunk nested documents into parent-child levels so retrieval can move from broad sections to fine-grained passages.
Use semantic boundaries and embedding similarity to chunk text for higher-relevance retrieval.
Route RAG chunking decisions across semantic, hierarchical, sliding-window, contextual-header, and framework-selection strategies.
Use overlapping windows to preserve context across chunk boundaries while controlling retrieval size.
Reduce retrieval latency with caching, batching, and index-level optimization.
| name | generation-evaluation-metrics |
| title | Generation Evaluation Metrics |
| description | Score RAG answers for faithfulness, answer relevancy, and correctness using LLM-as-judge methods. |
| allowed-tools | ["Read","Grep","Glob","Bash"] |
| category | evaluation-metrics |
| tags | ["generation","faithfulness","llm-as-judge","hallucination"] |
Generation evaluation measures the answer itself: whether it is grounded in the retrieved context, addresses the question, and is factually correct. These metrics catch hallucination and drift that retrieval metrics cannot see.
A pipeline can retrieve perfect context and still produce a wrong, unsupported, or off-topic answer. Exact-match and BLEU-style scores fail on open-ended answers, so teams need semantic and grounding-aware evaluation instead.
Score faithfulness against the retrieved context and correctness against a reference answer as two different numbers.
Why: An answer can be faithful to bad context yet wrong, or correct yet unsupported; conflating them hides both failure modes.
Define an explicit rubric and score with a strong judge model, requesting a short justification alongside each score.
Why: A fixed rubric and recorded rationale make LLM judgments repeatable and auditable instead of arbitrary.
Spot-check a sample of judge scores against human labels and adjust the rubric until they agree.
Why: An uncalibrated judge produces confident but misleading numbers; calibration keeps automated scores trustworthy.
For rubrics and implementations, see RAGAS generation metrics, DeepEval metrics, and the G-Eval paper.
