rag-evaluation-observability

name	rag-evaluation-observability
description	Use when designing RAG/agent evaluation frameworks, golden datasets, retrieval metrics, generation scoring, latency benchmarks, regression gates, traces, and AI observability.
version	1.0.0
author	Hermes Agent
license	MIT
platforms	["linux","macos","windows"]
metadata	{"hermes":{"tags":["rag","evaluation","observability","metrics","regression","llmops"],"related_skills":["principal-ai-architect","production-rag-architecture","weights-and-biases","evaluating-llms-harness"]}}

RAG Evaluation and Observability

Overview

Use this skill when building or reviewing evaluation and observability for RAG and agentic systems. Production AI can look operationally healthy while producing low-quality answers, so evals and traces must cover retrieval, context use, generation, routing, tool calls, latency, cost, and regression behavior.

When to Use

Use when asked to:

• build an evaluation framework
• define golden datasets
• choose retrieval/generation metrics
• create regression gates
• benchmark latency
• instrument cost, token usage, or quality signals
• design anomaly detection
• close the loop from production failures to tests

Evaluation Architecture

Golden datasets
  -> retrieval evals: Recall@k, MRR, NDCG@k, precision
  -> generation evals: faithfulness, relevance, completeness
  -> agent evals: task completion, tool correctness, escalation behavior
  -> latency benchmarks: p50/p95/p99 by span
  -> cost benchmarks: tokens, model tier, reranker, failed request cost
  -> regression gate: compare against baseline
  -> production sampling: online quality/drift signals
  -> issue creation: convert failures into tests and implementation work

Golden Dataset Schema

Each example should include:

• id
• question or task
• expected_answer or grading rubric
• relevant_doc_ids
• relevant_chunk_ids when available
• required_citations
• query_type
• expected_namespace or strategy when known
• difficulty
• freshness_expectation
• business_workflow
• expected_tool_behavior for agents
• no_answer_expected when applicable

Required Dataset Slices

Slice	Purpose
factual lookup	precise retrieval and MRR
policy/process answer	completeness and citations
comparison/synthesis	cross-document retrieval and NDCG
exact lexical lookup	hybrid retrieval
ambiguous query	clarification behavior
stale/conflicting source	freshness and conflict handling
no-answer query	hallucination resistance
tool-action workflow	agent planning and tool safety

Metrics

Layer	Metric	Interpretation
Retrieval	Recall@k	Did we retrieve the needed source?
Retrieval	MRR	How early did the first relevant result appear?
Retrieval	NDCG@k	Are better results ranked higher?
Retrieval	Context precision	Are retrieved chunks useful or noisy?
Generation	Faithfulness / groundedness	Are claims supported by context?
Generation	Relevance	Did the answer address the question?
Generation	Completeness	Did it cover required aspects?
Agent	Task completion rate	Did the workflow finish?
Agent	Tool error rate	Are integrations reliable?
Operations	p50/p95/p99 latency	Is UX acceptable?
Economics	Cost per successful answer	Are unit economics acceptable?
Regression	Baseline delta	Did the change make anything worse?

Regression Gate Defaults

Block or require explicit approval when:

• Recall@5 drops more than 5% on critical slices
• MRR drops more than 5% on factual lookup
• NDCG@5 drops more than 5% on synthesis/comparison
• faithfulness falls below threshold
• p95 latency increases more than 20% without justification
• cost per successful answer increases more than 20% without quality lift
• no-answer hallucination rate increases
• tool-call failure rate increases materially

Observability Trace Fields

Every AI request should capture:

• trace_id
• user_workflow
• intent_class
• risk_level
• namespace_candidates
• selected_namespace
• retrieval_top_k
• retrieved_chunk_ids
• reranker_used
• prompt_version
• model_selected
• input_tokens
• output_tokens
• estimated_cost
• latency_ms_by_span
• quality_score
• failure_mode

Anomaly Signals

Create alerts or investigation issues when:

• cost per request exceeds baseline by 3 standard deviations
• p95 latency degrades materially
• retrieval recall drops for monitored slices
• premium-model usage spikes without query mix change
• no-answer queries receive confident answers
• namespace traffic shifts unexpectedly
• tool failure rate increases
• groundedness drops below threshold

Common Pitfalls

1. Only judging final answers. RAG failures often originate in retrieval or context assembly.
2. No chunk IDs in golden data. Without IDs, retrieval metrics become subjective.
3. No no-answer tests. Hallucination resistance needs explicit negative examples.
4. Ignoring latency/cost. Quality wins that double p95 or cost may not be production wins.
5. No production feedback loop. Online failures should become offline regression cases.
6. Opaque LLM judges. Judge prompts need rubrics, scales, and examples.

Verification Checklist

• Golden dataset includes source/chunk IDs where possible
• Retrieval and generation are scored separately
• Agent tool behavior is tested when applicable
• Regression thresholds are explicit
• Latency and cost are benchmarked
• Trace fields support root-cause analysis
• Online failures can be promoted into offline eval cases
• Quality, cost, and latency are visible by namespace and model tier