evaluate

name	evaluate
description	Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.
effort	medium
disable-model-invocation	true
argument-hint	[--threshold N]
allowed-tools	Bash, Read

RAG Evaluation

Evaluate RAG quality using LLM-as-a-Judge methodology.

Usage

/evaluate [--threshold 0.7]

Execution

Direct Execution (recommended for most projects)

# Run RAG evaluation
python3 scripts/evaluate_rag.py

# With custom thresholds
python3 scripts/evaluate_rag.py \
  --faithfulness 0.7 \
  --relevancy 0.7 \
  --context 0.6

# Detect knowledge gaps
python3 scripts/knowledge_gaps.py --detect

# Generate gap report
python3 scripts/knowledge_gaps.py --report

Docker Execution (containerized projects)

# Replace {api-container} with your API server container name
docker exec {api-container} python3 scripts/evaluate_rag.py

# With custom thresholds
docker exec {api-container} python3 scripts/evaluate_rag.py \
  --faithfulness 0.7 \
  --relevancy 0.7 \
  --context 0.6

# Detect knowledge gaps
docker exec {api-container} python3 scripts/knowledge_gaps.py --detect

# Generate gap report
docker exec {api-container} python3 scripts/knowledge_gaps.py --report

Metrics

Metric	Description	Target
Faithfulness	Is answer based on context?	>70%
Relevancy	Does answer address question?	>70%
Context Precision	Is found context accurate?	>60%

Evaluation Process

1. Generate test queries from golden dataset
2. Execute RAG pipeline for each query
3. LLM judges each response on metrics
4. Report aggregate scores

Golden Dataset

Located at: scripts/golden_dataset.json (or project-specific path)

{
  "queries": [
    {
      "query": "How to configure rate limiting?",
      "expected_topics": ["nginx", "rate-limiting"],
      "expected_sources": ["kb/nginx/howto/rate-limiting.md"]
    }
  ]
}

Output Example

RAG Evaluation Results
======================
Total Queries: 50
Average Faithfulness: 0.82
Average Relevancy: 0.78
Average Context Precision: 0.71

Quality: GOOD

Failed Queries (faithfulness < 0.7):
- Query: "How to backup PostgreSQL?"
  Score: 0.45
  Issue: No relevant documents found

Knowledge Gaps

After evaluation, check for gaps:

# Direct execution
python3 scripts/knowledge_gaps.py --detect

# Docker execution
docker exec {api-container} python3 scripts/knowledge_gaps.py --detect

Output:

Knowledge Gaps Detected:
1. PostgreSQL backup procedures (5 failed queries)
2. Redis caching configuration (3 failed queries)
3. Ollama model selection (2 failed queries)

Quality Gates

• Faithfulness >70%
• Relevancy >70%
• Context Precision >60%
• No critical knowledge gaps

Rules

• MUST use a golden dataset — never evaluate on synthetic queries only
• NEVER report a score without listing the failed queries alongside it
• CRITICAL: if the golden dataset is missing, stop and ask the user to provide one
• MANDATORY: thresholds come from project config, not hardcoded defaults, when available

Gotchas

• LLM-as-a-judge scores are non-deterministic; a single run fluctuates by ±10 points even with temperature=0. Always report the average and stddev over ≥3 runs, not a one-shot number.
• The default threshold trio (0.7 / 0.7 / 0.6) was calibrated on English KBs. Multilingual corpora (Polish + English in the same index) score systematically 5-15 points lower — recalibrate per language, or split the golden dataset by language.
• Golden datasets drift: when the KB is reindexed or documents are renamed, expected_sources may point at moved or deleted paths. A sudden drop in context_precision across unrelated queries usually means dataset rot, not RAG regression — validate the dataset paths first.
• Judges often reward verbose answers as "more faithful" because there is more text to ground. Tune the judge prompt to penalize padding, or cap answer length in the generator before evaluation.

When NOT to Use

• For auditing skill quality (the 5-criteria check) — that lives in scripts/evaluate_skills.py
• For general-purpose LLM output scoring without a KB — use /review or a tailored prompt
• For unit tests or code correctness — use /test
• For continuous evaluation without a golden dataset — build the dataset first