| name | llm-evaluation |
| description | Comprehensive LLM evaluation framework with automated metrics, LLM-as-Judge, and RAG evaluation. Trigger: When evaluating LLM outputs, building eval pipelines, comparing models, or measuring RAG quality.
|
| license | MIT |
| metadata | {"author":"gentleman-programming","version":"1.1"} |
Core Principle
Never ship LLM features without evaluation. An untested prompt is a broken prompt you haven't found yet.
Automated Metrics
When to Use Each Metric
| Metric | Best For | Limitation |
|---|
| BLEU | Translation, structured output | Penalizes paraphrasing |
| ROUGE | Summarization | Ignores semantic equivalence |
| BERTScore | General quality | Compute-heavy, model-dependent |
| Exact Match | Classification, extraction | No partial credit |
| Custom Regex | Format validation | Brittle to minor changes |
Key Implementations
- Text similarity: BLEU (n-gram precision), ROUGE (reference recall), BERTScore (semantic embeddings)
- Custom metrics: format compliance (regex patterns), factual overlap (fact checklist against output)
LLM-as-Judge Patterns
Three evaluation modes:
- Single output — score on accuracy, completeness, clarity, conciseness (1-5 each)
- Pairwise comparison — compare two responses head-to-head, run both orderings to reduce position bias
- Reference-based — compare against gold standard for factual consistency, coverage, hallucination detection
Key rule: always run pairwise comparisons in BOTH orderings (A vs B, then B vs A) and check consistency.
@reference references/code-examples.md — Load when implementing metrics, LLM-as-Judge prompts, or RAG evaluation functions
RAG Evaluation
Dimensions
Evaluate RAG pipelines on 4 independent axes:
- Context Relevance — are retrieved contexts relevant to the question?
- Answer Faithfulness — is the answer grounded in contexts (no hallucination)?
- Answer Relevance — does the answer address the question?
- Context Utilization — does the answer use available contexts effectively?
Retrieval Metrics
- Precision@k — fraction of retrieved docs that are relevant
- Recall@k — fraction of relevant docs that were retrieved
- MRR — reciprocal rank of first relevant result
- NDCG@k — normalized discounted cumulative gain (rewards relevant docs at top)
Always evaluate retrieval and generation separately.
A/B Testing and Regression Detection
A/B Testing
Require BOTH statistical AND practical significance:
- Statistical: Welch's t-test, p < 0.05
- Practical: Cohen's d >= 0.2 (minimum meaningful effect size)
- A p-value alone is NOT enough to justify switching
Regression Detection
Use one-sided t-test: is the new version significantly WORSE?
- H0: current >= baseline
- H1: current < baseline
- Flag regression only if p < 0.05 AND percentage drop exceeds threshold
@reference references/frameworks.md — Load when implementing A/B tests, regression detection, LangSmith integration, or eval pipeline scaffolding
Building an Eval Pipeline
Core components:
- EvalCase — question + expected answer + expected facts + tags
- EvalSuite — collection of cases, loaded from JSONL, run through pipeline + evaluators
- Aggregation — mean, std, min, max per metric across all cases
LangSmith Integration
@traceable decorator for automatic tracing- Create datasets with
create_example()
- Run evaluations with custom evaluator functions
- Track experiments with
experiment_prefix
Best Practices
- Start with deterministic metrics first — regex, exact match, format checks are fast and cheap
- Use LLM-as-Judge for subjective quality — but always validate the judge with human annotations first
- Always check position bias in pairwise comparisons — run both orderings
- Require both statistical AND practical significance — a p-value alone is not enough (use Cohen's d)
- Build eval datasets incrementally — add failure cases from production as you find them
- Version your eval datasets — changes to evals are as impactful as changes to prompts
- Run evals in CI — block deployments on regression detection
- Use at least 30 examples per eval to get reliable statistics
- Separate retrieval eval from generation eval in RAG systems
- Log everything — you cannot evaluate what you did not record
Common Pitfalls
- Overfitting to evals: if you tune prompts to maximize eval scores, the eval stops being meaningful. Use held-out test sets.
- LLM-as-Judge bias: Claude/GPT tend to prefer verbose, confident-sounding answers regardless of accuracy. Calibrate with human labels.
- Small sample sizes: 5-10 examples give unreliable means. Target 50+ for serious decisions.
- Ignoring distribution: means hide bimodal distributions. Always look at histograms.
- Evaluating the wrong thing: high BLEU with low user satisfaction means your metric is wrong, not your model.
Keywords
llm evaluation, eval pipeline, bleu, rouge, bertscore, llm-as-judge, rag evaluation, a/b testing, regression detection, langsmith
