| license | Apache-2.0 |
| name | llm-evaluation-harness |
| description | Build automated LLM evaluation pipelines with benchmarks, regression tests, RAGAS, and human eval workflows. Activate on: LLM evaluation, benchmark testing, eval pipeline, RAGAS, model regression tests. NOT for: traditional software testing (testing-expert), model training (ai-engineer). |
| allowed-tools | Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*) |
| category | AI & Machine Learning |
| tags | ["evaluation","benchmarks","ragas","llm-testing","regression"] |
| pairs-with | [{"skill":"ai-engineer","reason":"Evaluation validates LLM application quality before deployment"},{"skill":"prompt-template-manager","reason":"A/B test results feed into prompt version promotion decisions"},{"skill":"fine-tuning-dataset-curator","reason":"Eval sets curated alongside training data measure fine-tune effectiveness"}] |
LLM Evaluation Harness
Build automated evaluation pipelines for LLM applications with benchmarks, regression tests, RAG evaluation (RAGAS), and human eval workflows.
Activation Triggers
Activate on: "evaluate LLM", "benchmark model", "regression test AI", "RAGAS evaluation", "eval pipeline", "LLM quality metrics", "compare model versions", "human evaluation workflow", "test AI responses"
NOT for: Traditional unit/integration testing (testing-expert), model training loops (ai-engineer), or prompt writing (prompt-engineer)
Quick Start
- Define eval dimensions — Correctness, faithfulness, relevance, coherence, safety. Pick the 2-3 that matter most for your use case.
- Build eval dataset — 50-200 curated test cases with expected outputs or rubrics. Include edge cases and adversarial inputs.
- Choose eval methods — LLM-as-judge for scalable scoring, exact-match for structured outputs, RAGAS for RAG systems, human eval for nuance.
- Automate in CI — Run evals on every prompt change, model upgrade, or pipeline modification. Fail the build if scores regress.
- Track trends — Store eval results over time. A 2% quality drop per release compounds into a 20% drop over 10 releases.
Core Capabilities
| Domain | Technologies | Notes |
|---|
| RAG Evaluation | RAGAS, DeepEval, custom | Faithfulness, answer relevance, context precision |
| LLM-as-Judge | Claude, GPT-4o, Llama 3.1 as evaluators | Rubric-based scoring with calibration |
| Exact Match | Regex, JSON schema validation, string match | For structured outputs: classification, extraction |
| Human Eval | Argilla, Label Studio, custom UI | Gold-standard quality, expensive, slow |
| Benchmarks | MMLU, HumanEval, custom domain benchmarks | Standardized comparison across models |
| CI Integration | GitHub Actions, pytest, Vitest | Eval-as-tests with pass/fail thresholds |
Architecture Patterns
Pattern 1: Multi-Method Evaluation Pipeline
Eval Dataset (N test cases)
│
├──→ [Exact Match] ──→ Precision/Recall/F1 (for structured outputs)
│
├──→ [LLM-as-Judge] ──→ Rubric scores 1-5 per dimension
│ │
│ └── Calibrate: run judge on 20 pre-scored examples first
│
├──→ [RAGAS] ──→ Faithfulness, Answer Relevance, Context Precision
│ │
│ └── For RAG systems only; measures retrieval + generation quality
│
└──→ [Human Eval] ──→ Gold-standard labels (sample 10-20%)
│
└── Use for calibrating LLM-as-judge, not as primary method
All results ──→ [Score Aggregation] ──→ [Trend Tracker] ──→ [CI Gate]
