llm-evaluation-harness

license	Apache-2.0
name	llm-evaluation-harness
description	Build automated LLM evaluation pipelines with benchmarks, regression tests, RAGAS, and human eval workflows. Activate on: LLM evaluation, benchmark testing, eval pipeline, RAGAS, model regression tests. NOT for: traditional software testing (testing-expert), model training (ai-engineer).
allowed-tools	Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*)
category	AI & Machine Learning
tags	["evaluation","benchmarks","ragas","llm-testing","regression"]
pairs-with	[{"skill":"ai-engineer","reason":"Evaluation validates LLM application quality before deployment"},{"skill":"prompt-template-manager","reason":"A/B test results feed into prompt version promotion decisions"},{"skill":"fine-tuning-dataset-curator","reason":"Eval sets curated alongside training data measure fine-tune effectiveness"}]

LLM Evaluation Harness

Build automated evaluation pipelines for LLM applications with benchmarks, regression tests, RAG evaluation (RAGAS), and human eval workflows.

Activation Triggers

Activate on: "evaluate LLM", "benchmark model", "regression test AI", "RAGAS evaluation", "eval pipeline", "LLM quality metrics", "compare model versions", "human evaluation workflow", "test AI responses"

NOT for: Traditional unit/integration testing (testing-expert), model training loops (ai-engineer), or prompt writing (prompt-engineer)

Quick Start

1. Define eval dimensions — Correctness, faithfulness, relevance, coherence, safety. Pick the 2-3 that matter most for your use case.
2. Build eval dataset — 50-200 curated test cases with expected outputs or rubrics. Include edge cases and adversarial inputs.
3. Choose eval methods — LLM-as-judge for scalable scoring, exact-match for structured outputs, RAGAS for RAG systems, human eval for nuance.
4. Automate in CI — Run evals on every prompt change, model upgrade, or pipeline modification. Fail the build if scores regress.
5. Track trends — Store eval results over time. A 2% quality drop per release compounds into a 20% drop over 10 releases.

Core Capabilities

Domain	Technologies	Notes
RAG Evaluation	RAGAS, DeepEval, custom	Faithfulness, answer relevance, context precision
LLM-as-Judge	Claude, GPT-4o, Llama 3.1 as evaluators	Rubric-based scoring with calibration
Exact Match	Regex, JSON schema validation, string match	For structured outputs: classification, extraction
Human Eval	Argilla, Label Studio, custom UI	Gold-standard quality, expensive, slow
Benchmarks	MMLU, HumanEval, custom domain benchmarks	Standardized comparison across models
CI Integration	GitHub Actions, pytest, Vitest	Eval-as-tests with pass/fail thresholds

Architecture Patterns

Pattern 1: Multi-Method Evaluation Pipeline

Eval Dataset (N test cases)
    │
    ├──→ [Exact Match] ──→ Precision/Recall/F1 (for structured outputs)
    │
    ├──→ [LLM-as-Judge] ──→ Rubric scores 1-5 per dimension
    │        │
    │        └── Calibrate: run judge on 20 pre-scored examples first
    │
    ├──→ [RAGAS] ──→ Faithfulness, Answer Relevance, Context Precision
    │        │
    │        └── For RAG systems only; measures retrieval + generation quality
    │
    └──→ [Human Eval] ──→ Gold-standard labels (sample 10-20%)
             │
             └── Use for calibrating LLM-as-judge, not as primary method

All results ──→ [Score Aggregation] ──→ [Trend Tracker] ──→ [CI Gate]