Systematic evaluation of ML models, experiments, and AI system outputs. Multi-dimensional rubrics, LLM-as-judge, bias detection, and structured comparison frameworks. Use when the user asks to "evaluate model performance", "compare models", "build evaluation rubrics", "assess output quality", "detect model bias", or mentions evaluation frameworks, LLM-as-judge, model comparison, or quality assessment.
Systematic evaluation of ML models, experiments, and AI system outputs. Multi-dimensional rubrics, LLM-as-judge, bias detection, and structured comparison frameworks. Use when the user asks to "evaluate model performance", "compare models", "build evaluation rubrics", "assess output quality", "detect model bias", or mentions evaluation frameworks, LLM-as-judge, model comparison, or quality assessment.
allowed-tools
Bash, Read, Write, Glob, Grep
argument-hint
path to predictions, results.tsv, or model outputs (e.g. "results.tsv" or "predictions.csv")
Frameworks for systematic evaluation of ML models, experiment results, and AI system outputs. Covers traditional ML metrics, LLM-as-judge patterns, bias detection, and structured comparison.
Statistical significance: Is the improvement real or noise?
Bootstrap confidence intervals (1000 resamples)
Paired t-test or Wilcoxon for matched samples
If delta < 2x std of CV folds, it's likely noise
Efficiency trade-off: Score per resource unit
Score improvement / memory increase
Score improvement / training time increase
Diminishing returns detection
Robustness: Does it hold across conditions?
Cross-validation variance (low = robust)
Performance on minority classes
Sensitivity to random seed
Deployability: Can it run in production?
Inference latency (ms per prediction)
Memory footprint at serving time
Dependency complexity
LLM-as-judge evaluation
For evaluating LLM application outputs (RAG, chatbots, agents):
Direct scoring
EVAL_PROMPT = """Rate the following AI response on a scale of 1-5 for each dimension.
**Task**: {task_description}
**Input**: {user_input}
**Response**: {model_output}
**Reference** (if available): {reference}
Rate each dimension:
- Relevance (1-5): Does the response address the question?
- Accuracy (1-5): Are the facts correct?
- Completeness (1-5): Does it cover all aspects?
- Clarity (1-5): Is it well-organized and clear?
- Helpfulness (1-5): Would this actually help the user?
Output as JSON:
{{"relevance": X, "accuracy": X, "completeness": X, "clarity": X, "helpfulness": X, "reasoning": "..."}}
"""
Pairwise comparison
More reliable than direct scoring for subjective quality:
PAIRWISE_PROMPT = """Which response better answers the question?
**Question**: {question}
**Response A**: {response_a}
**Response B**: {response_b}
Choose: A is better / B is better / Tie
Explain your reasoning in 2-3 sentences.
"""
Bias mitigation: Run twice with A/B swapped. If results disagree, mark as tie.
Research (MT-Bench, Zheng et al. 2023): pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation. Use direct scoring for objective criteria with clear ground truth; use pairwise for subjective quality comparisons.
Research (BrowseComp benchmark) shows three factors explain 95% of agent performance variance:
Token usage: 80% of variance — more context/turns = better performance
Number of tool calls: ~10%
Model choice: ~5%
Implication: evaluate with realistic token budgets, not unlimited resources. Upgrading from an older model to Claude Sonnet 4.5 or GPT-5.2 provides larger gains than doubling token budget on the same model.
Test set design
Complexity stratification
Build test sets covering multiple difficulty levels:
test_set = {
"simple": [
# Single-step, clear answer, common patterns
],
"medium": [
# Multi-step, some ambiguity, less common patterns
],
"complex": [
# Many steps, significant ambiguity, edge cases
],
"adversarial": [
# Deliberately tricky, boundary conditions, known failure modes
]
}
