evaluate

name	evaluate
description	Systematic evaluation of ML models, experiments, and AI system outputs. Multi-dimensional rubrics, LLM-as-judge, bias detection, and structured comparison frameworks. Use when the user asks to "evaluate model performance", "compare models", "build evaluation rubrics", "assess output quality", "detect model bias", or mentions evaluation frameworks, LLM-as-judge, model comparison, or quality assessment.
allowed-tools	Bash, Read, Write, Glob, Grep
argument-hint	path to predictions, results.tsv, or model outputs (e.g. "results.tsv" or "predictions.csv")
model	sonnet
effort	medium
paths	**/results.tsv,**/predictions*.csv,**/metrics*.json
compatibility	>=1.0
metadata	{"category":"model-evaluation","tags":["evaluation","llm-as-judge","bias-detection","metrics","model-comparison","rubrics"],"phase":"evaluate"}

Model & System Evaluation

Frameworks for systematic evaluation of ML models, experiment results, and AI system outputs. Covers traditional ML metrics, LLM-as-judge patterns, bias detection, and structured comparison.

When to use

• Comparing multiple trained models beyond val_score
• Evaluating LLM/AI application outputs (RAG quality, prompt effectiveness)
• Building evaluation rubrics for subjective tasks
• Detecting bias in model predictions
• Creating test sets for systematic assessment
• Deciding between experiment results in results.tsv

Traditional ML evaluation

Classification metrics

Metric	When to use	Formula
Accuracy	Balanced classes	correct / total
Precision	False positives costly	TP / (TP + FP)
Recall	False negatives costly	TP / (TP + FN)
F1	Imbalanced classes	2 * P * R / (P + R)
AUC-ROC	Ranking / threshold selection	Area under ROC curve
Log loss	Probability calibration	-mean(y*log(p))

Regression metrics

Metric	When to use	Sensitivity
RMSE	Penalize large errors	Outlier sensitive
MAE	Robust to outliers	Linear penalty
R-squared	Variance explained	Scale independent
MAPE	Percentage error	Zero sensitive

Beyond single metrics

from sklearn.metrics import classification_report, confusion_matrix
import numpy as np

# Multi-dimensional evaluation
report = classification_report(y_true, y_pred, output_dict=True)

# Per-class performance (find weak spots)
cm = confusion_matrix(y_true, y_pred)
per_class_acc = cm.diagonal() / cm.(axis=)


 sklearn.calibration  calibration_curve
fraction_pos, mean_predicted = calibration_curve(y_true, y_prob, n_bins=)


 group  [, ]:
    mask = df[] == group
    group_score = metric(y_true[mask], y_pred[mask])
    ()