evaluation

name	evaluation
description	Use for tasks that compute, compare, optimize, or report metrics: classification and retrieval metrics, campaign uplift, model or prompt comparisons, LLM-as-judge reports, statistical significance, budgeted optimization, and behavior-preserving performance improvements.

Evaluation

When To Use

Use this skill when the task is primarily about measuring quality or choosing a better configuration. The inputs may be predictions, model outputs, prompts, experiments, optimization traces, or human preference records.

First Pass

1. Identify the unit of evaluation: row, query, prompt, sample, pairwise vote, model output, or experiment run.
2. Extract metric formulas and averaging rules from the task. Macro and micro averages are not interchangeable.
3. Preserve baselines so improvement claims can be checked.
4. Save both aggregate metrics and enough per-item detail to diagnose failures when the task asks for reports.

Implementation Patterns

Classification Metrics

from sklearn.metrics import accuracy_score, precision_recall_fscore_support, roc_auc_score

precision, recall, f1, _ = precision_recall_fscore_support(
    y_true, y_pred, average="binary", zero_division=0
)
metrics = {
    "accuracy": accuracy_score(y_true, y_pred),
    "precision": precision,
    "recall": recall,
    "f1": f1,
}

For multilabel tasks, use the requested averaging mode and threshold.

Retrieval Metrics

Compute NDCG@K, MAP@K, and Recall@K per query and then average. Relevance judgments are keyed by query and document ID, not by row index.

Campaign and Causal Evaluation

For uplift/matching tasks, keep treatment/control definitions clear, compute balance diagnostics, and report effect estimates with confidence intervals when requested. Do not compare raw post-treatment means if matching or propensity scores are part of the task.

Prompt and LLM Evaluation

For prompt optimization, separate candidate generation from scoring. Keep the evaluation set fixed across candidates and record the best prompt plus scores for all attempted prompts. If using LLM-as-judge, save judge prompts, raw scores, and aggregation rules.

Deterministic NLG Metrics

For text-generation scoring without network/GPU, use offline reference-based metrics:

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
from rouge_score import rouge_scorer
import numpy as np

# Sentence-level BLEU with smoothing (handles short outputs)
smooth = SmoothingFunction().method1
bleu = sentence_bleu([ref.split()], hyp.split(), smoothing_function=smooth)

# ROUGE-L with stemming
scorer = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
rouge_l = scorer.score(ref, hyp)["rougeL"].fmeasure

# Normalized exact-match (case/whitespace insensitive)
em = int(ref.lower().strip().split() == hyp.lower().strip().split())

# Bootstrap 95% CI over per-item scores
rng = np.random.default_rng(42)
boot = [rng.choice(scores, size=len(scores), replace=True).mean() for _ in range(1000)]
ci_low, ci_high = np.percentile(boot, [2.5, 97.5])

Seed bootstrap RNG so the CI is reproducible across runs.

Behavior-Preserving Optimization

When optimizing runtime or memory, prove behavior equivalence first, then report resource changes. Use the same inputs for before/after measurement.

Validation

• Recompute metrics from saved predictions, not from in-memory variables only.
• Check all requested rounding and JSON key names.
• Validate metric direction: higher is better for AUC/NDCG/F1, lower is better for loss/error.
• Confirm the same evaluation set is used for all compared systems.
• Include per-class or per-query breakdowns when aggregate metrics can hide regressions.

Common Failures

• Mixing macro, micro, and weighted averages.
• Evaluating after filtering out hard examples.
• Selecting the best config on test data when validation is required.
• Rounding before ranking configurations.
• Reporting improvement without the baseline metric.