evalyn-calibrate

name	evalyn-calibrate
description	Use when LLM judges need calibration, evaluation metrics seem misaligned with expectations, or annotation and judge tuning is needed

evalyn-calibrate

Pre-flight

1. Verify evaluation runs exist:

evalyn list-runs --limit 1

If no runs: "You need evaluation results first. Invoke evalyn-eval."

2. Identify calibration targets:

evalyn analyze --run <latest-run-id>

Look for subjective metrics (type [llm]) with pass rates below 90%. Only subjective metrics can be calibrated - objective metrics are deterministic.

Step 1: Annotate Results

Run interactive annotation with per-metric mode for targeted feedback:

evalyn annotate --run-id <latest-run-id> --dataset <dataset-path> --per-metric

This is an interactive terminal session. The user will:

• See each item's input, output, and LLM judge results
• Agree or disagree with each metric's judgment
• Commands: [y]es/pass, [n]o/fail, [s]kip, [v]iew full, [q]uit

Aim for 20-30+ annotations. Focus on disagreements. Annotations save immediately - quit and resume anytime.

Step 2: Run Calibration

Start with the basic optimizer (fast, single-shot analysis):

evalyn calibrate --metric-id <target-metric> --annotations <annotations-dir>

The --annotations flag points to the directory containing annotation files (created by evalyn annotate).

The output shows alignment metrics:

• Accuracy: overall agreement rate
• F1 Score: balanced precision/recall
• Cohen's Kappa: agreement adjusted for chance

Optimizer comparison

Optimizer	Flag	Speed	Best For
basic	--optimizer basic	Fast (1 API call)	First pass, small annotation sets
ape	--optimizer ape	Medium	Exploring many prompt variants
opro	--optimizer opro	Medium	Iterative refinement
gepa-native	--optimizer gepa-native	Slow	Best quality, built-in token tracking
gepa	--optimizer gepa	Slow	Evolutionary (requires pip install gepa)
evoprompt	--optimizer evoprompt	Medium	Evolutionary mutation/crossover of prompts
textgrad	--optimizer textgrad	Medium	Critique-revise gradient descent on text
miprov2	--optimizer miprov2	Medium	Instruction + few-shot demo co-optimization
promptbreeder	--optimizer promptbreeder	Slow	Self-referential evolutionary prompt search

Optimizer-specific settings are available as CLI flags (e.g., --opro-iterations, --ape-candidates, --gepa-task-lm, --gepa-reflection-lm, --gepa-max-calls).

If basic doesn't improve alignment enough:

evalyn calibrate --metric-id <target-metric> --annotations <annotations-dir> --optimizer gepa

Step 3: Re-evaluate with Calibrated Judges

evalyn run-eval --dataset <dataset-path> --use-calibrated

The --use-calibrated flag loads optimized prompts from the calibrations/ directory.

Step 4: Compare Results

evalyn compare --run1 <original-run-id> --run2 <calibrated-run-id>

Check whether calibrated pass rates better match your expectations.

Step 5: Assess and Iterate

View calibration history:

evalyn list-calibrations

If alignment improved: calibration successful. The judges now match your expectations.

If alignment is still poor:

• Try a different optimizer (gepa for maximum quality)
• Add more annotations (more data = better calibration)
• Review the metric's rubric in the metrics JSON file - it may need manual editing
• Run evalyn cluster-misalignments --run-id <run-id> to see patterns in disagreements

No Automatic Next Step

This is the terminal skill. The user decides:

• Iterate: add more annotations and re-calibrate
• Fix agent: if judges are correct and agent has real issues
• Expand testing: evalyn simulate --dataset <path> --modes similar,outlier then re-evaluate
• Ship: export final report with evalyn export --run <id> --format html