| name | nasde-benchmark-calibration |
| description | Calibrate assessment rubrics by reviewing agent work in GitHub/GitLab PRs and feeding human comments back into the rubric. Use this skill when the user wants to:
- Calibrate, tune, or sanity-check assessment criteria / dimensions of a benchmark
- Review trial diffs alongside the LLM-as-a-Judge scores in a PR/MR
- Investigate why judge scores feel off, too harsh, too lenient, or misaligned with how a human would grade the code
- Pull review comments back from PRs/MRs and turn them into concrete rubric edits
Even if the user doesn't say "calibrate" — if they're worried the LLM judge's scores diverge from human judgment, or want to align scores with a real developer's opinion before freezing a benchmark, this skill applies.
|
NASDE Benchmark Calibration
Close the loop between the LLM-as-a-Judge and a human reviewer. The judge scores trials against
assessment_criteria.md (per task) and assessment_dimensions.json (benchmark-wide) — but an LLM
judge is an imperfect grader, and how it reads the rubric may diverge from how a human grades the code.
This skill publishes trial diffs + scores as Pull/Merge Requests for human review, pulls the comments
back, and proposes concrete rubric edits.
This is the third skill in the benchmark lifecycle: nasde-benchmark-creator writes the rubric,
nasde-benchmark-runner runs trials and scores them, and this skill calibrates the rubric against
human judgment before the benchmark is frozen.
Prerequisites
Workflow
1. Decide which trials to calibrate
Calibration is about the criteria, not individual runs. Discuss with the user which trials to
publish: a representative trial per (task, variant) is usually enough; publishing every repetition
floods the sink with PRs. Selecting trials is the agent's judgment here — there is no --select flag
yet (that logic may move into the CLI once the workflow settles).
You can pass job directories (all their trials) or individual trial directories — mixed is fine.
2. Publish
nasde calibrate publish jobs/<job>/<trial>__<id> -C <project_dir>
nasde calibrate publish jobs/<job> -C <project_dir>
Each trial becomes one PR/MR:
- An orphan base branch
base/<repo>-<sha> holds the agent's start-state codebase (seeded once
per (repo, commit); git deduplicates shared blobs across bases).
- The feature branch
calib/<repo>-<sha>/<trial> carries the agent's diff applied as a real
commit, so the PR diff is exactly the agent's work — clean to review.
- The description renders the dominant evaluator cluster's per-dimension
mean ± std, the
normalized score, and a "How to calibrate" note.
- A
.calibration/ directory ships the review context: the task's instruction.md (what the
agent was asked to do), the assessment_criteria.md + assessment_dimensions.json the judge scored
against, every assessment_eval_<N>.json (full per-dimension reasoning), and metrics.json. This
lets the reviewer judge whether a score is fair without leaving the PR.
Re-running is idempotent — a trial whose open PR/MR already exists is skipped. Closing a
calibration round lets the same trials be re-published into a fresh round. Give the user the PR/MR
URLs and ask them to review the diffs and comment inline where a score disagrees with their judgment.
If a second person should review, the user adds them as a collaborator on the sink repo.
3. Pull the comments back
After the user has reviewed:
nasde calibrate pull-comments jobs/<job>/<trial>__<id> -C <project_dir> --json
--json emits a machine-readable structure (issue-level + inline comments, each with path/line
when inline) — consume it directly. Without --json it prints a summary table.
4. Diagnose the divergence per dimension
For each comment, line it up against the judge's score for that trial (read
assessment_summary.json for the per-dimension mean ± std, and assessment_eval_*.json for the
per-dimension reasoning). Inline comments carry a path/line — map that back to which
dimension's rubric is implicated.
Classify each divergence:
- Judge too harsh — human says the code is fine, judge scored low → the rubric's threshold for
that dimension is too strict, or its description rewards the wrong thing.
- Judge too lenient — human flags a problem the judge missed → the rubric doesn't describe the
failure mode, or its scale lacks resolution there.
- Aligned — no action; record it as a positive signal.
A human "I think this score should be X" comment is a free-text gold label — interpret it; do not
expect a rigid format.
5. Propose the rubric edit (for approval)
The rubric to edit lives at evals/<source>/tasks/<task>/assessment_criteria.md (per task) or
evals/<source>/assessment_dimensions.json (benchmark-wide, when the divergence is about a
dimension's scale/description, not one task's thresholds). The <source> and <task> come from the
trial's result.json (source, task_name).
Calibration can also restructure a single task's dimensions: an assessment_dimensions.json
placed next to the task's assessment_criteria.md overrides the benchmark-wide file for that task
only (different fingerprint — old and new evaluations are never mixed in one summary group). For
change-related checks, the evaluator already hands every judge the agent's full diff
(<trial>/agent_changes.diff + inline diffstat) — rubrics should direct the judge to answer
"what did the agent change" questions from that diff. When a task additionally needs hard mechanical
enforcement (e.g. a disqualification score cap), ship a precheck.sh — the evaluator runs it,
injects its JSON into the judge prompt as ground facts, and enforces its optional
normalized_score_cap. See examples/ddd-architectural-challenges/tasks/ddd-weather-discount/ for
the reference calibrated task and CALIBRATION_ROUND2_2026-07-07.md for the loop's acceptance
criteria (repeatability, judge-model agreement, human-ranking correlation, dimension disjointness,
regression assertions).
Show the user a concrete diff of the rubric — the specific threshold/description change that would
have moved the judge toward the human's score — and wait for approval before writing. Never edit
the rubric silently.
After the edit, the loop repeats: re-run the trial through nasde-benchmark-runner (or re-eval an
existing job with nasde eval) and re-publish to confirm the judge now aligns. This iterative
measure → diagnose → fix → re-measure loop is the point of calibration.
Notes
- Trajectories are intentionally NOT published to the PR (they may contain secrets and clutter the
diff). The PR carries the diff, the assessment files, and metrics only.
- One trial = one PR. All of that trial's repeated evaluations are summarized in the description and
shipped as
.calibration/assessment_eval_<N>.json.
- The platform is inferred from the repo URL — GitHub and GitLab both work with no code change, only
the right CLI installed.
