tune

name

tune

description

Run one eval-driven instruction-tuning iteration on the harness itself. A blind sub-agent solves a CLOSED PrairieLearn issue (no access to the merged fix or this conversation), a blind judge compares its answer to the accepted PR, and you diagnose where the harness/.claude failed it and apply GENERALIZED improvements. The product is the instructions, not the PL fix — the code is thrown away. Runs inline (parent-orchestrated). Use when the user says "tune", "improve the harness", "run an eval iteration", "look back how can we improve .claude", or after picking a closed-issue case.

tune — the instruction-tuning loop

You are the static parent of an eval loop (recurse pattern). The harness .claude/* + scripts/ + packages/ is the mutable artifact being tuned. A real closed PL issue is only a probe. The metric: can a blind agent, driven only by the tuned instructions, reach the accepted answer? Closing that gap — generally — is the whole job.

MANDATORY law: read and obey rules/generalize.md. Every change generalizes or it is dropped. Nothing specific.

Invariants

• Blindness. The solver never sees the merged fix, ground-truth.diff, evals/, the tune skill, generalize.md, or this conversation. It is driven only by the curated .claude + the scrubbed issue.md. Build its workspace with evals/bin/setup-blind.sh, which enforces this and runs a leakage check.
• The judge is blind too. It sees only the solver's diff, the ground-truth diff, and the issue — not our context, not the tune frame.
• Code is disposable. The solver's fix and the merged diff are probes, discarded at the end. Only the forensics in evals/runs/ and the generalized .claude/harness improvements survive.
• One probe at a time. Finish the loop, throw away the code, then the next case.

The loop

1. Prepare the case (once per issue): evals/cases/<issue>/ holds issue.md (scrubbed — symptom + repro, no fix pointers), ground-truth.diff (the merged PR), meta.json (base_commit = buggy tree). Scrub aggressively: remove PR/issue numbers, permalinks, named root-causes, and any test the reporter wrote. Base-commit choice drives boot cost: the warm image's baked deps must match the base commit's, or boot crashes on a missing module and you must rebuild the image at that commit (build-image.sh pl-debug:base <worktree>, slow). Prefer the most recent buggy commit (e.g. the fix's first-parent) so the base sits near the warm image and the run mirrors real current-code debugging — rebuild at the base only when its deps genuinely diverge.
2. Set up the blind workspace: ./evals/bin/setup-blind.sh <case-id> <iter> (e.g. issue-NNNNN 003). Confirm its leakage check passes (it reads base_commit/pr/leak_terms from the case meta.json — no case specifics live in the script).
3. Run the blind solver (one Agent): its entire world is the /tmp workspace. Tell it only: read .claude/ and issue.md, reproduce and fix the bug, output a unified diff + a one-paragraph root cause. Capture → evals/runs/iter-<n>/solver-answer.diff. Save its process notes.
4. Grade — strongest grader first:
   • Deterministic oracle (use whenever the PR ships a regression test). evals/bin/grade-oracle.sh <case-id> <worktree> <container> drops the held-out PR test into the worktree that already holds the solver's fix and runs it in the warm journey container — PASS/FAIL is objective, and it grades behavior, so a solver that implemented the fix differently than the merged PR still passes iff its fix is actually correct. Add an oracle block to the case meta.json ({ "test_files": [...repo-relative...], "cmd": "npx vitest run <path>" }); the test lands only in ground-truth.diff, so setup-blind already holds the new cases out of the solver's view. Note: PL's vitest globalSetup needs Postgres, so run it in the journey container (which has it), not a bare one.
   • Blind judge agent (always — for process, and as the only grader when there's no test). Give it only the solver diff, ground-truth.diff, and issue.md. It scores same-root-cause? / same-effect? and lists concrete gaps in the solver's process. Capture → judge-verdict.md. When the oracle disagrees with the judge, the oracle wins on correctness; the judge still informs process diagnosis.
5. Diagnose → diagnosis.md. For each gap ask: which instruction was soft, missing, or contradictory such that a blind agent went wrong? Record FINDING → ROOT CAUSE → FIX (general text) → FILE → GENERALIZED=yes/no. Mental model: "would a fresh agent reading these files make the choice I'd make?" If not, that's the gap.
6. Improve — generalized only. Apply only GENERALIZED=yes fixes. For each edit:
   • Consistency-check: grep -rn "<concept>" .claude/ — every mention must now agree (a soft "consider" elsewhere undoes a hard "MUST" here).
   • Prune: delete redundancy; shorter instructions are followed better. CLAUDE.md stays < 250 lines.
   • Record applied changes → improvements.md.
7. Throw away the code: git -C $PL_MAIN worktree remove --force the solver's PL tree; rm -rf /tmp/pltune-iter-<n>. Keep only evals/runs/ + the instruction edits.
8. Commit the harness improvements (instructions only). Then the next probe.

Pass-rate — reliability, not just capability

One solve proves the harness can reach the answer; it doesn't prove it reliably does. For a case that matters, run the blind solver K times (independent setup-blind workspaces, spawned in parallel) and tally the deterministic-oracle verdicts → a pass-rate (e.g. 3/3). Grade each trial by applying its solver-answer.diff to one shared oracle worktree and running grade-oracle.sh (reuse a single Postgres-up container across trials). Variance is itself a finding: if 1/K fails, some instruction is underspecified in a spot the other trials happened to get right — diagnose that, generalized. n=1 hides this; n=K surfaces it.

Solver agent prompt (template)

You are working in <WS>. Read everything under .claude/ (your operating instructions) and issue.md (a bug report). Reproduce the bug empirically (run it with injected trace logs; screenshot any rendered surface) per your instructions, find its root cause, and fix it in the PL source under prairielearn/. Do NOT consult GitHub for any PR or fix; do NOT create/post a PR or issue. Output: (a) a unified git diff of your change, (b) one paragraph on the root cause and how you located it, (c) the empirical evidence you gathered — or, if you could not run the stack, exactly what blocked you, (d) a human-readable fix summary (problem / root cause / what changed and why / evidence) a maintainer could read at a glance. Budget ~60 tool calls.

Treat "could not run empirically" as a first-class finding, not a footnote: if the solver had to fall back to static reasoning, the harness failed its core purpose — diagnose the boot/dependency gap and fix it (generalized) like any other gap. A static-only solve that the judge happens to score well is still a harness failure to record.

Judge agent prompt (template)

Compare a candidate bug fix against the accepted fix. You get: issue.md (the bug), ground-truth.diff (accepted), solver-answer.diff (candidate). Score: SAME ROOT CAUSE (yes/partial/no), SAME EFFECT (yes/partial/no), and list specific places the candidate's approach was weaker or stronger. Do not assume either is correct because it's "official" — reason from the bug. Output a short structured verdict. Budget ~30 tool calls.

Cleanup

End with the /tmp workspace and PL worktree removed, git -C $PL_MAIN worktree list clean, and git status in the harness showing only .claude/harness/evals changes. The Stop hook reaps any stray processes.