| name | tune |
| description | Run one eval-driven instruction-tuning iteration on the harness itself. A blind sub-agent solves a CLOSED PrairieLearn issue (no access to the merged fix or this conversation), a blind judge compares its answer to the accepted PR, and you diagnose where the harness/.claude failed it and apply GENERALIZED improvements. The product is the instructions, not the PL fix — the code is thrown away. Runs inline (parent-orchestrated). Use when the user says "tune", "improve the harness", "run an eval iteration", "look back how can we improve .claude", or after picking a closed-issue case. |
tune — the instruction-tuning loop
You are the static parent of an eval loop (recurse pattern). The harness .claude/* + scripts/ + packages/ is the mutable artifact being tuned. A real closed PL issue is only a probe. The metric: can a blind agent, driven only by the tuned instructions, reach the accepted answer? Closing that gap — generally — is the whole job.
MANDATORY law: read and obey rules/generalize.md. Every change generalizes or it is dropped. Nothing specific.
Invariants
- Blindness. The solver never sees the merged fix,
ground-truth.diff, evals/, the tune skill, generalize.md, or this conversation. It is driven only by the curated .claude + the scrubbed issue.md. Build its workspace with evals/bin/setup-blind.sh, which enforces this and runs a leakage check.
- The judge is blind too. It sees only the solver's diff, the ground-truth diff, and the issue — not our context, not the tune frame.
- Code is disposable. The solver's fix and the merged diff are probes, discarded at the end. Only the forensics in
evals/runs/ and the generalized .claude/harness improvements survive.
- One probe at a time. Finish the loop, throw away the code, then the next case.
The loop
- Prepare the case (once per issue):
evals/cases/<issue>/ holds issue.md (scrubbed — symptom + repro, no fix pointers), ground-truth.diff (the merged PR), meta.json (base_commit = buggy tree). Scrub aggressively: remove PR/issue numbers, permalinks, named root-causes, and any test the reporter wrote. Base-commit choice drives boot cost: the warm image's baked deps must match the base commit's, or boot crashes on a missing module and you must rebuild the image at that commit (build-image.sh pl-debug:base <worktree>, slow). Prefer the most recent buggy commit (e.g. the fix's first-parent) so the base sits near the warm image and the run mirrors real current-code debugging — rebuild at the base only when its deps genuinely diverge.
- Set up the blind workspace:
./evals/bin/setup-blind.sh <case-id> <iter> (e.g. issue-NNNNN 003). Confirm its leakage check passes (it reads base_commit/pr/leak_terms from the case meta.json — no case specifics live in the script).
- Run the blind solver (one
Agent): its entire world is the /tmp workspace. Tell it only: read .claude/ and issue.md, reproduce and fix the bug, output a unified diff + a one-paragraph root cause. Capture → evals/runs/iter-<n>/solver-answer.diff. Save its process notes.
- Grade — strongest grader first:
- Deterministic oracle (use whenever the PR ships a regression test).
evals/bin/grade-oracle.sh <case-id> <worktree> <container> drops the held-out PR test into the worktree that already holds the solver's fix and runs it in the warm journey container — PASS/FAIL is objective, and it grades behavior, so a solver that implemented the fix differently than the merged PR still passes iff its fix is actually correct. Add an oracle block to the case meta.json ({ "test_files": [...repo-relative...], "cmd": "npx vitest run <path>" }); the test lands only in ground-truth.diff, so setup-blind already holds the new cases out of the solver's view. Note: PL's vitest globalSetup needs Postgres, so run it in the journey container (which has it), not a bare one.
- Give it only the solver diff, , and . It scores same-root-cause? / same-effect? and lists concrete gaps in the solver's . Capture → . When the oracle disagrees with the judge, the oracle wins on ; the judge still informs diagnosis.
Pass-rate — reliability, not just capability
One solve proves the harness can reach the answer; it doesn't prove it reliably does. For a case that matters, run the blind solver K times (independent setup-blind workspaces, spawned in parallel) and tally the deterministic-oracle verdicts → a pass-rate (e.g. 3/3). Grade each trial by applying its solver-answer.diff to one shared oracle worktree and running grade-oracle.sh (reuse a single Postgres-up container across trials). Variance is itself a finding: if 1/K fails, some instruction is underspecified in a spot the other trials happened to get right — diagnose that, generalized. n=1 hides this; n=K surfaces it.
Solver agent prompt (template)
You are working in <WS>. Read everything under .claude/ (your operating instructions) and issue.md (a bug report). Reproduce the bug empirically (run it with injected trace logs; screenshot any rendered surface) per your instructions, find its root cause, and fix it in the PL source under prairielearn/. Do NOT consult GitHub for any PR or fix; do NOT create/post a PR or issue. Output: (a) a unified git diff of your change, (b) one paragraph on the root cause and how you located it, (c) the empirical evidence you gathered — or, if you could not run the stack, exactly what blocked you, (d) a human-readable fix summary (problem / root cause / what changed and why / evidence) a maintainer could read at a glance. Budget ~60 tool calls.
Treat "could not run empirically" as a first-class finding, not a footnote: if the solver had to fall back to static reasoning, the harness failed its core purpose — diagnose the boot/dependency gap and fix it (generalized) like any other gap. A static-only solve that the judge happens to score well is still a harness failure to record.
Judge agent prompt (template)
Compare a candidate bug fix against the accepted fix. You get: issue.md (the bug), ground-truth.diff (accepted), solver-answer.diff (candidate). Score: SAME ROOT CAUSE (yes/partial/no), SAME EFFECT (yes/partial/no), and list specific places the candidate's approach was weaker or stronger. Do not assume either is correct because it's "official" — reason from the bug. Output a short structured verdict. Budget ~30 tool calls.
Cleanup
End with the /tmp workspace and PL worktree removed, git -C $PL_MAIN worktree list clean, and git status in the harness showing only .claude/harness/evals changes. The Stop hook reaps any stray processes.
