| name | dogfood-marathon |
| description | Process a backlog of real cases (bugs / deliveries / ship-its / tickets) by driving a kitsoki story (bugfix, delivery-tail, ship-it, …) LIVE over each one via the kitsoki-mcp-driver agent, independently verifying every deliverable, capturing friction as findings, and hardening the story/prompts for GENERAL use WITHOUT OVERFITTING to the cases in the run. Use when the user says "run a dogfood marathon", "smash these bugs through bugfix", "process this backlog live", "dogfood <story> over these tickets and fix the pipeline", or wants a marathon report (outcomes / verdicts / cost / fixes shipped / findings). Distinct from story-coverage-mining (transcript-driven coverage) and matrix-task-comparison (harness×model bake-off on fixed tasks). |
Dogfood marathon
Take a backlog of real cases and drive a kitsoki story live over each one,
treating every point of friction as a chance to harden the pipeline for the
general class — never to paper over this one case. The output is fixed
deliverables (or honest needs-human parks), a set of findings, generic
prompt/story hardenings, and a marathon summary.
The discipline that makes it worth anything: improve without overfitting,
and trust only an independent verify — never the maker's self-report.
Read first (don't re-derive):
.context/bakeoff-learnings.md — the full method, the 13 hard-won gotchas, the dogfood findings (F1–F5), the no-overfit principle. This skill is the runbook; that doc is the evidence.stories/bugfix/README.md — the pipeline being driven: rooms, exits (shipped / needs-human / not-reproducible), the RED→GREEN regression gate, bugfix_mode=triage.- The hardened prompts under
stories/bugfix/prompts/ — what a legitimate generic fix looks like (commit d210ea67).
issues/bugs/README.md + the ticket frontmatter for how findings become filed tickets.- MEMORY:
workflow-gate-on-independent-verify, workflow-dead-impl-poisons-demo, bugfix-pipeline-verify-gaps, bugfix-triage-mode, set-block-atomic-determinism.
The marathon loop (per case)
Run the whole loop through the kitsoki-mcp-driver agent — the studio MCP is
the only write surface; you never touch the filesystem to drive the story.
Headless? The in-process Agent/subagent path does NOT attach the studio MCP
(it inherits the parent's empty MCP set → "No MCP servers configured"). To drive
from a script/cron/another agent, use the raw-claude -p primitive
tools/mcp-drive/drive.sh (--mcp-config
--strict-mcp-config); the live worker model is chosen per session by
session.new {profile, harness:"live"}. See MEMORY mcp-first-delegation-runbook.
-
Pick the case and pin its baseline. The baseline is the buggy state
the pipeline must reproduce against — for a merged-fix case that is the fix's
PARENT commit (<fix>^); for a live ticket it is current main (or the SHA the
ticket was filed against). Confirm the bug is actually present at the baseline.
-
(Optional) triage-only pre-flight. Cheap read-only verdict before spending
maker budget. Drive bugfix with bugfix_mode=triage → the triaging room emits a
standardized verdict: ALREADY-FIXED | STILL-LIVE | PARTIAL | UNCLEAR (read-only,
no worktree). Drop ALREADY-FIXED cases (a test/lint layered on an already-merged
behavioural fix → baseline is GREEN → degenerate); only run the full pipeline on
STILL-LIVE / PARTIAL. See MEMORY bugfix-triage-mode.
-
Drive the full pipeline live via session_new (through kitsoki-mcp-driver):
harness: live, an explicit profile: (the profile, not the story agent-def,
picks the maker model — see Gotchas),- an explicit
trace: path — otherwise the session writes its trace to a
random temp file and the cost/token evidence is unrecoverable (the filed P1
missing-trace bug; gotcha #12),
base / base_branch = the baseline SHA so the pipeline cuts its worktree
from the buggy parent, not main where the bug is already gone (gotcha #11),- a scoped
test_cmd restricted to the changed-area packages — a repo with
pre-existing unrelated reds would bounce every fix forever (gotcha #13),
- a hermetic per-case worktree — cells/cases must NEVER share a worktree;
sharing IS bug #9 (gotcha #1).
Then drive the rooms (
start → reproducing → … → testing → tail) and answer
judge checkpoints as an honest operator would.
-
Independently VERIFY the deliverable — never trust the maker's self-report.
Run the oracle / targeted tests yourself on the produced worktree or merged
commit. The maker "submitted" or returned success means nothing on its own — an
impl agent that dies mid-response can still have landed (or half-landed) edits,
and a weak maker confabulates "submit missing." Gate on deliverable existence
(the files + the key edit are present) AND a clean oracle pass — not on build-green
alone and not on the agent's return. (MEMORY: workflow-gate-on-independent-verify,
workflow-dead-impl-poisons-demo, bugfix-pipeline-verify-gaps.)
-
Record the outcome. verdict, exit (shipped / needs-human / not-reproducible),
independent-verify result (PASS/FAIL and how you checked), cost (sum the trace's
per-call payload.meta.cost_usd) + tokens (primary, provider-neutral axis), wall
time, and every friction point as a finding (next section).
The regression gate parks un-instrumented tickets at needs-human (a ticket
with no seeded repro_command → regression_red_pre_fix=false → never auto-ships).
That is correct RED→GREEN discipline, not a failure: the maker still produces
the fix; a human verifies + merges. Expect it for a marathon over tickets that lack
repro commands; don't "fix" it.
Driving mechanics & gotchas (condensed)
| # | Gotcha | Why |
|---|
explicit trace: | pass a real trace path to session_new | else trace lands in $TMPDIR random file → cost evidence lost (P1) |
base=baseline SHA | cut the worktree from <fix>^ / the buggy SHA | main has the bug fixed → nothing reproduces |
scoped test_cmd | restrict CI to changed-area packages | pre-existing unrelated reds bounce every fix forever; your oracle is authoritative |
| per-case worktree | one fresh worktree per case, never shared | shared worktree IS bug #9 |
| maker model = profile | set the model via session profile:, not story model: | profile supersedes the agent-def (claude-native→opus, claude-sonnet→sonnet, synthetic-claude→GLM, codex-native→gpt) |
| needs-human park | un-instrumented ticket parks at needs-human | expected RED→GREEN discipline; human verifies+merges |
| CLEAN pre-flight | before each case git worktree remove --force AND git branch -D fix/<id> | removing only the worktree dir leaves the branch; workspace.create silently REATTACHES it → the run inherits a prior run's commits (bug already "fixed" → not-reproducible). Delete BOTH. |
| heartbeat watchdog | watch the trace mtime; kill on stall, NOT on wall-clock | a live run can legitimately take hours; a STUCK one stops growing the trace. See below. |
Background runs: clean pre-flight + a heartbeat watchdog
A long live dogfood is driven by a background agent and can run for hours —
that's fine as long as it's making external progress. The failure to guard
against is the agent getting truly stuck (a blocked studio call, a hung
maker, an owner-marker collision) and waiting forever with no signal.
Improve WITHOUT overfitting (the core discipline)
When the pipeline fails a case, fix the story / prompts generically — a change
is legitimate only if it helps the general class, not just this case. The
worked example: the blind-implementer failure on one bug drove a generic hardening
(self-verifying implementer + RED-now reproducer + honest test_author + smallest-fix
proposer) — commit d210ea67 — phrased so it helps any bug, naming none.
stories/AGENTS.md is load-bearing here: never paper-over, hack, or work around
runtime issues — the stories exist to expose them through real use and force a
real fix. A set:-block patch is fine (atomic vs a frozen pre-block snapshot —
MEMORY set-block-atomic-determinism); a case-specific special-case is not.
"Is this change overfitting?" checklist — a fix must pass ALL of these:
If a change fails any box, it's overfit — generalise it or drop it.
Capturing findings → tickets
Each friction point becomes a concrete finding; the consequential ones become
filed tickets via the studio issue_create tool (through kitsoki-mcp-driver), in
the issues/bugs/ frontmatter shape (id, title, target: story|kitsoki,
severity, component, trace_ref, Body / Expected / Actual / Impact). Worked
examples from the 2026-06 bugfix marathon:
- F1 (fixed): blind implementer (told not to run tests) shipped a 65-test-breaking
fix → hardened generically in
d210ea67.
- F2:
needs-human parking surfaces the regression-gate technicality instead of
the louder "your fix breaks N tests" — under-reports severity.
- F3: a
needs-human park leaves the worktree/branch uncleaned with no surfaced
path / resume / cleanup hint.
- P1 missing-trace:
session_new writes the trace to os.CreateTemp instead of
store.DefaultTracePath → filed
issues/bugs/2026-06-24T090000Z-mcp-live-sessions-no-discoverable-trace.md.
A finding that points at a runtime/story bug → file the ticket AND, where the class
warrants it, add the no-LLM flow regression that pins the fix.
Honesty controls (non-negotiable)
- Independent verify, oracle-gated. Run the oracle/tests yourself; the deliverable
is graded by your oracle, not the pipeline's internal CI status and not the agent's
return string.
- Don't trust agent returns. A
failed: or dead agent can still have landed edits;
re-verify the actual commit/worktree after any such agent. Abort before any
demo/report step if a deliverable is null (no lookalike substitution).
- Cost on one consistent basis. Sum the trace's native
payload.meta.cost_usd;
report TOKENS as the primary provider-neutral axis, USD second.
- No real LLM in automated tests. The live marathon drive is a manual, gated action.
Every regression you add is cassette-stubbed (
stories/AGENTS.md, root AGENTS.md).
Output: the marathon summary
Produce a summary table + narrative covering, per case: case · triage verdict ·
outcome/exit · independent-verify result · cost (tokens primary, USD) · wall time,
then roll up: fixes shipped · findings (with ticket links) · prompt/story hardenings
(generic only) and the honest headline (structure isn't automatically cheaper, but
it's more thorough — regression test, safe gate-parking, refine loop — and catches
bad fixes a naive single prompt would ship).
This summary is the same payload the dogfood-marathon kitsoki story (built in
parallel under stories/dogfood-marathon/) wraps into a drivable workflow that emits
a slidey report of outcomes / effectiveness / time / cost / what-fixed /
what-worked / what-didn't. When that story exists, prefer driving it (it journals the
per-case data for the deck); use this skill's by-hand loop to bootstrap it or for
one-off marathons.
Runbook (crisp)
- Assemble the backlog + pin each case's baseline; confirm the bug is present at baseline.
- (Optional) triage-only pass (
bugfix_mode=triage); drop ALREADY-FIXED.
- Per case, via kitsoki-mcp-driver:
session_new (harness:live, profile:,
explicit trace:, base=baseline, scoped test_cmd, fresh worktree) → drive rooms.
- Independently verify the deliverable yourself (oracle + deliverable-existence).
- Record outcome + cost/tokens + time + findings.
- For each failure: harden the story/prompts generically — run the overfitting
checklist; add a no-LLM flow regression; file tickets for runtime/story bugs.
- Emit the marathon summary (and feed it to the
dogfood-marathon story's slidey deck).
Maintenance
Codex discovers this skill directly. After adding/moving it, re-link into Claude
Code's .claude/skills/:
make setup
