// Methodology for running multi-profile LLM bake-offs via megaplan and presenting fair, blind-assessed comparisons. Cost/quality discipline, prompt hygiene, pre-merge gates, and reporting patterns. Use when the user says "bakeoff", "bake off", "megaplan bakeoff", or asks to compare profile mixes head-to-head.
Pick the right megaplan profile, thinking-strength tier, and robustness level for the work in front of you — for both Codex and Claude harnesses. Consult before invoking megaplan.
Three-round adversarial critique of epic drafts (high / mid / low abstraction) with revision after each round. Produces a chain-ready revised epic.
Observe an in-flight megaplan — introspect state, trace events, diagnose blockages, detect drift. Companion to megaplan-decision. Use during and after a run, not before.
AI agent harness for coordinating Claude and GPT to make and execute extremely robust plans.
Run megaplan plans and chains inside a provider-managed container (today, Railway) with a persistent workspace volume. Use when the run needs to outlast a local terminal session, span multiple repos, or share a long-lived dev box across concurrent chains. Covers `cloud.yaml` fields, `extra_repos[]` + `chain_session` multi-tenancy, the operator loop, and the gotchas that wedge fresh runs.
Run an epic — a chain of sprint-sized megaplans driven sequentially via `megaplan chain`. Use when the work is bigger than ~2 weeks and needs to be decomposed into multiple plans with state, ordering, and failure semantics handled by the harness.
| name | megaplan-bakeoff |
| description | Methodology for running multi-profile LLM bake-offs via megaplan and presenting fair, blind-assessed comparisons. Cost/quality discipline, prompt hygiene, pre-merge gates, and reporting patterns. Use when the user says "bakeoff", "bake off", "megaplan bakeoff", or asks to compare profile mixes head-to-head. |
A bake-off runs the same task through N profiles concurrently and picks a winner. Use it to test profile mixes (which model leads, who critiques, who executes), to evaluate cost-quality trade-offs, or to compare codebases head-to-head. This skill captures the operational knowledge that separates "well-run bake-off" from "expensive shrug."
Don't bake off for: tiny tasks (overhead > value), tasks with no measurable deliverable, or one-shot questions.
Built-in profiles (megaplan): standard, all-claude, kimi, claude-kimi-deepseek. Project-local in <project>/.megaplan/profiles.toml.
Useful mix dimensions:
Test profiles 2×2: lead × critic. all-claude is the control.
Always smoke-test profiles with a tiny --mode doc task before launching code-mode bake-offs. Reasons:
max_tokens > 4096 requires stream=true) — fixed in current megaplan but always worth verifying.Smoke prompt template: a 250-word note on a topic the model knows. Cost: ~$0.15-1.70/arm depending on profile.
Do NOT include absolute paths in the idea text. Specifically: never write Project: /abs/path/to/repo or any equivalent. Megaplan injects its own Project directory: <worktree> line. If your idea text has a competing path, models will trust the user-authored line and execute against the wrong directory.
Symptom when this happens: bakeoff arm reports final_state: done but git -C <worktree> diff main --stat is empty. Files appear in your main repo instead, polluting it. Megaplan's audit catches this (files_in_diff: [] while files_claimed is non-empty) but light-robustness review auto-approves it anyway.
Defense in depth:
a0ed9f51 (sandbox fix) enforces this at the tool layer — any cd /escape && ... is refused.PYENV_VERSION=3.11.11 python -m megaplan bakeoff run \
--idea-file /tmp/<task>-idea.txt \
--profiles all-claude deepseek-kimi-deepseek claude-kimi-deepseek-or all-deepseek-or deepseek-claude-critique \
--robustness light \
--allow-dirty \
--exp-id <task>-<date> \
--detach
--allow-dirty lets you start from a worktree with unrelated uncommitted changes (they stay on main, don't enter worktrees). --detach returns immediately; poll with bakeoff status --exp <id>.
Light robustness loop: plan → critique → revise → finalize → execute → review. No prep, no gate.
bakeoff status table can lag — last write to state.json is the truth. Check <worktree>/.megaplan/plans/<plan>/state.json's current_state field if status seems stuck.Before bakeoff pick, verify each arm:
for p in <profile-list>; do
d=/Users/.../.megaplan-worktrees/<exp-id>/$p
echo "$p: $(git -C $d diff main --stat 2>/dev/null | tail -1)"
done
An arm with empty diff after final_state: done is misdirected (or fabricated, pre-sandbox). Exclude it from comparison. Stash any pollution in main:
git status --short # if dirty
git stash push -u -m "<exp-id>-pollution"
Always blind. Sub-agent must not know which profile produced which output. Pattern:
.megaplan/ files inside worktrees (those reveal the profile via plan_v2.md or state.json).Sub-agent returns ranking + winner pick + tiebreaker reasoning. Un-blind on receipt and present to user.
Standard table format:
| Profile | Cost | Time | Score | Verdict |
|---|
Add a "cost-adjusted take" paragraph that calls out:
Quote 1-2 sentences from each output that capture its voice — scores miss style.
End with a "ship this one" pick and the dimension that tipped it.
PYENV_VERSION=3.11.11 python -m megaplan bakeoff compare --exp <id> # required before pick
PYENV_VERSION=3.11.11 python -m megaplan bakeoff pick --exp <id> --profile <winner> --rationale "..."
PYENV_VERSION=3.11.11 python -m megaplan bakeoff merge --exp <id>
Merge brings the winner's worktree changes into main as uncommitted edits — review, run pytest, then commit yourself. Don't auto-commit blindly; the merge step doesn't run tests.
Rough order-of-magnitude per arm at light robustness:
Multi-arm bake-off totals = sum of arms. A 5-arm code-mode phase can run $10-40. A 4-phase pipeline of 5-arm bake-offs can run $50-200.
ps aux | grep megaplan and lsof -p <pid> | grep TCP — if no socket activity, kill and re-launch single-profile via megaplan auto.<plan>/critique_v1_raw.txt for HTTP errors. Fireworks 400 streaming bug pattern: "max_tokens > 4096 must have stream=true". Megaplan ≥ c2bbc729 fixes this.bakeoff status table: check actual outcome.json files in ~/.megaplan/bakeoffs/<exp>/<profile>/ — those are authoritative when the run completes.--phase-model execute=claude or use a profile that doesn't depend on codex.final_state: done (model misdirected; sandbox may not be installed).{"checks": [], "flags": []} and state stuck at planned (silent worker failure).megaplan bakeoff abandon --exp <id> discards worktrees but keeps audit data for forensics.
For sequential phases (e.g. orchestrator V1 phases 6→9), bake-off each phase, merge winner, then bake off the next from the new main. Don't bake off all phases as one giant idea — losses compound and you can't isolate which phase regressed.
After all phases, deploy a final-assessment sub-agent to write a structured .md covering: profile mix performance across phases, cost trends, quality trends, harness bugs surfaced and fixed. Pin it in docs/.