| name | launch-prep |
| description | Prepare a benchflow release for launch. Use when the user says "prepare for launch", "release prep", "cut a release", "bump version", or "launch checklist". Runs docs/labs/code review, CI gate, e2e smoke test, CHANGELOG update, version bump, and opens a PR to main. |
| user-invocable | true |
BenchFlow Launch Prep
Prepares a release: docs/labs/code review → CI gate → e2e smoke → CHANGELOG → version bump → PR.
Arguments passed: $ARGUMENTS
Dispatch
No args — show release state: current version (grep '^version' pyproject.toml), [Unreleased] section of CHANGELOG.md, branch and dirty files (git status --short). Recommend next step.
check — run CI gate, report pass/fail, no edits:
.venv/bin/ruff format --check src tests && .venv/bin/ruff check src tests
.venv/bin/ty check
.venv/bin/python -m pytest tests/ -q
uv lock --check
patch / minor / major — follow Steps 0–6 below.
Step 0 — Pre-flight
- Version:
grep '^version' pyproject.toml, compute new version, ask user to confirm before proceeding. Use their value if they override.
- Branch:
git branch --show-current — if main, stop: "Switch to a feature branch or: git checkout -b release/v<NEW_VERSION>". Any non-main branch is fine.
Step 1 — Docs, labs, code, and test review
Delegate to purpose-built skills in report-only mode — each skill
normally waits for per-finding approval, but launch-prep just triages
their output into Blockers/Polish/Clean. Tell each skill up front: "no
edits, no approval loop; return the punch list."
Run A → B → C → D serially (each skill spawns its own subagents; stacking
them saturates the pool).
A — /docs-review (full). Covers README.md, docs/*.md, AGENTS.md,
and the light-touch .claude/dev-docs/ set. Captures drift vs. code, stale refs,
link integrity, registry alignment. Supersedes the old ad-hoc docs pass.
B — labs/ (ad-hoc subagent) — /docs-review skips labs. Spawn one
Explore agent: for each experiment, README matches code, no broken
links, no stale src/benchflow API calls, no notebook tracebacks. Grep
for TODO, FIXME.
C — /code-cleanup --recent (report-only). Covers the old src/
TODO/FIXME/HACK/debug-symbol grep plus dead code, over-defensive checks,
latent bugs. --recent scopes to files changed since the last tag so the
two-pass verify stays fast. Also grep explicitly for hardcoded secrets
(api_key\s*=\s*["'], sk-[A-Za-z0-9]{20,}) — outside code-cleanup's
category list.
D — /test-review --recent (report-only). Catches mock-echo bloat
and coverage gaps in tests whose source changed this cycle. Findings
route to Polish unless a gap covers a shipped bug fix (→ Blocker).
Synthesize all four into one list: Blockers / Polish / Clean. Stop on
blockers.
Step 2 — CI gate
Mirrors .github/workflows/test.yml exactly. Stop on any failure.
.venv/bin/ruff format src tests
.venv/bin/ruff check src tests
.venv/bin/ty check
.venv/bin/python -m pytest tests/ -q
uv lock --check
If ruff format changed files: git diff --name-only, then git add <those files only> — not git add ..
Step 3 — e2e smoke test
source .env 2>/dev/null || true
.venv/bin/python -m pytest -m live tests/test_smoke.py -v -ra \
--junitxml=/tmp/smoke.xml
.venv/bin/python -c 'import sys,xml.etree.ElementTree as ET; \
es=list(ET.parse("/tmp/smoke.xml").getroot().iter("testsuite")); \
t=sum(int(e.get("tests",0)) for e in es); \
s=sum(int(e.get("skipped",0)) for e in es); \
sys.exit(0) if t and not s else (print("LIVE SMOKE SKIPPED OR NOT RUN — gate is RED, not green") or sys.exit(1))'
The live smoke skipifs when Docker is down or the chosen model has no
credential, and pytest exits 0 on a skip. The JUnit check above turns that
silent pass into a hard failure so a skipped smoke cannot false-green the gate.
Expected: test_hello_world_smoke passes with reward > 0 and non-empty
trajectory.
If the only credential on the machine is non-Anthropic, point the smoke at an
agent/model it can authenticate instead of skipping (proven combo:
openhands + deepseek):
export BENCHFLOW_SMOKE_AGENT=openhands
export BENCHFLOW_SMOKE_MODEL=deepseek/deepseek-chat
export DEEPSEEK_API_KEY=... DEEPSEEK_BASE_URL=https://api.deepseek.com
BENCHFLOW_SMOKE_AGENT and BENCHFLOW_SMOKE_MODEL must be set together; the
skip reason names the exact missing credential for whichever model is selected.
Step 4 — CHANGELOG draft
If [Unreleased] is non-empty: show it, ask to confirm.
If empty: generate from git log —
git log $(git describe --tags --abbrev=0)..HEAD --oneline --no-merges
Map prefixes: feat: → Added, fix: → Fixed, chore:/refactor:/build: → Changed. Omit docs: and test: unless notable. Show draft and wait for user approval before writing anything.
Step 5 — Write CHANGELOG, pyproject.toml, and uv.lock
CHANGELOG: Insert approved entries as ## <NEW_VERSION> — $(date +%Y-%m-%d) immediately after ## [Unreleased], leaving [Unreleased] empty. Use Edit, not Write.
pyproject.toml + uv.lock: Edit only version = "..." under [project] (line 3). Do not touch target-version (ruff) or python-version (ty) — those are Python version pins. Then run uv lock so the editable benchflow package entry in uv.lock matches the new version. Do not edit __init__.py; it reads version from importlib.metadata automatically.
Step 6 — Commit and PR
git add CHANGELOG.md pyproject.toml uv.lock
git commit -m "chore: release v<NEW_VERSION>"
git push -u origin HEAD
gh pr create --title "chore: release v<NEW_VERSION>" --body "$(cat <<'EOF'
## Release v<NEW_VERSION>
See CHANGELOG.md for details.
## Checklist
- [ ] CI gate passes
- [ ] e2e smoke test passes
- [ ] CHANGELOG updated
- [ ] Version bumped in pyproject.toml
- [ ] uv.lock refreshed
- [ ] Merge and tag after review
🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
Return the PR URL.
Step 7 — Post-merge (remind user, do not run)
git checkout main && git pull
git tag v<NEW_VERSION> && git push origin v<NEW_VERSION>
awk '/^## <NEW_VERSION> /{flag=1; next} /^## [0-9]/{flag=0} flag' CHANGELOG.md > /tmp/release_notes.md
gh release create v<NEW_VERSION> --title "v<NEW_VERSION>" --notes-file /tmp/release_notes.md
uv build && uv publish
