| name | ship |
| description | Ship workflow: detect + merge base branch, run tests, review diff, bump VERSION, update CHANGELOG, commit, push, create PR.
|
| allowed-tools | ["Bash","Read","Write","Edit","Grep","Glob","Agent","AskUserQuestion","WebSearch"] |
| triggers | ["ship it","create a pr","push to main","deploy this"] |
When to invoke
Use when asked to "ship", "deploy", "push to main", "create a PR", "merge and push", or "get it deployed". Proactively invoke this skill (do NOT push/PR directly) when the user says code is ready, asks about deploying, wants to push code up, or asks to create a PR.
Preamble
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
_LEARN_FILE="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
_LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
echo "LEARNINGS: $_LEARN_COUNT entries loaded"
if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
~/.vibestack/bin/vibe-learnings-search --limit 5 2>/dev/null || true
fi
else
echo "LEARNINGS: none yet"
fi
{{include lib/snippets/session-host.md}}
{{include lib/snippets/decision-brief.md}}
{{include lib/snippets/working-protocols.md}}
{{include lib/snippets/state-protocols.md}}
Step 0: Detect platform and base branch
First, detect the git hosting platform from the remote URL:
git remote get-url origin 2>/dev/null
- If the URL contains "github.com" → platform is GitHub
- If the URL contains "gitlab" → platform is GitLab
- Otherwise, check CLI availability:
gh auth status 2>/dev/null succeeds → platform is GitHub (covers GitHub Enterprise)
glab auth status 2>/dev/null succeeds → platform is GitLab (covers self-hosted)
- Neither → unknown (use git-native commands only)
Determine which branch this PR/MR targets, or the repo's default branch if no
PR/MR exists. Use the result as "the base branch" in all subsequent steps.
If GitHub:
gh pr view --json baseRefName -q .baseRefName — if succeeds, use it
gh repo view --json defaultBranchRef -q .defaultBranchRef.name — if succeeds, use it
If GitLab:
glab mr view -F json 2>/dev/null and extract the target_branch field — if succeeds, use it
glab repo view -F json 2>/dev/null and extract the default_branch field — if succeeds, use it
Git-native fallback (if unknown platform, or CLI commands fail):
git symbolic-ref refs/remotes/origin/HEAD 2>/dev/null | sed 's|refs/remotes/origin/||'
- If that fails:
git rev-parse --verify origin/main 2>/dev/null → use main
- If that fails:
git rev-parse --verify origin/master 2>/dev/null → use master
If all fail, fall back to main.
Print the detected base branch name. In every subsequent git diff, git log,
git fetch, git merge, and PR/MR creation command, substitute the detected
branch name wherever the instructions say "the base branch" or <default>.
Ship: Fully Automated Ship Workflow
You are running the /ship workflow. This is a non-interactive, fully automated workflow. Do NOT ask for confirmation at any step. The user said /ship which means DO IT. Run straight through and output the PR URL at the end.
Only stop for:
- On the base branch (abort)
- Merge conflicts that can't be auto-resolved (stop, show conflicts)
- In-branch test failures (pre-existing failures are triaged, not auto-blocking)
- Pre-landing review finds ASK items that need user judgment
- MINOR or MAJOR version bump needed (ask — see Step 12)
- Greptile review comments that need user decision (complex fixes, false positives)
- AI-assessed coverage below minimum threshold (hard gate with user override — see Step 7)
- Plan items NOT DONE with no user override (see Step 8)
- Plan verification failures (see Step 8.1)
- TODOS.md missing and user wants to create one (ask — see Step 14)
- TODOS.md disorganized and user wants to reorganize (ask — see Step 14)
Never stop for:
- Uncommitted changes (always include them)
- Version bump choice (auto-pick MICRO or PATCH — see Step 12)
- CHANGELOG content (auto-generate from diff)
- Commit message approval (auto-commit)
- Multi-file changesets (auto-split into bisectable commits)
- TODOS.md completed-item detection (auto-mark)
- Auto-fixable review findings (dead code, N+1, stale comments — fixed automatically)
- Test coverage gaps within target threshold (auto-generate and commit, or flag in PR body)
Re-run behavior (idempotency):
Re-running /ship means "run the whole checklist again." Every verification step
(tests, coverage audit, plan completion, pre-landing review, adversarial review,
VERSION/CHANGELOG check, TODOS, document-release) runs on every invocation.
Only actions are idempotent:
- Step 12: If VERSION already bumped, skip the bump but still read the version
- Step 17: If already pushed, skip the push command
- Step 19: If PR exists, update the body instead of creating a new PR
Never skip a verification step because a prior
/ship run already performed it.
Step 1: Pre-flight
-
Check the current branch. If on the base branch or the repo's default branch, abort: "You're on the base branch. Ship from a feature branch."
-
Run git status (never use -uall). Uncommitted changes are always included — no need to ask.
-
Run git diff <base>...HEAD --stat and git log <base>..HEAD --oneline to understand what's being shipped.
-
Check review readiness:
Review Readiness Dashboard
After completing the review, read the review log and config to display the dashboard.
~/.vibestack/bin/vibe-review-read --json 2>/dev/null
Parse the output. Find the most recent entry for each skill (plan-ceo-review, plan-eng-review, review, plan-design-review, design-review-lite, adversarial-review, codex-review, codex-plan-review). Ignore entries with timestamps older than 7 days. For the Eng Review row, show whichever is more recent between review (diff-scoped pre-landing review) and plan-eng-review (plan-stage architecture review). Append "(DIFF)" or "(PLAN)" to the status to distinguish. For the Adversarial row, show whichever is more recent between adversarial-review (new auto-scaled) and codex-review (legacy). For Design Review, show whichever is more recent between plan-design-review (full visual audit) and design-review-lite (code-level check). Append "(FULL)" or "(LITE)" to the status to distinguish. For the Outside Voice row, show the most recent codex-plan-review entry — this captures outside voices from both /plan-ceo-review and /plan-eng-review.
Source attribution: If the most recent entry for a skill has a `"via"` field, append it to the status label in parentheses. Examples: plan-eng-review with via:"autoplan" shows as "CLEAR (PLAN via /autoplan)". review with via:"ship" shows as "CLEAR (DIFF via /ship)". Entries without a via field show as "CLEAR (PLAN)" or "CLEAR (DIFF)" as before.
Note: autoplan-voices and design-outside-voices entries are audit-trail-only (forensic data for cross-model consensus analysis). They do not appear in the dashboard and are not checked by any consumer.
Display:
+====================================================================+
| REVIEW READINESS DASHBOARD |
+====================================================================+
| Review | Runs | Last Run | Status | Required |
|-----------------|------|---------------------|-----------|----------|
| Eng Review | 1 | 2026-03-16 15:00 | CLEAR | YES |
| CEO Review | 0 | — | — | no |
| Design Review | 0 | — | — | no |
| Adversarial | 0 | — | — | no |
| Outside Voice | 0 | — | — | no |
+--------------------------------------------------------------------+
| VERDICT: CLEARED — Eng Review passed |
+====================================================================+
Review tiers:
- Eng Review (required by default): The only review that gates shipping. Covers architecture, code quality, tests, performance. Can be disabled globally with `vibe-config set skip_eng_review true` (the "don't bother me" setting).
- CEO Review (optional): Use your judgment. Recommend it for big product/business changes, new user-facing features, or scope decisions. Skip for bug fixes, refactors, infra, and cleanup.
- Design Review (optional): Use your judgment. Recommend it for UI/UX changes. Skip for backend-only, infra, or prompt-only changes.
- Adversarial Review (automatic): Always-on for every review. Every diff gets both Claude adversarial subagent and Codex adversarial challenge. Large diffs (200+ lines) additionally get Codex structured review with P1 gate. No configuration needed.
- Outside Voice (optional): Independent plan review from a different AI model. Offered after all review sections complete in /plan-ceo-review and /plan-eng-review. Falls back to Claude subagent if Codex is unavailable. Never gates shipping.
Verdict logic:
- CLEARED: Eng Review has >= 1 entry within 7 days from either `review` or `plan-eng-review` with status "clean" (or `skip_eng_review` is `true`)
- NOT CLEARED: Eng Review missing, stale (>7 days), or has open issues
- CEO, Design, and Codex reviews are shown for context but never block shipping
- If `skip_eng_review` config is `true`, Eng Review shows "SKIPPED (global)" and verdict is CLEARED
Staleness detection: After displaying the dashboard, check if any existing reviews may be stale:
- Parse the `---HEAD---` section from the bash output to get the current HEAD commit hash
- For each review entry that has a `commit` field: compare it against the current HEAD. If different, count elapsed commits: `git rev-list --count STORED_COMMIT..HEAD`. Display: "Note: {skill} review from {date} may be stale — {N} commits since review"
- For entries without a `commit` field (legacy entries): display "Note: {skill} review from {date} has no commit tracking — consider re-running for accurate staleness detection"
- If all reviews match the current HEAD, do not display any staleness notes
If the Eng Review is NOT "CLEAR":
Print: "No prior eng review found — ship will run its own pre-landing review in Step 9."
Check diff size: git diff <base>...HEAD --stat | tail -1. If the diff is >200 lines, add: "Note: This is a large diff. Consider running /plan-eng-review or /autoplan for architecture-level review before shipping."
If CEO Review is missing, mention as informational ("CEO Review not run — recommended for product changes") but do NOT block.
For Design Review: run eval "$(~/.vibestack/bin/vibe-diff-scope <base> 2>/dev/null || true)". If SCOPE_FRONTEND=true and no design review (plan-design-review or design-review-lite) exists in the dashboard, mention: "Design Review not run — this PR changes frontend code. The lite design check will run automatically in Step 9, but consider running /design-review for a full visual audit post-implementation." Still never block.
Continue to Step 2 — do NOT block or ask. Ship runs its own review in Step 9.
Step 2: Distribution Pipeline Check
If the diff introduces a new standalone artifact (CLI binary, library package, tool) — not a web
service with existing deployment — verify that a distribution pipeline exists.
-
Check if the diff adds a new cmd/ directory, main.go, or bin/ entry point:
git diff $(git merge-base origin/<base> HEAD) --name-only | grep -E '(cmd/.*/main\.go|bin/|Cargo\.toml|setup\.py|package\.json)' | head -5
-
If new artifact detected, check for a release workflow:
ls .github/workflows/ 2>/dev/null | grep -iE 'release|publish|dist'
grep -qE 'release|publish|deploy' .gitlab-ci.yml 2>/dev/null && echo "GITLAB_CI_RELEASE"
-
If no release pipeline exists and a new artifact was added: Use AskUserQuestion:
- "This PR adds a new binary/tool but there's no CI/CD pipeline to build and publish it.
Users won't be able to download the artifact after merge."
- A) Add a release workflow now (CI/CD release pipeline — GitHub Actions or GitLab CI depending on platform)
- B) Defer — add to TODOS.md
- C) Not needed — this is internal/web-only, existing deployment covers it
-
If release pipeline exists: Continue silently.
-
If no new artifact detected: Skip silently.
Step 3: Merge the base branch (BEFORE tests)
Fetch and merge the base branch into the feature branch so tests run against the merged state:
git fetch origin <base> && git merge origin/<base> --no-edit
If there are merge conflicts: Try to auto-resolve if they are simple (VERSION, schema.rb, CHANGELOG ordering). If conflicts are complex or ambiguous, STOP and show them.
If already up to date: Continue silently.
Step 4: Test Framework Bootstrap
Test Framework Bootstrap
Detect existing test framework and project runtime:
setopt +o nomatch 2>/dev/null || true
[ -f Gemfile ] && echo "RUNTIME:ruby"
[ -f package.json ] && echo "RUNTIME:node"
[ -f requirements.txt ] || [ -f pyproject.toml ] && echo "RUNTIME:python"
[ -f go.mod ] && echo "RUNTIME:go"
[ -f Cargo.toml ] && echo "RUNTIME:rust"
[ -f composer.json ] && echo "RUNTIME:php"
[ -f mix.exs ] && echo "RUNTIME:elixir"
[ -f Gemfile ] && grep -q "rails" Gemfile 2>/dev/null && echo "FRAMEWORK:rails"
[ -f package.json ] && grep -q '"next"' package.json 2>/dev/null && echo "FRAMEWORK:nextjs"
ls jest.config.* vitest.config.* playwright.config.* .rspec pytest.ini pyproject.toml phpunit.xml 2>/dev/null
ls -d test/ tests/ spec/ __tests__/ cypress/ e2e/ 2>/dev/null
[ -f .vibestack/no-test-bootstrap ] && echo "BOOTSTRAP_DECLINED"
If test framework detected (config files or test directories found):
Print "Test framework detected: {name} ({N} existing tests). Skipping bootstrap."
Read 2-3 existing test files to learn conventions (naming, imports, assertion style, setup patterns).
Store conventions as prose context for use in Phase 8e.5 or Step 7. Skip the rest of bootstrap.
If BOOTSTRAP_DECLINED appears: Print "Test bootstrap previously declined — skipping." Skip the rest of bootstrap.
If NO runtime detected (no config files found): Use AskUserQuestion:
"I couldn't detect your project's language. What runtime are you using?"
Options: A) Node.js/TypeScript B) Ruby/Rails C) Python D) Go E) Rust F) PHP G) Elixir H) This project doesn't need tests.
If user picks H → write .vibestack/no-test-bootstrap and continue without tests.
If runtime detected but no test framework — bootstrap:
B2. Research best practices
Use WebSearch to find current best practices for the detected runtime:
"[runtime] best test framework 2025 2026"
"[framework A] vs [framework B] comparison"
If WebSearch is unavailable, use this built-in knowledge table:
| Runtime | Primary recommendation | Alternative |
|---|
| Ruby/Rails | minitest + fixtures + capybara | rspec + factory_bot + shoulda-matchers |
| Node.js | vitest + @testing-library | jest + @testing-library |
| Next.js | vitest + @testing-library/react + playwright | jest + cypress |
| Python | pytest + pytest-cov | unittest |
| Go | stdlib testing + testify | stdlib only |
| Rust | cargo test (built-in) + mockall | — |
| PHP | phpunit + mockery | pest |
| Elixir | ExUnit (built-in) + ex_machina | — |
B3. Framework selection
Use AskUserQuestion:
"I detected this is a [Runtime/Framework] project with no test framework. I researched current best practices. Here are the options:
A) [Primary] — [rationale]. Includes: [packages]. Supports: unit, integration, smoke, e2e
B) [Alternative] — [rationale]. Includes: [packages]
C) Skip — don't set up testing right now
RECOMMENDATION: Choose A because [reason based on project context]"
If user picks C → write .vibestack/no-test-bootstrap. Tell user: "If you change your mind later, delete .vibestack/no-test-bootstrap and re-run." Continue without tests.
If multiple runtimes detected (monorepo) → ask which runtime to set up first, with option to do both sequentially.
B4. Install and configure
- Install the chosen packages (npm/bun/gem/pip/etc.)
- Create minimal config file
- Create directory structure (test/, spec/, etc.)
- Create one example test matching the project's code to verify setup works
If package installation fails → debug once. If still failing → revert with git checkout -- package.json package-lock.json (or equivalent for the runtime). Warn user and continue without tests.
B4.5. First real tests
Generate 3-5 real tests for existing code:
- Find recently changed files:
git log --since=30.days --name-only --format="" | sort | uniq -c | sort -rn | head -10
- Prioritize by risk: Error handlers > business logic with conditionals > API endpoints > pure functions
- For each file: Write one test that tests real behavior with meaningful assertions. Never
expect(x).toBeDefined() — test what the code DOES.
- Run each test. Passes → keep. Fails → fix once. Still fails → delete silently.
- Generate at least 1 test, cap at 5.
Never import secrets, API keys, or credentials in test files. Use environment variables or test fixtures.
B5. Verify
{detected test command}
If tests fail → debug once. If still failing → revert all bootstrap changes and warn user.
B5.5. CI/CD pipeline
ls -d .github/ 2>/dev/null && echo "CI:github"
ls .gitlab-ci.yml .circleci/ bitrise.yml 2>/dev/null
If .github/ exists (or no CI detected — default to GitHub Actions):
Create .github/workflows/test.yml with:
runs-on: ubuntu-latest
- Appropriate setup action for the runtime (setup-node, setup-ruby, setup-python, etc.)
- The same test command verified in B5
- Trigger: push + pull_request
If non-GitHub CI detected → skip CI generation with note: "Detected {provider} — CI pipeline generation supports GitHub Actions only. Add test step to your existing pipeline manually."
B6. Create TESTING.md
First check: If TESTING.md already exists → read it and update/append rather than overwriting. Never destroy existing content.
Write TESTING.md with:
- Philosophy: "100% test coverage is the key to great vibe coding. Tests let you move fast, trust your instincts, and ship with confidence — without them, vibe coding is just yolo coding. With tests, it's a superpower."
- Framework name and version
- How to run tests (the verified command from B5)
- Test layers: Unit tests (what, where, when), Integration tests, Smoke tests, E2E tests
- Conventions: file naming, assertion style, setup/teardown patterns
B7. Update CLAUDE.md
First check: If CLAUDE.md already has a ## Testing section → skip. Don't duplicate.
Append a ## Testing section:
- Run command and test directory
- Reference to TESTING.md
- Test expectations:
- 100% test coverage is the goal — tests make vibe coding safe
- When writing new functions, write a corresponding test
- When fixing a bug, write a regression test
- When adding error handling, write a test that triggers the error
- When adding a conditional (if/else, switch), write tests for BOTH paths
- Never commit code that makes existing tests fail
B8. Commit
git status --porcelain
Only commit if there are changes. Stage all bootstrap files (config, test directory, TESTING.md, CLAUDE.md, .github/workflows/test.yml if created):
git commit -m "chore: bootstrap test framework ({framework name})"
Step 5: Run tests (on merged code)
Do NOT run RAILS_ENV=test bin/rails db:migrate — bin/test-lane already calls
db:test:prepare internally, which loads the schema into the correct lane database.
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.
Run both test suites in parallel:
bin/test-lane 2>&1 | tee /tmp/ship_tests.txt &
npm run test 2>&1 | tee /tmp/ship_vitest.txt &
wait
After both complete, read the output files and check pass/fail.
If any test fails: Do NOT immediately stop. Apply the Test Failure Ownership Triage:
Test Failure Ownership Triage
When tests fail, do NOT immediately stop. First, determine ownership:
Step T1: Classify each failure
For each failing test:
-
Get the files changed on this branch:
git diff origin/<base>...HEAD --name-only
-
Classify the failure:
- In-branch if: the failing test file itself was modified on this branch, OR the test output references code that was changed on this branch, OR you can trace the failure to a change in the branch diff.
- Likely pre-existing if: neither the test file nor the code it tests was modified on this branch, AND the failure is unrelated to any branch change you can identify.
- When ambiguous, default to in-branch. It is safer to stop the developer than to let a broken test ship. Only classify as pre-existing when you are confident.
This classification is heuristic — use your judgment reading the diff and the test output. You do not have a programmatic dependency graph.
Step T2: Handle in-branch failures
STOP. These are your failures. Show them and do not proceed. The developer must fix their own broken tests before shipping.
Step T3: Handle pre-existing failures
Check REPO_MODE from the preamble output.
If REPO_MODE is solo:
Use AskUserQuestion:
These test failures appear pre-existing (not caused by your branch changes):
[list each failure with file:line and brief error description]
Since this is a solo repo, you're the only one who will fix these.
RECOMMENDATION: Choose A — fix now while the context is fresh. Completeness: 9/10.
A) Investigate and fix now (human: ~2-4h / CC: ~15min) — Completeness: 10/10
B) Add as P0 TODO — fix after this branch lands — Completeness: 7/10
C) Skip — I know about this, ship anyway — Completeness: 3/10
If REPO_MODE is collaborative or unknown:
Use AskUserQuestion:
These test failures appear pre-existing (not caused by your branch changes):
[list each failure with file:line and brief error description]
This is a collaborative repo — these may be someone else's responsibility.
RECOMMENDATION: Choose B — assign it to whoever broke it so the right person fixes it. Completeness: 9/10.
A) Investigate and fix now anyway — Completeness: 10/10
B) Blame + assign GitHub issue to the author — Completeness: 9/10
C) Add as P0 TODO — Completeness: 7/10
D) Skip — ship anyway — Completeness: 3/10
Step T4: Execute the chosen action
If "Investigate and fix now":
- Switch to /investigate mindset: root cause first, then minimal fix.
- Fix the pre-existing failure.
- Commit the fix separately from the branch's changes:
git commit -m "fix: pre-existing test failure in <test-file>"
- Continue with the workflow.
If "Add as P0 TODO":
- If
TODOS.md exists, add the entry following the format in review/TODOS-format.md (or .claude/skills/review/TODOS-format.md).
- If
TODOS.md does not exist, create it with the standard header and add the entry.
- Entry should include: title, the error output, which branch it was noticed on, and priority P0.
- Continue with the workflow — treat the pre-existing failure as non-blocking.
If "Blame + assign GitHub issue" (collaborative only):
If "Skip":
- Continue with the workflow.
- Note in output: "Pre-existing test failure skipped: "
After triage: If any in-branch failures remain unfixed, STOP. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, or skipped), continue to Step 6.
If all pass: Continue silently — just note the counts briefly.
Step 6: Eval Suites (conditional)
Evals are mandatory when prompt-related files change. Skip this step entirely if no prompt files are in the diff.
1. Check if the diff touches prompt-related files:
git diff $(git merge-base origin/<base> HEAD) --name-only
Match against these patterns (from CLAUDE.md):
app/services/*_prompt_builder.rb
app/services/*_generation_service.rb, *_writer_service.rb, *_designer_service.rb
app/services/*_evaluator.rb, *_scorer.rb, *_classifier_service.rb, *_analyzer.rb
app/services/concerns/*voice*.rb, *writing*.rb, *prompt*.rb, *token*.rb
app/services/chat_tools/*.rb, app/services/x_thread_tools/*.rb
config/system_prompts/*.txt
test/evals/**/* (eval infrastructure changes affect all suites)
If no matches: Print "No prompt-related files changed — skipping evals." and continue to Step 9.
2. Identify affected eval suites:
Each eval runner (test/evals/*_eval_runner.rb) declares PROMPT_SOURCE_FILES listing which source files affect it. Grep these to find which suites match the changed files:
grep -l "changed_file_basename" test/evals/*_eval_runner.rb
Map runner → test file: post_generation_eval_runner.rb → post_generation_eval_test.rb.
Special cases:
- Changes to
test/evals/judges/*.rb, test/evals/support/*.rb, or test/evals/fixtures/ affect ALL suites that use those judges/support files. Check imports in the eval test files to determine which.
- Changes to
config/system_prompts/*.txt — grep eval runners for the prompt filename to find affected suites.
- If unsure which suites are affected, run ALL suites that could plausibly be impacted. Over-testing is better than missing a regression.
3. Run affected suites at EVAL_JUDGE_TIER=full:
/ship is a pre-merge gate, so always use full tier (Sonnet structural + Opus persona judges).
EVAL_JUDGE_TIER=full EVAL_VERBOSE=1 bin/test-lane --eval test/evals/<suite>_eval_test.rb 2>&1 | tee /tmp/ship_evals.txt
If multiple suites need to run, run them sequentially (each needs a test lane). If the first suite fails, stop immediately — don't burn API cost on remaining suites.
4. Check results:
- If any eval fails: Show the failures, the cost dashboard, and STOP. Do not proceed.
- If all pass: Note pass counts and cost. Continue to Step 9.
5. Save eval output — include eval results and cost dashboard in the PR body (Step 19).
Tier reference (for context — /ship always uses full):
| Tier | When | Speed (cached) | Cost |
|---|
fast (Haiku) | Dev iteration, smoke tests | ~5s (14x faster) | ~$0.07/run |
standard (Sonnet) | Default dev, bin/test-lane --eval | ~17s (4x faster) | ~$0.37/run |
full (Opus persona) | /ship and pre-merge | ~72s (baseline) | ~$1.27/run |
Step 7: Test Coverage Audit
Dispatch this step as a subagent using the Agent tool with subagent_type: "general-purpose". The subagent runs the coverage audit in a fresh context window — the parent only sees the conclusion, not intermediate file reads. This is context-rot defense.
Subagent prompt: Pass the following instructions to the subagent, with <base> substituted with the base branch:
You are running a ship-workflow test coverage audit. Run git diff <base>...HEAD as needed. Do not commit or push — report only.
100% coverage is the goal — every untested path is a path where bugs hide and vibe coding becomes yolo coding. Evaluate what was ACTUALLY coded (from the diff), not what was planned.
Test Framework Detection
Before analyzing coverage, detect the project's test framework:
- Read CLAUDE.md — look for a
## Testing section with test command and framework name. If found, use that as the authoritative source.
- If CLAUDE.md has no testing section, auto-detect:
setopt +o nomatch 2>/dev/null || true
[ -f Gemfile ] && echo "RUNTIME:ruby"
[ -f package.json ] && echo "RUNTIME:node"
[ -f requirements.txt ] || [ -f pyproject.toml ] && echo "RUNTIME:python"
[ -f go.mod ] && echo "RUNTIME:go"
[ -f Cargo.toml ] && echo "RUNTIME:rust"
ls jest.config.* vitest.config.* playwright.config.* cypress.config.* .rspec pytest.ini phpunit.xml 2>/dev/null
ls -d test/ tests/ spec/ __tests__/ cypress/ e2e/ 2>/dev/null
- If no framework detected: falls through to the Test Framework Bootstrap step (Step 4) which handles full setup.
0. Before/after test count:
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' | grep -v node_modules | wc -l
Store this number for the PR body.
1. Trace every codepath changed using git diff origin/<base>...HEAD:
Read every changed file. For each one, trace how data flows through the code — don't just list functions, actually follow the execution:
- Read the diff. For each changed file, read the full file (not just the diff hunk) to understand context.
- Trace data flow. Starting from each entry point (route handler, exported function, event listener, component render), follow the data through every branch:
- Where does input come from? (request params, props, database, API call)
- What transforms it? (validation, mapping, computation)
- Where does it go? (database write, API response, rendered output, side effect)
- What can go wrong at each step? (null/undefined, invalid input, network failure, empty collection)
- Diagram the execution. For each changed file, draw an ASCII diagram showing:
- Every function/method that was added or modified
- Every conditional branch (if/else, switch, ternary, guard clause, early return)
- Every error path (try/catch, rescue, error boundary, fallback)
- Every call to another function (trace into it — does IT have untested branches?)
- Every edge: what happens with null input? Empty array? Invalid type?
This is the critical step — you're building a map of every line of code that can execute differently based on input. Every branch in this diagram needs a test.
2. Map user flows, interactions, and error states:
Code coverage isn't enough — you need to cover how real users interact with the changed code. For each changed feature, think through:
- User flows: What sequence of actions does a user take that touches this code? Map the full journey (e.g., "user clicks 'Pay' → form validates → API call → success/failure screen"). Each step in the journey needs a test.
- Interaction edge cases: What happens when the user does something unexpected?
- Double-click/rapid resubmit
- Navigate away mid-operation (back button, close tab, click another link)
- Submit with stale data (page sat open for 30 minutes, session expired)
- Slow connection (API takes 10 seconds — what does the user see?)
- Concurrent actions (two tabs, same form)
- Error states the user can see: For every error the code handles, what does the user actually experience?
- Is there a clear error message or a silent failure?
- Can the user recover (retry, go back, fix input) or are they stuck?
- What happens with no network? With a 500 from the API? With invalid data from the server?
- Empty/zero/boundary states: What does the UI show with zero results? With 10,000 results? With a single character input? With maximum-length input?
Add these to your diagram alongside the code branches. A user flow with no test is just as much a gap as an untested if/else.
3. Check each branch against existing tests:
Go through your diagram branch by branch — both code paths AND user flows. For each one, search for a test that exercises it:
- Function
processPayment() → look for billing.test.ts, billing.spec.ts, test/billing_test.rb
- An if/else → look for tests covering BOTH the true AND false path
- An error handler → look for a test that triggers that specific error condition
- A call to
helperFn() that has its own branches → those branches need tests too
- A user flow → look for an integration or E2E test that walks through the journey
- An interaction edge case → look for a test that simulates the unexpected action
Quality scoring rubric:
- ★★★ Tests behavior with edge cases AND error paths
- ★★ Tests correct behavior, happy path only
- ★ Smoke test / existence check / trivial assertion (e.g., "it renders", "it doesn't throw")
E2E Test Decision Matrix
When checking each branch, also determine whether a unit test or E2E/integration test is the right tool:
RECOMMEND E2E (mark as [→E2E] in the diagram):
- Common user flow spanning 3+ components/services (e.g., signup → verify email → first login)
- Integration point where mocking hides real failures (e.g., API → queue → worker → DB)
- Auth/payment/data-destruction flows — too important to trust unit tests alone
RECOMMEND EVAL (mark as [→EVAL] in the diagram):
- Critical LLM call that needs a quality eval (e.g., prompt change → test output still meets quality bar)
- Changes to prompt templates, system instructions, or tool definitions
STICK WITH UNIT TESTS:
- Pure function with clear inputs/outputs
- Internal helper with no side effects
- Edge case of a single function (null input, empty array)
- Obscure/rare flow that isn't customer-facing
REGRESSION RULE (mandatory)
IRON RULE: When the coverage audit identifies a REGRESSION — code that previously worked but the diff broke — a regression test is written immediately. No AskUserQuestion. No skipping. Regressions are the highest-priority test because they prove something broke.
A regression is when:
- The diff modifies existing behavior (not new code)
- The existing test suite (if any) doesn't cover the changed path
- The change introduces a new failure mode for existing callers
When uncertain whether a change is a regression, err on the side of writing the test.
Format: commit as test: regression test for {what broke}
4. Output ASCII coverage diagram:
Include BOTH code paths and user flows in the same diagram. Mark E2E-worthy and eval-worthy paths:
CODE PATHS USER FLOWS
[+] src/services/billing.ts [+] Payment checkout
├── processPayment() ├── [★★★ TESTED] Complete purchase — checkout.e2e.ts:15
│ ├── [★★★ TESTED] happy + declined + timeout ├── [GAP] [→E2E] Double-click submit
│ ├── [GAP] Network timeout └── [GAP] Navigate away mid-payment
│ └── [GAP] Invalid currency
└── refundPayment() [+] Error states
├── [★★ TESTED] Full refund — :89 ├── [★★ TESTED] Card declined message
└── [★ TESTED] Partial (non-throw only) — :101 └── [GAP] Network timeout UX
LLM integration: [GAP] [→EVAL] Prompt template change — needs eval test
COVERAGE: 5/13 paths tested (38%) | Code paths: 3/5 (60%) | User flows: 2/8 (25%)
QUALITY: ★★★:2 ★★:2 ★:1 | GAPS: 8 (2 E2E, 1 eval)
Legend: ★★★ behavior + edge + error | ★★ happy path | ★ smoke check
[→E2E] = needs integration test | [→EVAL] = needs LLM eval
Fast path: All paths covered → "Step 7: All new code paths have test coverage ✓" Continue.
5. Generate tests for uncovered paths:
If test framework detected (or bootstrapped in Step 4):
- Prioritize error handlers and edge cases first (happy paths are more likely already tested)
- Read 2-3 existing test files to match conventions exactly
- Generate unit tests. Mock all external dependencies (DB, API, Redis).
- For paths marked [→E2E]: generate integration/E2E tests using the project's E2E framework (Playwright, Cypress, Capybara, etc.)
- For paths marked [→EVAL]: generate eval tests using the project's eval framework, or flag for manual eval if none exists
- Write tests that exercise the specific uncovered path with real assertions
- Run each test. Passes → commit as
test: coverage for {feature}
- Fails → fix once. Still fails → revert, note gap in diagram.
Caps: 30 code paths max, 20 tests generated max (code + user flow combined), 2-min per-test exploration cap.
If no test framework AND user declined bootstrap → diagram only, no generation. Note: "Test generation skipped — no test framework configured."
Diff is test-only changes: Skip Step 7 entirely: "No new application code paths to audit."
6. After-count and coverage summary:
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' | grep -v node_modules | wc -l
For PR body: Tests: {before} → {after} (+{delta} new)
Coverage line: Test Coverage Audit: N new code paths. M covered (X%). K tests generated, J committed.
7. Coverage gate:
Before proceeding, check CLAUDE.md for a ## Test Coverage section with Minimum: and Target: fields. If found, use those percentages. Otherwise use defaults: Minimum = 60%, Target = 80%.
Using the coverage percentage from the diagram in substep 4 (the COVERAGE: X/Y (Z%) line):
-
>= target: Pass. "Coverage gate: PASS ({X}%)." Continue.
-
>= minimum, < target: Use AskUserQuestion:
- "AI-assessed coverage is {X}%. {N} code paths are untested. Target is {target}%."
- RECOMMENDATION: Choose A because untested code paths are where production bugs hide.
- Options:
A) Generate more tests for remaining gaps (recommended)
B) Ship anyway — I accept the coverage risk
C) These paths don't need tests — mark as intentionally uncovered
- If A: Loop back to substep 5 (generate tests) targeting the remaining gaps. After second pass, if still below target, present AskUserQuestion again with updated numbers. Maximum 2 generation passes total.
- If B: Continue. Include in PR body: "Coverage gate: {X}% — user accepted risk."
- If C: Continue. Include in PR body: "Coverage gate: {X}% — {N} paths intentionally uncovered."
-
< minimum: Use AskUserQuestion:
- "AI-assessed coverage is critically low ({X}%). {N} of {M} code paths have no tests. Minimum threshold is {minimum}%."
- RECOMMENDATION: Choose A because less than {minimum}% means more code is untested than tested.
- Options:
A) Generate tests for remaining gaps (recommended)
B) Override — ship with low coverage (I understand the risk)
- If A: Loop back to substep 5. Maximum 2 passes. If still below minimum after 2 passes, present the override choice again.
- If B: Continue. Include in PR body: "Coverage gate: OVERRIDDEN at {X}%."
Coverage percentage undetermined: If the coverage diagram doesn't produce a clear numeric percentage (ambiguous output, parse error), skip the gate with: "Coverage gate: could not determine percentage — skipping." Do not default to 0% or block.
Test-only diffs: Skip the gate (same as the existing fast-path).
100% coverage: "Coverage gate: PASS (100%)." Continue.
Test Plan Artifact
After producing the coverage diagram, write a test plan artifact so /qa and /qa-only can consume it:
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" && mkdir -p ~/.vibestack/projects/$SLUG
USER=$(whoami)
DATETIME=$(date +%Y%m%d-%H%M%S)
Write to ~/.vibestack/projects/{slug}/{user}-{branch}-ship-test-plan-{datetime}.md:
# Test Plan
Generated by /ship on {date}
Branch: {branch}
Repo: {owner/repo}
## Affected Pages/Routes
- {URL path} — {what to test and why}
## Key Interactions to Verify
- {interaction description} on {page}
## Edge Cases
- {edge case} on {page}
## Critical Paths
- {end-to-end flow that must work}
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
{"coverage_pct":N,"gaps":N,"diagram":"<full markdown coverage diagram for PR body>","tests_added":["path",...]}
Parent processing:
- Read the subagent's final output. Parse the LAST line as JSON.
- Store
coverage_pct (for Step 20 metrics), gaps (user summary), tests_added (for the commit).
- Embed
diagram verbatim in the PR body's ## Test Coverage section (Step 19).
- Print a one-line summary:
Coverage: {coverage_pct}%, {gaps} gaps. {tests_added.length} tests added.
If the subagent fails, times out, or returns invalid JSON: Fall back to running the audit inline in the parent. Do not block /ship on subagent failure — partial results are better than none.
Step 8: Plan Completion Audit
Dispatch this step as a subagent using the Agent tool with subagent_type: "general-purpose". The subagent reads the plan file and every referenced code file in its own fresh context. Parent gets only the conclusion.
Subagent prompt: Pass these instructions to the subagent:
You are running a ship-workflow plan completion audit. The base branch is <base>. Use git diff <base>...HEAD to see what shipped. Do not commit or push — report only.
Plan File Discovery
-
Conversation context (primary): Check if there is an active plan file in this conversation. The host agent's system messages include plan file paths when in plan mode. If found, use it directly — this is the most reliable signal.
-
Content-based search (fallback): If no plan file is referenced in conversation context, search by content:
setopt +o nomatch 2>/dev/null || true
BRANCH=$(git branch --show-current 2>/dev/null | tr '/' '-')
REPO=$(basename "$(git rev-parse --show-toplevel 2>/dev/null)")
_PLAN_SLUG=$(git remote get-url origin 2>/dev/null | sed 's|.*[:/]\([^/]*/[^/]*\)\.git$|\1|;s|.*[:/]\([^/]*/[^/]*\)$|\1|' | tr '/' '-' | tr -cd 'a-zA-Z0-9._-') || true
_PLAN_SLUG="${_PLAN_SLUG:-$(basename "$PWD" | tr -cd 'a-zA-Z0-9._-')}"
for PLAN_DIR in "$HOME/.vibestack/projects/$_PLAN_SLUG" "$HOME/.claude/plans" "$HOME/.codex/plans" ".vibestack/plans"; do
[ -d "$PLAN_DIR" ] || continue
PLAN=$(ls -t "$PLAN_DIR"/*.md 2>/dev/null | xargs grep -l "$BRANCH" 2>/dev/null | head -1)
[ -z "$PLAN" ] && PLAN=$(ls -t "$PLAN_DIR"/*.md 2>/dev/null | xargs grep -l "$REPO" 2>/dev/null | head -1)
[ -z "$PLAN" ] && PLAN=$(find "$PLAN_DIR" -name '*.md' -mmin -1440 -maxdepth 1 2>/dev/null | xargs ls -t 2>/dev/null | head -1)
[ -n "$PLAN" ] && break
done
[ -n "$PLAN" ] && echo "PLAN_FILE: $PLAN" || echo "NO_PLAN_FILE"
- Validation: If a plan file was found via content-based search (not conversation context), read the first 20 lines and verify it is relevant to the current branch's work. If it appears to be from a different project or feature, treat as "no plan file found."
Error handling:
- No plan file found → skip with "No plan file detected — skipping."
- Plan file found but unreadable (permissions, encoding) → skip with "Plan file found but unreadable — skipping."
Actionable Item Extraction
Read the plan file. Extract every actionable item — anything that describes work to be done. Look for:
- Checkbox items:
- [ ] ... or - [x] ...
- Numbered steps under implementation headings: "1. Create ...", "2. Add ...", "3. Modify ..."
- Imperative statements: "Add X to Y", "Create a Z service", "Modify the W controller"
- File-level specifications: "New file: path/to/file.ts", "Modify path/to/existing.rb"
- Test requirements: "Test that X", "Add test for Y", "Verify Z"
- Data model changes: "Add column X to table Y", "Create migration for Z"
Ignore:
- Context/Background sections (
## Context, ## Background, ## Problem)
- Questions and open items (marked with ?, "TBD", "TODO: decide")
- Review report sections (
## VIBESTACK REVIEW REPORT)
- Explicitly deferred items ("Future:", "Out of scope:", "NOT in scope:", "P2:", "P3:", "P4:")
- CEO Review Decisions sections (these record choices, not work items)
Cap: Extract at most 50 items. If the plan has more, note: "Showing top 50 of N plan items — full list in plan file."
No items found: If the plan contains no extractable actionable items, skip with: "Plan file contains no actionable items — skipping completion audit."
For each item, note:
- The item text (verbatim or concise summary)
- Its category: CODE | TEST | MIGRATION | CONFIG | DOCS
Verification Mode
Before judging completion, classify HOW each item can be verified. The diff alone cannot prove every kind of work — items outside the current repo or system are structurally invisible to git diff.
- DIFF-VERIFIABLE — A code change in this repo would manifest in
git diff origin/<base>...HEAD. Examples: "add UserService" (file appears), "validate input X" (validation logic appears), "create users table" (migration appears).
- CROSS-REPO — Item names a file or change in a sibling repo (e.g.
~/Development/<other-repo>/docs/dashboard.md). The current diff CANNOT prove this.
- EXTERNAL-STATE — Item names state in an external system: managed-DB config/RLS, DNS records, hosting env vars, OAuth allowlists, third-party SaaS. The current diff CANNOT prove this.
- CONTENT-SHAPE — Item requires a file to follow a specific convention. In this repo: diff-verifiable. In another repo or system: see CROSS-REPO / EXTERNAL-STATE.
Verification dispatch:
- DIFF-VERIFIABLE → cross-reference against the diff (next section).
- CROSS-REPO → if the sibling repo is reachable on disk (try
~/Development/<repo>/, ~/code/<repo>/, the parent of the current repo), run [ -f <path> ]. File exists → DONE (cite path). Missing → NOT DONE (cite path). Path unreachable → UNVERIFIABLE (cite the manual check).
- EXTERNAL-STATE → UNVERIFIABLE. Cite the system and the specific check the user must perform.
- CONTENT-SHAPE in another repo → if the file exists, run any project-detected validator (scan the target repo's
package.json for a validate-*/check-docs/lint-* script) before falling back to UNVERIFIABLE. Pass → DONE; fail → NOT DONE (cite output). No validator: UNVERIFIABLE, citing both the path and the convention to confirm.
Path concreteness rule. If a plan item names a concrete filesystem path (absolute, ~/..., or <sibling-repo>/<file>), it MUST be classified DONE or NOT DONE based on [ -f <path> ]. UNVERIFIABLE is only valid when the path is genuinely abstract ("DNS record", "managed-DB allowlist") or the sibling root is unreachable on this machine. "I don't want to check" is not unreachable.
Honesty rule. Do NOT classify an item DONE just because related code shipped. Code that handles a deliverable is not the deliverable. When in doubt between DONE and UNVERIFIABLE, prefer UNVERIFIABLE — better to surface a confirmation prompt than silently miss a deliverable.
Cross-Reference Against Diff
Run git diff origin/<base>...HEAD and git log origin/<base>..HEAD --oneline to understand what was implemented.
For each extracted plan item, run the verification dispatch above, then classify:
- DONE — Clear evidence the item shipped. Cite the specific file(s) changed in the diff for DIFF-VERIFIABLE items, or the verified path that exists for CROSS-REPO items with a reachable sibling repo.
- PARTIAL — Some work toward this item exists but it's incomplete (e.g., model created but controller missing, function exists but edge cases not handled).
- NOT DONE — Verification ran and produced negative evidence (file missing, code absent in the diff, sibling-repo file confirmed absent).
- CHANGED — The item was implemented using a different approach than the plan described, but the same goal is achieved. Note the difference.
- UNVERIFIABLE — The diff and any reachable sibling-repo checks cannot prove or disprove this. Always applies to EXTERNAL-STATE items and to CROSS-REPO items where the sibling repo isn't reachable. Cite the specific manual verification the user must perform.
Be conservative with DONE — require clear evidence. A file being touched is not enough; the specific functionality described must be present.
Be generous with CHANGED — if the goal is met by different means, that counts as addressed.
Be honest with UNVERIFIABLE — better to surface items the user must manually confirm than silently classify them DONE.
Output Format
PLAN COMPLETION AUDIT
═══════════════════════════════
Plan: {plan file path}
## Implementation Items
[DONE] Create UserService — src/services/user_service.rb (+142 lines)
[PARTIAL] Add validation — model validates but missing controller checks
[NOT DONE] Add caching layer — no cache-related changes in diff
[CHANGED] "Redis queue" → implemented with Sidekiq instead
## Test Items
[DONE] Unit tests for UserService — test/services/user_service_test.rb
[NOT DONE] E2E test for signup flow
## Migration Items
[DONE] Create users table — db/migrate/20240315_create_users.rb
## Cross-Repo / External Items
[DONE] Add dashboard doc — ~/Development/other-repo/docs/dashboard.md (file exists)
[UNVERIFIABLE] DNS-only mode for dashboard.example.com — confirm in your DNS provider
─────────────────────────────────
COMPLETION: 5/9 DONE, 1 PARTIAL, 1 NOT DONE, 1 CHANGED, 1 UNVERIFIABLE
─────────────────────────────────
Gate Logic
After producing the completion checklist:
- All DONE or CHANGED: Pass. "Plan completion: PASS — all items addressed." Continue.
- Only PARTIAL items (no NOT DONE): Continue with a note in the PR body. Not blocking.
- UNVERIFIABLE items present (no NOT DONE): Blocking confirmation, per item. Never silently treat UNVERIFIABLE as DONE, and never blanket-confirm them with one question (that is the failure shape where the user picks "yes" without opening a single file).
- For each UNVERIFIABLE item, use AskUserQuestion with that item's specific manual check — "Confirm: does
~/Development/other-repo/docs/dashboard.md exist?", not "Have you checked all items?".
- Cap: if there are more than 5, present them as a numbered list first and ask whether to (1) confirm each individually (default, recommended), (2) stop and reduce scope, or (3) explicitly accept blanket-confirmation with a note that this skips real verification.
- Items the user confirms → treat as DONE and embed under
## Plan Completion — Manual Verifications in the PR body. Items they cannot confirm → carry as still-open manual checks in the PR body.
- Any NOT DONE items: Use AskUserQuestion:
- Show the completion checklist above
- "{N} items from the plan are NOT DONE. These were part of the original plan but are missing from the implementation."
- RECOMMENDATION: depends on item count and severity. If 1-2 minor items (docs, config), recommend B. If core functionality is missing, recommend A.
- Options:
A) Stop — implement the missing items before shipping
B) Ship anyway — defer these to a follow-up (will create P1 TODOs in Step 5.5)
C) These items were intentionally dropped — remove from scope
- If A: STOP. List the missing items for the user to implement.
- If B: Continue. For each NOT DONE item, create a P1 TODO in Step 5.5 with "Deferred from plan: {plan file path}".
- If C: Continue. Note in PR body: "Plan items intentionally dropped: {list}."
No plan file found: Skip entirely. "No plan file detected — skipping plan completion audit."
Include in PR body (Step 8): Add a ## Plan Completion section with the checklist summary.
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
{"total_items":N,"done":N,"changed":N,"deferred":N,"summary":"<markdown checklist for PR body>"}
Parent processing:
- Parse the LAST line of the subagent's output as JSON.
- Store
done, deferred for Step 20 metrics; use summary in PR body.
- If
deferred > 0 and no user override, present the deferred items via AskUserQuestion before continuing.
- Embed
summary in PR body's ## Plan Completion section (Step 19).
If the subagent fails or returns invalid JSON: Fall back to running the audit inline (parent runs the same plan-extraction + classification). If the inline fallback ALSO fails (plan file unreadable, parser error): do NOT silently pass. Surface it as an explicit AskUserQuestion — "Plan Completion audit could not run ({reason}). A) Skip audit and ship anyway (record 'audit skipped' in the PR body + Step 20 metrics), B) Stop and fix the audit." Default and recommended: B. A silent fail-open here is exactly how a missed deliverable ships unnoticed.
Step 8.1: Plan Verification
Automatically verify the plan's testing/verification steps using the /qa-only skill.
1. Check for verification section
Using the plan file already discovered in Step 8, look for a verification section. Match any of these headings: ## Verification, ## Test plan, ## Testing, ## How to test, ## Manual testing, or any section with verification-flavored items (URLs to visit, things to check visually, interactions to test).
If no verification section found: Skip with "No verification steps found in plan — skipping auto-verification."
If no plan file was found in Step 8: Skip (already handled).
2. Check for running dev server
Before invoking browse-based verification, check if a dev server is reachable:
curl -s -o /dev/null -w '%{http_code}' http://localhost:3000 2>/dev/null || \
curl -s -o /dev/null -w '%{http_code}' http://localhost:8080 2>/dev/null || \
curl -s -o /dev/null -w '%{http_code}' http://localhost:5173 2>/dev/null || \
curl -s -o /dev/null -w '%{http_code}' http://localhost:4000 2>/dev/null || echo "NO_SERVER"
If NO_SERVER: Skip with "No dev server detected — skipping plan verification. Run /qa separately after deploying."
3. Invoke /qa-only inline
Read the /qa-only skill from disk:
cat ${CLAUDE_SKILL_DIR:-$HOME/.claude/skills/ship}/../qa-only/SKILL.md
If unreadable: Skip with "Could not load /qa-only — skipping plan verification."
Follow the /qa-only workflow with these modifications:
- Skip the preamble (already handled by /ship)
- Use the plan's verification section as the primary test input — treat each verification item as a test case
- Use the detected dev server URL as the base URL
- Skip the fix loop — this is report-only verification during /ship
- Cap at the verification items from the plan — do not expand into general site QA
4. Gate logic
- All verification items PASS: Continue silently. "Plan verification: PASS."
- Any FAIL: Use AskUserQuestion:
- Show the failures with screenshot evidence
- RECOMMENDATION: Choose A if failures indicate broken functionality. Choose B if cosmetic only.
- Options:
A) Fix the failures before shipping (recommended for functional issues)
B) Ship anyway — known issues (acceptable for cosmetic issues)
- No verification section / no server / unreadable skill: Skip (non-blocking).
5. Include in PR body
Add a ## Verification Results section to the PR body (Step 19):
- If verification ran: summary of results (N PASS, M FAIL, K SKIPPED)
- If skipped: reason for skipping (no plan, no server, no verification section)
{{include lib/snippets/prior-learnings.md}}
Step 8.2: Scope Drift Detection
Before reviewing code quality, check: did they build what was requested — nothing more, nothing less?
-
Read TODOS.md (if it exists). Read PR description (gh pr view --json body --jq .body 2>/dev/null || true).
Read commit messages (git log origin/<base>..HEAD --oneline).
If no PR exists: rely on commit messages and TODOS.md for stated intent — this is the common case since /review runs before /ship creates the PR.
-
Identify the stated intent — what was this branch supposed to accomplish?
-
Run git diff origin/<base>...HEAD --stat and compare the files changed against the stated intent.
-
Evaluate with skepticism (incorporating plan completion results if available from an earlier step or adjacent section):
SCOPE CREEP detection:
- Files changed that are unrelated to the stated intent
- New features or refactors not mentioned in the plan
- "While I was in there..." changes that expand blast radius
MISSING REQUIREMENTS detection:
- Requirements from TODOS.md/PR description not addressed in the diff
- Test coverage gaps for stated requirements
- Partial implementations (started but not finished)
-
Output (before the main review begins):
```
Scope Check: [CLEAN / DRIFT DETECTED / REQUIREMENTS MISSING]
Intent: <1-line summary of what was requested>
Delivered: <1-line summary of what the diff actually does>
[If drift: list each out-of-scope change]
[If missing: list each unaddressed requirement]
```
-
This is INFORMATIONAL — does not block the review. Proceed to the next step.
Step 9: Pre-Landing Review
Review the diff for structural issues that tests don't catch.
-
Read .claude/skills/review/checklist.md. If the file cannot be read, STOP and report the error.
-
Run git diff $(git merge-base origin/<base> HEAD) to get the full diff (scoped to feature changes against the freshly-fetched base branch).
-
Apply the review checklist in two passes:
- Pass 1 (CRITICAL): SQL & Data Safety, LLM Output Trust Boundary
- Pass 2 (INFORMATIONAL): All remaining categories
Confidence Calibration
Every finding MUST include a confidence score (1-10):
| Score | Meaning | Display rule |
|---|
| 9-10 | Verified by reading specific code. Concrete bug or exploit demonstrated. | Show normally |
| 7-8 | High confidence pattern match. Very likely correct. | Show normally |
| 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" |
| 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. |
| 1-2 | Speculation. | Only report if severity would be P0. |
Finding format:
`[SEVERITY] (confidence: N/10) file:line — description`
Example:
`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause`
`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs`
Pre-emit verification gate (kills the "field doesn't exist" FP class)
Before any finding is promoted to the report, the gate requires:
-
Quote the specific code line that motivates the finding — file:line plus
the verbatim text of the line(s) that triggered it. If the finding is "field
X doesn't exist on model Y", quote the lines of class Y where the field
would live. If "dict.get() might return None", quote the dict initialization.
If "race condition between A and B", quote both A and B.
-
If you cannot quote the motivating line(s), the finding is unverified.
Force its confidence to 4-5 (suppressed from the main report). It still goes
into the appendix so reviewers can audit calibration, but the user does NOT
see it in the critical-pass output. Do not work around this by inventing
speculative confidence 7+ — that defeats the gate.
Framework-meta nudge: When the symbol is generated by a framework
metaclass, descriptor, ORM Meta inner-class, or migration history (Django
Meta, Rails has_many/scope, SQLAlchemy relationship/Column,
TypeORM decorators, Sequelize init/belongsTo, Prisma generated client),
quote the meta-construct (the Meta block, the migration, the decorator,
the schema file) instead of expecting the literal name in the class body.
The verification is "I read the source that creates this symbol", not "I
grep'd for the name and didn't find it." Deeper framework-aware verification
(model introspection, migration-history-aware checks, ORM dialect detection)
is deliberately out of scope for this lighter gate.
Calibration learning: If you report a finding with confidence < 7 and the user
confirms it IS a real issue, that is a calibration event. Your initial confidence was
too low. Log the corrected pattern as a learning so future reviews catch it with
higher confidence.
Design Review (conditional, diff-scoped)
Check if the diff touches frontend files:
eval "$(~/.vibestack/bin/vibe-diff-scope <base> 2>/dev/null || true)"
echo "SCOPE_FRONTEND=${SCOPE_FRONTEND:-false}"
If SCOPE_FRONTEND=false: Skip design review silently. No output.
If SCOPE_FRONTEND=true:
-
Check for DESIGN.md. If DESIGN.md or design-system.md exists in the repo root, read it. All design findings are calibrated against it — patterns blessed in DESIGN.md are not flagged. If not found, use universal design principles.
-
Read .claude/skills/review/design-checklist.md. If the file cannot be read, skip design review with a note: "Design checklist not found — skipping design review."
-
Read each changed frontend file (full file, not just diff hunks). Frontend files are identified by the patterns listed in the checklist.
-
Apply the design checklist against the changed files. For each item:
- [HIGH] mechanical CSS fix (
outline: none, !important, font-size < 16px): classify as AUTO-FIX
- [HIGH/MEDIUM] design judgment needed: classify as ASK
- [LOW] intent-based detection: present as "Possible — verify visually or run /design-review"
-
Include findings in the review output under a "Design Review" header, following the output format in the checklist. Design findings merge with code review findings into the same Fix-First flow.
-
Log the result for the Review Readiness Dashboard:
~/.vibestack/bin/vibe-review-log '{"skill":"design-review-lite","timestamp":"TIMESTAMP","status":"STATUS","findings":N,"auto_fixed":M,"commit":"COMMIT"}'
Substitute: TIMESTAMP = ISO 8601 datetime, STATUS = "clean" if 0 findings or "issues_found", N = total findings, M = auto-fixed count, COMMIT = output of git rev-parse --short HEAD.
- Codex design voice (optional, automatic if available):
command -v codex >/dev/null 2>&1 && echo "CODEX_AVAILABLE" || echo "CODEX_NOT_AVAILABLE"
If Codex is available, run a lightweight design check on the diff:
TMPERR_DRL=$(mktemp /tmp/codex-drl-XXXXXXXX)
_REPO_ROOT=$(git rev-parse --show-toplevel) || { echo "ERROR: not in a git repo" >&2; exit 1; }
command -v codex >/dev/null 2>&1 && codex exec "Review the git diff on this branch. Run 7 litmus checks (YES/NO each): 1. Brand/product unmistakable in first screen? 2. One strong visual anchor present? 3. Page understandable by scanning headlines only? 4. Each section has one job? 5. Are cards actually necessary? 6. Does motion improve hierarchy or atmosphere? 7. Would design feel premium with all decorative shadows removed? Flag any hard rejections: 1. Generic SaaS card grid as first impression 2. Beautiful image with weak brand 3. Strong headline with no clear action 4. Busy imagery behind text 5. Sections repeating same mood statement 6. Carousel with no narrative purpose 7. App UI made of stacked cards instead of layout 5 most important design findings only. Reference file:line." -C "$_REPO_ROOT" -s read-only -c 'model_reasoning_effort="high"' --enable web_search_cached < /dev/null 2>"$TMPERR_DRL"
Use a 5-minute timeout (timeout: 300000). After the command completes, read stderr:
cat "$TMPERR_DRL" && rm -f "$TMPERR_DRL"
Error handling: All errors are non-blocking. On auth failure, timeout, or empty response — skip with a brief note and continue.
Present Codex output under a CODEX (design): header, merged with the checklist findings above.
Include any design findings alongside the code review findings. They follow the same Fix-First flow below.
Step 9.1: Review Army — Specialist Dispatch
Detect stack and scope
eval "$(~/.vibestack/bin/vibe-diff-scope <base> 2>/dev/null || true)"
echo "SCOPE_FRONTEND=${SCOPE_FRONTEND:-false} SCOPE_BACKEND=${SCOPE_BACKEND:-false} SCOPE_AUTH=${SCOPE_AUTH:-false} SCOPE_MIGRATIONS=${SCOPE_MIGRATIONS:-false} SCOPE_API=${SCOPE_API:-false}"
STACK=""
[ -f Gemfile ] && STACK="${STACK}ruby "
[ -f package.json ] && STACK="${STACK}node "
[ -f requirements.txt ] || [ -f pyproject.toml ] && STACK="${STACK}python "
[ -f go.mod ] && STACK="${STACK}go "
[ -f Cargo.toml ] && STACK="${STACK}rust "
echo "STACK: ${STACK:-unknown}"
DIFF_INS=$(git diff $(git merge-base origin/<base> HEAD) --stat | tail -1 | grep -oE '[0-9]+ insertion' | grep -oE '[0-9]+' || echo "0")
DIFF_DEL=$(git diff $(git merge-base origin/<base> HEAD) --stat | tail -1 | grep -oE '[0-9]+ deletion' | grep -oE '[0-9]+' || echo "0")
DIFF_LINES=$((DIFF_INS + DIFF_DEL))
echo "DIFF_LINES: $DIFF_LINES"
TEST_FW=""
{ [ -f jest.config.ts ] || [ -f jest.config.js ]; } && TEST_FW="jest"
[ -f vitest.config.ts ] && TEST_FW="vitest"
{ [ -f spec/spec_helper.rb ] || [ -f .rspec ]; } && TEST_FW="rspec"
{ [ -f pytest.ini ] || [ -f conftest.py ]; } && TEST_FW="pytest"
[ -f go.mod ] && TEST_FW="go-test"
echo "TEST_FW: ${TEST_FW:-unknown}"
Read specialist hit rates (adaptive gating)
~/.vibestack/bin/vibe-specialist-stats 2>/dev/null || true
Each line tags a specialist GATE_CANDIDATE (dispatched 10+ times, never found
anything — safe to skip), NEVER_GATE (security / data-migration insurance —
always run), or active. Fresh project → everything active, full set runs.
Select specialists
Based on the scope signals above, select which specialists to dispatch.
Always-on (dispatch on every review with 50+ changed lines):
- Testing — read
~/.claude/skills/review/specialists/testing.md
- Maintainability — read
~/.claude/skills/review/specialists/maintainability.md
If DIFF_LINES < 50: Skip all specialists. Print: "Small diff ($DIFF_LINES lines) — specialists skipped." Continue to the Fix-First flow (item 4).
Conditional (dispatch if the matching scope signal is true):
3. Security — if SCOPE_AUTH=true, OR if SCOPE_BACKEND=true AND DIFF_LINES > 100. Read ~/.claude/skills/review/specialists/security.md
4. Performance — if SCOPE_BACKEND=true OR SCOPE_FRONTEND=true. Read ~/.claude/skills/review/specialists/performance.md
5. Data Migration — if SCOPE_MIGRATIONS=true. Read ~/.claude/skills/review/specialists/data-migration.md
6. API Contract — if SCOPE_API=true. Read ~/.claude/skills/review/specialists/api-contract.md
7. Design — if SCOPE_FRONTEND=true. Use the existing design review checklist at ~/.claude/skills/review/design-checklist.md
Adaptive gating
After scope-based selection, apply adaptive gating based on specialist hit rates:
For each conditional specialist that passed scope gating, check the specialist stats output above:
- If tagged
[GATE_CANDIDATE] (0 findings in 10+ dispatches): skip it. Print: "[specialist] auto-gated (0 findings in N reviews)."
- If tagged
[NEVER_GATE]: always dispatch regardless of hit rate. Security and data-migration are insurance policy specialists — they should run even when silent.
Force flags: If the user's prompt includes --security, --performance, --testing, --maintainability, --data-migration, --api-contract, --design, or --all-specialists, force-include that specialist regardless of gating.
Note which specialists were selected, gated, and skipped. Print the selection:
"Dispatching N specialists: [names]. Skipped: [names] (scope not detected). Gated: [names] (0 findings in N+ reviews)."
Dispatch specialists in parallel
For each selected specialist, launch an independent subagent via the Agent tool.
Launch ALL selected specialists in a single message (multiple Agent tool calls)
so they run in parallel. Each subagent has fresh context — no prior review bias.
Each specialist subagent prompt:
Construct the prompt for each specialist. The prompt includes:
- The specialist's checklist content (you already read the file above)
- Stack context: "This is a {STACK} project."
- Past learnings for this domain (if any exist):
~/.vibestack/bin/vibe-learnings-search --type pitfall --query "{specialist domain}" --limit 5 2>/dev/null || true
If learnings are found, include them: "Past learnings for this domain: {learnings}"
- Instructions:
"You are a specialist code reviewer. Read the checklist below, then run
git diff $(git merge-base origin/<base> HEAD) to get the full diff. Apply the checklist against the diff.
For each finding, output a JSON object on its own line:
{"severity":"CRITICAL|INFORMATIONAL","confidence":N,"path":"file","line":N,"category":"category","summary":"description","fix":"recommended fix","fingerprint":"path:line:category","specialist":"name"}
Required fields: severity, confidence, path, category, summary, specialist.
Optional: line, fix, fingerprint, evidence, test_stub.
If you can write a test that would catch this issue, include it in the test_stub field.
Use the detected test framework ({TEST_FW}). Write a minimal skeleton — describe/it/test
blocks with clear intent. Skip test_stub for architectural or design-only findings.
If no findings: output NO FINDINGS and nothing else.
Do not output anything else — no preamble, no summary, no commentary.
Stack context: {STACK}
Past learnings: {learnings or 'none'}
CHECKLIST:
{checklist content}"
Subagent configuration:
- Use
subagent_type: "general-purpose"
- Do NOT use
run_in_background — all specialists must complete before merge
- If any specialist subagent fails or times out, log the failure and continue with results from successful specialists. Specialists are additive — partial results are better than no results.
Step 9.2: Collect and merge findings
After all specialist subagents complete, collect their outputs.
Parse findings:
For each specialist's output:
- If output is "NO FINDINGS" — skip, this specialist found nothing
- Otherwise, parse each line as a JSON object. Skip lines that are not valid JSON.
- Collect all parsed findings into a single list, tagged with their specialist name.
Fingerprint and deduplicate:
For each finding, compute its fingerprint:
- If
fingerprint field is present, use it
- Otherwise:
{path}:{line}:{category} (if line is present) or {path}:{category}
Group findings by fingerprint. For findings sharing the same fingerprint:
- Keep the finding with the highest confidence score
- Tag it: "MULTI-SPECIALIST CONFIRMED ({specialist1} + {specialist2})"
- Boost confidence by +1 (cap at 10)
- Note the confirming specialists in the output
Apply confidence gates:
- Confidence 7+: show normally in the findings output
- Confidence 5-6: show with caveat "Medium confidence — verify this is actually an issue"
- Confidence 3-4: move to appendix (suppress from main findings)
- Confidence 1-2: suppress entirely
Compute PR Quality Score:
After merging, compute the quality score:
quality_score = max(0, 10 - (critical_count * 2 + informational_count * 0.5))
Cap at 10. Log this in the review result at the end.
Output merged findings:
Present the merged findings in the same format as the current review:
SPECIALIST REVIEW: N findings (X critical, Y informational) from Z specialists
[For each finding, in order: CRITICAL first, then INFORMATIONAL, sorted by confidence descending]
[SEVERITY] (confidence: N/10, specialist: name) path:line — summary
Fix: recommended fix
[If MULTI-SPECIALIST CONFIRMED: show confirmation note]
PR Quality Score: X/10
These findings flow into the Fix-First flow (item 4) alongside the checklist pass (Step 9).
The Fix-First heuristic applies identically — specialist findings follow the same AUTO-FIX vs ASK classification.
Compile per-specialist stats:
After merging findings, compile a specialists object for the review-log persist.
For each specialist (testing, maintainability, security, performance, data-migration, api-contract, design, red-team):
- If dispatched:
{"dispatched": true, "findings": N, "critical": N, "informational": N}
- If skipped by scope:
{"dispatched": false, "reason": "scope"}
- If skipped by gating:
{"dispatched": false, "reason": "gated"}
- If not applicable (e.g., red-team not activated): omit from the object
Include the Design specialist even though it uses design-checklist.md instead of the specialist schema files.
Remember these stats — you will need them for the review-log entry in Step 5.8.
Red Team dispatch (conditional)
Activation: Only if DIFF_LINES > 200 OR any specialist produced a CRITICAL finding.
If activated, dispatch one more subagent via the Agent tool (foreground, not background).
The Red Team subagent receives:
- The red-team checklist from
~/.claude/skills/review/specialists/red-team.md
- The merged specialist findings from Step 9.2 (so it knows what was already caught)
- The git diff command
Prompt: "You are a red team reviewer. The code has already been reviewed by N specialists
who found the following issues: {merged findings summary}. Your job is to find what they
MISSED. Read the checklist, run git diff $(git merge-base origin/<base> HEAD), and look for gaps.
Output findings as JSON objects (same schema as the specialists). Focus on cross-cutting
concerns, integration boundary issues, and failure modes that specialist checklists
don't cover."
If the Red Team finds additional issues, merge them into the findings list before
the Fix-First flow (item 4). Red Team findings are tagged with "specialist":"red-team".
If the Red Team returns NO FINDINGS, note: "Red Team review: no additional issues found."
If the Red Team subagent fails or times out, skip silently and continue.
Step 9.3: Cross-review finding dedup
Before classifying findings, check if any were previously skipped by the user in a prior review on this branch.
~/.vibestack/bin/vibe-review-read --json 2>/dev/null
Parse the output: only lines BEFORE ---CONFIG--- are JSONL entries (the output also contains ---CONFIG--- and ---HEAD--- footer sections that are not JSONL — ignore those).
For each JSONL entry that has a findings array:
- Collect all fingerprints where
action: "skipped"
- Note the
commit field from that entry
If skipped fingerprints exist, get the list of files changed since that review:
git diff --name-only <prior-review-commit> HEAD
For each current finding (from both the checklist pass (Step 9) and specialist review (Step 9.1-9.2)), check:
- Does its fingerprint match a previously skipped finding?
- Is the finding's file path NOT in the changed-files set?
If both conditions are true: suppress the finding. It was intentionally skipped and the relevant code hasn't changed.
Print: "Suppressed N findings from prior reviews (previously skipped by user)"
Only suppress skipped findings — never fixed or auto-fixed (those might regress and should be re-checked).
If no prior reviews exist or none have a findings array, skip this step silently.
Output a summary header: Pre-Landing Review: N issues (X critical, Y informational)
-
Classify each finding from both the checklist pass and specialist review (Step 9.1-Step 9.2) as AUTO-FIX or ASK per the Fix-First Heuristic in
checklist.md. Critical findings lean toward ASK; informational lean toward AUTO-FIX.
-
Auto-fix all AUTO-FIX items. Apply each fix. Output one line per fix:
[AUTO-FIXED] [file:line] Problem → what you did
-
If ASK items remain, present them in ONE AskUserQuestion:
- List each with number, severity, problem, recommended fix
- Per-item options: A) Fix B) Skip
- Overall RECOMMENDATION
- If 3 or fewer ASK items, you may use individual AskUserQuestion calls instead
-
After all fixes (auto + user-approved):
- If ANY fixes were applied: commit fixed files by name (
git add <fixed-files> && git commit -m "fix: pre-landing review fixes"), then STOP and tell the user to run /ship again to re-test.
- If no fixes applied (all ASK items skipped, or no issues found): continue to Step 12.
-
Output summary: Pre-Landing Review: N issues — M auto-fixed, K asked (J fixed, L skipped)
If no issues found: Pre-Landing Review: No issues found.
-
Persist the review result to the review log:
~/.vibestack/bin/vibe-review-log '{"skill":"review","timestamp":"TIMESTAMP","status":"STATUS","issues_found":N,"critical":N,"informational":N,"quality_score":SCORE,"specialists":SPECIALISTS_JSON,"findings":FINDINGS_JSON,"commit":"'"$(git rev-parse --short HEAD)"'","via":"ship"}'
Substitute TIMESTAMP (ISO 8601), STATUS ("clean" if no issues, "issues_found" otherwise),
and N values from the summary counts above. The via:"ship" distinguishes from standalone /review runs.
quality_score = the PR Quality Score computed in Step 9.2 (e.g., 7.5). If specialists were skipped (small diff), use 10.0
specialists = the per-specialist stats object compiled in Step 9.2. Each specialist that was considered gets an entry: {"dispatched":true/false,"findings":N,"critical":N,"informational":N} if dispatched, or {"dispatched":false,"reason":"scope|gated"} if skipped. Example: {"testing":{"dispatched":true,"findings":2,"critical":0,"informational":2},"security":{"dispatched":false,"reason":"scope"}}
findings = array of per-finding records. For each finding (from checklist pass and specialists), include: {"fingerprint":"path:line:category","severity":"CRITICAL|INFORMATIONAL","action":"ACTION"}. ACTION is "auto-fixed", "fixed" (user approved), or "skipped" (user chose Skip).
Save the review output — it goes into the PR body in Step 19.
Step 10: Address Greptile review comments (if PR exists)