benchmark-models

name	benchmark-models
preamble-tier	1
version	1.0.0
description	Cross-model benchmark for gstack skills. Runs the same prompt through Claude, GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost, and optionally quality via LLM judge. Answers "which model is actually best for this skill?" with data instead of vibes. Separate from /benchmark, which measures web page performance. Use when: "benchmark models", "compare models", "which model is best for X", "cross-model comparison", "model shootout". (gstack) Voice triggers (speech-to-text aliases): "compare models", "model shootout", "which model is best".
triggers	["cross model benchmark","compare claude gpt gemini","benchmark skill across models","which model should I use"]
allowed-tools	["Bash","Read","AskUserQuestion"]

Preamble (run first)

_UPD=$(~/.claude/skills/gstack/bin/gstack-update-check 2>/dev/null || .claude/skills/gstack/bin/gstack-update-check 2>/dev/null || true)
[ -n "$_UPD" ] && echo "$_UPD" || true
mkdir -p ~/.gstack/sessions
touch ~/.gstack/sessions/"$PPID"
_SESSIONS=$(find ~/.gstack/sessions -mmin -120 -type f 2>/dev/null | wc -l | tr -d ' ')
find ~/.gstack/sessions -mmin +120 -type f -exec rm {} + 2>/dev/null || true
_PROACTIVE=$(~/.claude/skills/gstack/bin/gstack-config get proactive 2>/dev/null || echo "true")
_PROACTIVE_PROMPTED=$([ -f ~/.gstack/.proactive-prompted ] && echo "yes" || echo "no")
_BRANCH=$(git branch --show-current 2>/dev/null || echo "unknown")
echo "BRANCH: $_BRANCH"
_SKILL_PREFIX=$(~/.claude/skills/gstack/bin/gstack-config get skill_prefix 2>/dev/null || echo "false")
echo "PROACTIVE: $_PROACTIVE"
echo "PROACTIVE_PROMPTED: $_PROACTIVE_PROMPTED"
echo "SKILL_PREFIX: $_SKILL_PREFIX"
source <(~/.claude/skills/gstack/bin/gstack-repo-mode 2>/dev/null) || true
REPO_MODE=${REPO_MODE:-unknown}
echo "REPO_MODE: $REPO_MODE"
_LAKE_SEEN=$([ -f ~/.gstack/.completeness-intro-seen ] && echo "yes" || echo "no")
echo "LAKE_INTRO: $_LAKE_SEEN"
_TEL=$(~/.claude/skills/gstack/bin/gstack-config get telemetry 2>/dev/null || true)
_TEL_PROMPTED=$([ -f ~/.gstack/.telemetry-prompted ] && echo "yes" || echo "no")
_TEL_START=$(date +%s)
_SESSION_ID="$$-$(date +%s)"
echo "TELEMETRY: ${_TEL:-off}"
echo "TEL_PROMPTED: $_TEL_PROMPTED"
# Writing style verbosity (V1: default = ELI10, terse = tighter V0 prose.
# Read on every skill run so terse mode takes effect without a restart.)
_EXPLAIN_LEVEL=$(~/.claude/skills/gstack/bin/gstack-config get explain_level 2>/dev/null || echo "default")
if [ "$_EXPLAIN_LEVEL" != "default" ] && [ "$_EXPLAIN_LEVEL" != "terse" ]; then _EXPLAIN_LEVEL="default"; fi
echo "EXPLAIN_LEVEL: $_EXPLAIN_LEVEL"
# Question tuning (see /plan-tune). Observational only in V1.
_QUESTION_TUNING=$(~/.claude/skills/gstack/bin/gstack-config get question_tuning 2>/dev/null || echo "false")
echo "QUESTION_TUNING: $_QUESTION_TUNING"
mkdir -p ~/.gstack/analytics
if [ "$_TEL" != "off" ]; then
echo '{"skill":"benchmark-models","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","repo":"'$(basename "$(git rev-parse --show-toplevel 2>/dev/null)" 2>/dev/null || echo "unknown")'"}'  >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
fi
# zsh-compatible: use find instead of glob to avoid NOMATCH error
for _PF in $(find ~/.gstack/analytics -maxdepth 1 -name '.pending-*' 2>/dev/null); do
  if [ -f "$_PF" ]; then
    if [ "$_TEL" != "off" ] && [ -x "~/.claude/skills/gstack/bin/gstack-telemetry-log" ]; then
      ~/.claude/skills/gstack/bin/gstack-telemetry-log --event-type skill_run --skill _pending_finalize --outcome unknown --session-id "$_SESSION_ID" 2>/dev/null || true
    fi
    rm -f "$_PF" 2>/dev/null || true
  fi
  break
done
# Learnings count
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" 2>/dev/null || true
_LEARN_FILE="${GSTACK_HOME:-$HOME/.gstack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
  _LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
  echo "LEARNINGS: $_LEARN_COUNT entries loaded"
  if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
    ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 3 2>/dev/null || true
  fi
else
  echo "LEARNINGS: 0"
fi
# Session timeline: record skill start (local-only, never sent anywhere)
~/.claude/skills/gstack/bin/gstack-timeline-log '{"skill":"benchmark-models","event":"started","branch":"'"$_BRANCH"'","session":"'"$_SESSION_ID"'"}' 2>/dev/null &
# Check if CLAUDE.md has routing rules
_HAS_ROUTING="no"
if [ -f CLAUDE.md ] && grep -q "## Skill routing" CLAUDE.md 2>/dev/null; then
  _HAS_ROUTING="yes"
fi
_ROUTING_DECLINED=$(~/.claude/skills/gstack/bin/gstack-config get routing_declined 2>/dev/null || echo "false")
echo "HAS_ROUTING: $_HAS_ROUTING"
echo "ROUTING_DECLINED: $_ROUTING_DECLINED"
# Vendoring deprecation: detect if CWD has a vendored gstack copy
_VENDORED="no"
if [ -d ".claude/skills/gstack" ] && [ ! -L ".claude/skills/gstack" ]; then
  if [ -f ".claude/skills/gstack/VERSION" ] || [ -d ".claude/skills/gstack/.git" ]; then
    _VENDORED="yes"
  fi
fi
echo "VENDORED_GSTACK: $_VENDORED"
echo "MODEL_OVERLAY: claude"
# Checkpoint mode (explicit = no auto-commit, continuous = WIP commits as you go)
_CHECKPOINT_MODE=$(~/.claude/skills/gstack/bin/gstack-config get checkpoint_mode 2>/dev/null || echo "explicit")
_CHECKPOINT_PUSH=$(~/.claude/skills/gstack/bin/gstack-config get checkpoint_push 2>/dev/null || echo "false")
echo "CHECKPOINT_MODE: $_CHECKPOINT_MODE"
echo "CHECKPOINT_PUSH: $_CHECKPOINT_PUSH"
# Detect spawned session (OpenClaw or other orchestrator)
[ -n "$OPENCLAW_SESSION" ] && echo "SPAWNED_SESSION: true" || true

If PROACTIVE is "false", do not proactively suggest gstack skills AND do not auto-invoke skills based on conversation context. Only run skills the user explicitly types (e.g., /qa, /ship). If you would have auto-invoked a skill, instead briefly say: "I think /skillname might help here — want me to run it?" and wait for confirmation. The user opted out of proactive behavior.

If SKILL_PREFIX is "true", the user has namespaced skill names. When suggesting or invoking other gstack skills, use the /gstack- prefix (e.g., /gstack-qa instead of /qa, /gstack-ship instead of /ship). Disk paths are unaffected — always use ~/.claude/skills/gstack/[skill-name]/SKILL.md for reading skill files.

If output shows UPGRADE_AVAILABLE <old> <new>: read ~/.claude/skills/gstack/gstack-upgrade/SKILL.md and follow the "Inline upgrade flow" (auto-upgrade if configured, otherwise AskUserQuestion with 4 options, write snooze state if declined).

If output shows JUST_UPGRADED <from> <to> AND SPAWNED_SESSION is NOT set: tell the user "Running gstack v{to} (just updated!)" and then check for new features to surface. For each per-feature marker below, if the marker file is missing AND the feature is plausibly useful for this user, use AskUserQuestion to let them try it. Fire once per feature per user, NOT once per upgrade.

In spawned sessions (SPAWNED_SESSION = "true"): SKIP feature discovery entirely. Just print "Running gstack v{to}" and continue. Orchestrators do not want interactive prompts from sub-sessions.

Feature discovery markers and prompts (one at a time, max one per session):

1. ~/.claude/skills/gstack/.feature-prompted-continuous-checkpoint → Prompt: "Continuous checkpoint auto-commits your work as you go with WIP: prefix so you never lose progress to a crash. Local-only by default — doesn't push anywhere unless you turn that on. Want to try it?" Options: A) Enable continuous mode, B) Show me first (print the section from the preamble Continuous Checkpoint Mode), C) Skip. If A: run ~/.claude/skills/gstack/bin/gstack-config set checkpoint_mode continuous. Always: touch ~/.claude/skills/gstack/.feature-prompted-continuous-checkpoint

2. ~/.claude/skills/gstack/.feature-prompted-model-overlay → Inform only (no prompt): "Model overlays are active. MODEL_OVERLAY: {model} shown in the preamble output tells you which behavioral patch is applied. Override with --model when regenerating skills (e.g., bun run gen:skill-docs --model gpt-5.4). Default is claude." Always: touch ~/.claude/skills/gstack/.feature-prompted-model-overlay

After handling JUST_UPGRADED (prompts done or skipped), continue with the skill workflow.

If WRITING_STYLE_PENDING is yes: You're on the first skill run after upgrading to gstack v1. Ask the user once about the new default writing style. Use AskUserQuestion:

v1 prompts = simpler. Technical terms get a one-sentence gloss on first use, questions are framed in outcome terms, sentences are shorter.

Keep the new default, or prefer the older tighter prose?

Options:

• A) Keep the new default (recommended — good writing helps everyone)
• B) Restore V0 prose — set explain_level: terse

If A: leave explain_level unset (defaults to default). If B: run ~/.claude/skills/gstack/bin/gstack-config set explain_level terse.

Always run (regardless of choice):

rm -f ~/.gstack/.writing-style-prompt-pending
touch ~/.gstack/.writing-style-prompted

This only happens once. If WRITING_STYLE_PENDING is no, skip this entirely.

If LAKE_INTRO is no: Before continuing, introduce the Completeness Principle. Tell the user: "gstack follows the Boil the Lake principle — always do the complete thing when AI makes the marginal cost near-zero. Read more: https://garryslist.org/posts/boil-the-ocean" Then offer to open the essay in their default browser:

open https://garryslist.org/posts/boil-the-ocean
touch ~/.gstack/.completeness-intro-seen

Only run open if the user says yes. Always run touch to mark as seen. This only happens once.

If TEL_PROMPTED is no AND LAKE_INTRO is yes: After the lake intro is handled, ask the user about telemetry. Use AskUserQuestion:

Help gstack get better! Community mode shares usage data (which skills you use, how long they take, crash info) with a stable device ID so we can track trends and fix bugs faster. No code, file paths, or repo names are ever sent. Change anytime with gstack-config set telemetry off.

Options:

• A) Help gstack get better! (recommended)
• B) No thanks

If A: run ~/.claude/skills/gstack/bin/gstack-config set telemetry community

If B: ask a follow-up AskUserQuestion:

How about anonymous mode? We just learn that someone used gstack — no unique ID, no way to connect sessions. Just a counter that helps us know if anyone's out there.

Options:

• A) Sure, anonymous is fine
• B) No thanks, fully off

If B→A: run ~/.claude/skills/gstack/bin/gstack-config set telemetry anonymous If B→B: run ~/.claude/skills/gstack/bin/gstack-config set telemetry off

Always run:

touch ~/.gstack/.telemetry-prompted

This only happens once. If TEL_PROMPTED is yes, skip this entirely.

If PROACTIVE_PROMPTED is no AND TEL_PROMPTED is yes: After telemetry is handled, ask the user about proactive behavior. Use AskUserQuestion:

gstack can proactively figure out when you might need a skill while you work — like suggesting /qa when you say "does this work?" or /investigate when you hit a bug. We recommend keeping this on — it speeds up every part of your workflow.

Options:

• A) Keep it on (recommended)
• B) Turn it off — I'll type /commands myself

If A: run ~/.claude/skills/gstack/bin/gstack-config set proactive true If B: run ~/.claude/skills/gstack/bin/gstack-config set proactive false

Always run:

touch ~/.gstack/.proactive-prompted

This only happens once. If PROACTIVE_PROMPTED is yes, skip this entirely.

If HAS_ROUTING is no AND ROUTING_DECLINED is false AND PROACTIVE_PROMPTED is yes: Check if a CLAUDE.md file exists in the project root. If it does not exist, create it.

Use AskUserQuestion:

gstack works best when your project's CLAUDE.md includes skill routing rules. This tells Claude to use specialized workflows (like /ship, /investigate, /qa) instead of answering directly. It's a one-time addition, about 15 lines.

Options:

• A) Add routing rules to CLAUDE.md (recommended)
• B) No thanks, I'll invoke skills manually

If A: Append this section to the end of CLAUDE.md:

## Skill routing

When the user's request matches an available skill, ALWAYS invoke it using the Skill
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
The skill has specialized workflows that produce better results than ad-hoc answers.

Key routing rules:
- Product ideas, "is this worth building", brainstorming → invoke office-hours
- Bugs, errors, "why is this broken", 500 errors → invoke investigate
- Ship, deploy, push, create PR → invoke ship
- QA, test the site, find bugs → invoke qa
- Code review, check my diff → invoke review
- Update docs after shipping → invoke document-release
- Weekly retro → invoke retro
- Design system, brand → invoke design-consultation
- Visual audit, design polish → invoke design-review
- Architecture review → invoke plan-eng-review
- Save progress, checkpoint, resume → invoke checkpoint
- Code quality, health check → invoke health

Then commit the change: git add CLAUDE.md && git commit -m "chore: add gstack skill routing rules to CLAUDE.md"

If B: run ~/.claude/skills/gstack/bin/gstack-config set routing_declined true Say "No problem. You can add routing rules later by running gstack-config set routing_declined false and re-running any skill."

This only happens once per project. If HAS_ROUTING is yes or ROUTING_DECLINED is true, skip this entirely.

If VENDORED_GSTACK is yes: This project has a vendored copy of gstack at .claude/skills/gstack/. Vendoring is deprecated. We will not keep vendored copies up to date, so this project's gstack will fall behind.

Use AskUserQuestion (one-time per project, check for ~/.gstack/.vendoring-warned-$SLUG marker):

This project has gstack vendored in .claude/skills/gstack/. Vendoring is deprecated. We won't keep this copy up to date, so you'll fall behind on new features and fixes.

Want to migrate to team mode? It takes about 30 seconds.

Options:

• A) Yes, migrate to team mode now
• B) No, I'll handle it myself

If A:

1. Run git rm -r .claude/skills/gstack/
2. Run echo '.claude/skills/gstack/' >> .gitignore
3. Run ~/.claude/skills/gstack/bin/gstack-team-init required (or optional)
4. Run git add .claude/ .gitignore CLAUDE.md && git commit -m "chore: migrate gstack from vendored to team mode"
5. Tell the user: "Done. Each developer now runs: cd ~/.claude/skills/gstack && ./setup --team"

If B: say "OK, you're on your own to keep the vendored copy up to date."

Always run (regardless of choice):

eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" 2>/dev/null || true
touch ~/.gstack/.vendoring-warned-${SLUG:-unknown}

This only happens once per project. If the marker file exists, skip entirely.

If SPAWNED_SESSION is "true", you are running inside a session spawned by an AI orchestrator (e.g., OpenClaw). In spawned sessions:

• Do NOT use AskUserQuestion for interactive prompts. Auto-choose the recommended option.
• Do NOT run upgrade checks, telemetry prompts, routing injection, or lake intro.
• Focus on completing the task and reporting results via prose output.
• End with a completion report: what shipped, decisions made, anything uncertain.

Model-Specific Behavioral Patch (claude)

The following nudges are tuned for the claude model family. They are subordinate to skill workflow, STOP points, AskUserQuestion gates, plan-mode safety, and /ship review gates. If a nudge below conflicts with skill instructions, the skill wins. Treat these as preferences, not rules.

Todo-list discipline. When working through a multi-step plan, mark each task complete individually as you finish it. Do not batch-complete at the end. If a task turns out to be unnecessary, mark it skipped with a one-line reason.

Think before heavy actions. For complex operations (refactors, migrations, non-trivial new features), briefly state your approach before executing. This lets the user course-correct cheaply instead of mid-flight.

Dedicated tools over Bash. Prefer Read, Edit, Write, Glob, Grep over shell equivalents (cat, sed, find, grep). The dedicated tools are cheaper and clearer.

Voice

Tone: direct, concrete, sharp, never corporate, never academic. Sound like a builder, not a consultant. Name the file, the function, the command. No filler, no throat-clearing.

Writing rules: No em dashes (use commas, periods, "..."). No AI vocabulary (delve, crucial, robust, comprehensive, nuanced, etc.). Short paragraphs. End with what to do.

The user always has context you don't. Cross-model agreement is a recommendation, not a decision — the user decides.

Completion Status Protocol

When completing a skill workflow, report status using one of:

• DONE — All steps completed successfully. Evidence provided for each claim.
• DONE_WITH_CONCERNS — Completed, but with issues the user should know about. List each concern.
• BLOCKED — Cannot proceed. State what is blocking and what was tried.
• NEEDS_CONTEXT — Missing information required to continue. State exactly what you need.

Escalation

It is always OK to stop and say "this is too hard for me" or "I'm not confident in this result."

Bad work is worse than no work. You will not be penalized for escalating.

• If you have attempted a task 3 times without success, STOP and escalate.
• If you are uncertain about a security-sensitive change, STOP and escalate.
• If the scope of work exceeds what you can verify, STOP and escalate.

Escalation format:

STATUS: BLOCKED | NEEDS_CONTEXT
REASON: [1-2 sentences]
ATTEMPTED: [what you tried]
RECOMMENDATION: [what the user should do next]

Operational Self-Improvement

Before completing, reflect on this session:

• Did any commands fail unexpectedly?
• Did you take a wrong approach and have to backtrack?
• Did you discover a project-specific quirk (build order, env vars, timing, auth)?
• Did something take longer than expected because of a missing flag or config?

If yes, log an operational learning for future sessions:

~/.claude/skills/gstack/bin/gstack-learnings-log '{"skill":"SKILL_NAME","type":"operational","key":"SHORT_KEY","insight":"DESCRIPTION","confidence":N,"source":"observed"}'

Replace SKILL_NAME with the current skill name. Only log genuine operational discoveries. Don't log obvious things or one-time transient errors (network blips, rate limits). A good test: would knowing this save 5+ minutes in a future session? If yes, log it.

Telemetry (run last)

After the skill workflow completes (success, error, or abort), log the telemetry event. Determine the skill name from the name: field in this file's YAML frontmatter. Determine the outcome from the workflow result (success if completed normally, error if it failed, abort if the user interrupted).

PLAN MODE EXCEPTION — ALWAYS RUN: This command writes telemetry to ~/.gstack/analytics/ (user config directory, not project files). The skill preamble already writes to the same directory — this is the same pattern. Skipping this command loses session duration and outcome data.

Run this bash:

_TEL_END=$(date +%s)
_TEL_DUR=$(( _TEL_END - _TEL_START ))
rm -f ~/.gstack/analytics/.pending-"$_SESSION_ID" 2>/dev/null || true
# Session timeline: record skill completion (local-only, never sent anywhere)
~/.claude/skills/gstack/bin/gstack-timeline-log '{"skill":"SKILL_NAME","event":"completed","branch":"'$(git branch --show-current 2>/dev/null || echo unknown)'","outcome":"OUTCOME","duration_s":"'"$_TEL_DUR"'","session":"'"$_SESSION_ID"'"}' 2>/dev/null || true
# Local analytics (gated on telemetry setting)
if [ "$_TEL" != "off" ]; then
echo '{"skill":"SKILL_NAME","duration_s":"'"$_TEL_DUR"'","outcome":"OUTCOME","browse":"USED_BROWSE","session":"'"$_SESSION_ID"'","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
fi
# Remote telemetry (opt-in, requires binary)
if [ "$_TEL" != "off" ] && [ -x ~/.claude/skills/gstack/bin/gstack-telemetry-log ]; then
  ~/.claude/skills/gstack/bin/gstack-telemetry-log \
    --skill "SKILL_NAME" --duration "$_TEL_DUR" --outcome "OUTCOME" \
    --used-browse "USED_BROWSE" --session-id "$_SESSION_ID" 2>/dev/null &
fi

Replace SKILL_NAME with the actual skill name from frontmatter, OUTCOME with success/error/abort, and USED_BROWSE with true/false based on whether $B was used. If you cannot determine the outcome, use "unknown". The local JSONL always logs. The remote binary only runs if telemetry is not off and the binary exists.

Plan Mode Safe Operations

In plan mode, these are always allowed (they inform the plan, don't modify source): $B (browse), $D (design), codex exec/codex review, writes to ~/.gstack/, writes to the plan file, open for generated artifacts.

Skill Invocation During Plan Mode

If the user invokes a skill in plan mode, that skill takes precedence over generic plan mode behavior. Treat it as executable instructions, not reference. Follow step by step. AskUserQuestion calls satisfy plan mode's end-of-turn requirement. At a STOP point, stop immediately. Do not continue the workflow past a STOP point and do not call ExitPlanMode there. Commands marked "PLAN MODE EXCEPTION — ALWAYS RUN" execute. Other writes need to be already permitted above or explicitly exception-marked. Call ExitPlanMode only after the skill workflow completes — only then call ExitPlanMode (or if the user tells you to cancel the skill or leave plan mode).

Plan Status Footer

In plan mode, before ExitPlanMode: if the plan file lacks a ## GSTACK REVIEW REPORT section, run ~/.claude/skills/gstack/bin/gstack-review-read and append a report. With JSONL entries (before ---CONFIG---), format the standard runs/status/findings table. With NO_REVIEWS or empty, append a 5-row placeholder table (CEO/Codex/Eng/ Design/DX Review) with all zeros and verdict "NO REVIEWS YET — run /autoplan". If a richer review report already exists, skip — review skills wrote it.

PLAN MODE EXCEPTION — always allowed (it's the plan file).

/benchmark-models — Cross-Model Skill Benchmark

You are running the /benchmark-models workflow. Wraps the gstack-model-benchmark binary with an interactive flow that picks a prompt, confirms providers, previews auth, and runs the benchmark.

Different from /benchmark — that skill measures web page performance (Core Web Vitals, load times). This skill measures AI model performance on gstack skills or arbitrary prompts.

---

Step 0: Locate the binary

BIN="$HOME/.claude/skills/gstack/bin/gstack-model-benchmark"
[ -x "$BIN" ] || BIN=".claude/skills/gstack/bin/gstack-model-benchmark"
[ -x "$BIN" ] || { echo "ERROR: gstack-model-benchmark not found. Run ./setup in the gstack install dir." >&2; exit 1; }
echo "BIN: $BIN"

If not found, stop and tell the user to reinstall gstack.

---

Step 1: Choose a prompt

Use AskUserQuestion with the preamble format:

• Re-ground: current project + branch.
• Simplify: "A cross-model benchmark runs the same prompt through 2-3 AI models and shows you how they compare on speed, cost, and output quality. What prompt should we use?"
• RECOMMENDATION: A because benchmarking against a real skill exposes tool-use differences, not just raw generation.
• Options:
  • A) Benchmark one of my gstack skills (we'll pick which skill next). Completeness: 10/10.
  • B) Use an inline prompt — type it on the next turn. Completeness: 8/10.
  • C) Point at a prompt file on disk — specify path on the next turn. Completeness: 8/10.

If A: list top-level gstack skills that have SKILL.md files (from find . -maxdepth 2 -name SKILL.md -not -path './.*'), ask the user to pick one via a second AskUserQuestion. Use the picked SKILL.md path as the prompt file.

If B: ask the user for the inline prompt. Use it verbatim via --prompt "<text>".

If C: ask for the path. Verify it exists. Use as positional argument.

---

Step 2: Choose providers

"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini --dry-run

Show the dry-run output. The "Adapter availability" section tells the user which providers will actually run (OK) vs skip (NOT READY — remediation hint included).

If ALL three show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest claude login, codex login, or gemini login / export GOOGLE_API_KEY.

If at least one is OK: AskUserQuestion:

• Simplify: "Which models should we include? The dry-run above showed which are authed. Unauthed ones will be skipped cleanly — they won't abort the batch."
• RECOMMENDATION: A (all authed providers) because running as many as possible gives the richest comparison.
• Options:
  • A) All authed providers. Completeness: 10/10.
  • B) Only Claude. Completeness: 6/10 (no cross-model signal — use /ship's review for solo claude benchmarks instead).
  • C) Pick two — specify on next turn. Completeness: 8/10.

---

Step 3: Decide on judge

[ -n "$ANTHROPIC_API_KEY" ] || grep -q 'ANTHROPIC' "$HOME/.claude/.credentials.json" 2>/dev/null && echo "JUDGE_AVAILABLE" || echo "JUDGE_UNAVAILABLE"

If judge is available, AskUserQuestion:

• Simplify: "The quality judge scores each model's output on a 0-10 scale using Anthropic's Claude as a tiebreaker. Adds ~$0.05/run. Recommended if you care about output quality, not just latency and cost."
• RECOMMENDATION: A — the whole point is comparing quality, not just speed.
• Options:
  • A) Enable judge (adds ~$0.05). Completeness: 10/10.
  • B) Skip judge — speed/cost/tokens only. Completeness: 7/10.

If judge is NOT available, skip this question and omit the --judge flag.

---

Step 4: Run the benchmark

Construct the command from Step 1, 2, 3 decisions:

"$BIN" <prompt-spec> --models <picked-models> [--judge] --output table

Where <prompt-spec> is either --prompt "<text>" (Step 1B), a file path (Step 1A or 1C), and <picked-models> is the comma-separated list from Step 2.

Stream the output as it arrives. This is slow — each provider runs the prompt fully. Expect 30s-5min depending on prompt complexity and whether --judge is on.

---

Step 5: Interpret results

After the table prints, summarize for the user:

• Fastest — provider with lowest latency.
• Cheapest — provider with lowest cost.
• Highest quality (if --judge ran) — provider with highest score.
• Best overall — use judgment. If judge ran: quality-weighted. Otherwise: note the tradeoff the user needs to make.

If any provider hit an error (auth/timeout/rate_limit), call it out with the remediation path.

---

Step 6: Offer to save results

AskUserQuestion:

• Simplify: "Save this benchmark as JSON so you can compare future runs against it?"
• RECOMMENDATION: A — skill performance drifts as providers update their models; a saved baseline catches quality regressions.
• Options:
  • A) Save to ~/.gstack/benchmarks/<date>-<skill-or-prompt-slug>.json. Completeness: 10/10.
  • B) Just print, don't save. Completeness: 5/10 (loses trend data).

If A: re-run with --output json and tee to the dated file. Print the path so the user can diff future runs against it.

---

Important Rules

• Never run a real benchmark without Step 2's dry-run first. Users need to see auth status before spending API calls.
• Never hardcode model names. Always pass providers from user's Step 2 choice — the binary handles the rest.
• Never auto-include --judge. It adds real cost; user must opt in.
• If zero providers are authed, STOP. Don't attempt the benchmark — it produces no useful output.
• Cost is visible. Every run shows per-provider cost in the table. Users should see it before the next run.