showcase-driven-cli

name	showcase-driven-cli
description	Use when designing a CLI for new software OR evaluating/improving an existing CLI — before writing argument parsers, subcommands, or implementation code. Works for both greenfield CLIs and existing CLIs that need testing or redesign.

Showcase-Driven CLI Design

Overview

Design CLIs by building interactive showcases first, testing them on agents and humans, then implementing. A good CLI needs no second thought — if a domain expert can't figure out the command within a few guided attempts, the design is bad.

Core insight: Reading a spec tells you what a CLI says it does. Physically typing commands tells you what it feels like. Showcases are the artifact that drives every phase.

When to Use

• User wants to design a CLI for new software (greenfield)
• User wants to evaluate or improve an existing CLI's usability
• User wants to build showcases/tutorials for an existing CLI
• User is a domain expert (knows WHAT, not HOW to express it in commands)
• Before any argument parser code, clap derive, subcommand implementation

When NOT to Use

• Adding a single flag to an existing CLI (just do it)
• Non-interactive tools (pure stdin/stdout filters)

Entry Point Decision

Before starting, determine: does a CLI already exist?

digraph entry_decision {
    rankdir=TB;
    "CLI exists?" [shape=diamond];
    "Phase 0:\nDiscover Existing CLI" [shape=box, style=filled, fillcolor="#e2d9f3"];
    "Phase 1:\nSocratic Brainstorm\n(greenfield)" [shape=box, style=filled, fillcolor="#d4edda"];
    "Showcases derived\nfrom real commands" [shape=box];
    "Phase 2a:\nBuild Demo Shell" [shape=box, style=filled, fillcolor="#cce5ff"];

    "CLI exists?" -> "Phase 0:\nDiscover Existing CLI" [label="yes"];
    "CLI exists?" -> "Phase 1:\nSocratic Brainstorm\n(greenfield)" [label="no"];
    "Phase 0:\nDiscover Existing CLI" -> "Showcases derived\nfrom real commands";
    "Showcases derived\nfrom real commands" -> "Phase 2a:\nBuild Demo Shell";
    "Phase 1:\nSocratic Brainstorm\n(greenfield)" -> "Phase 2a:\nBuild Demo Shell";
}

• CLI exists → Phase 0 (discover), then skip Phase 1, go straight to Phase 2
• No CLI → Phase 1 (Socratic brainstorm), then Phase 2

Process Flow

digraph showcase_driven_cli {
    rankdir=TB;
    
    "CLI exists?" [shape=diamond, style=filled, fillcolor="#e2d9f3"];
    "Phase 0:\nDiscover Existing CLI" [shape=box, style=filled, fillcolor="#e2d9f3"];
    "Phase 1:\nSocratic Brainstorm" [shape=box, style=filled, fillcolor="#d4edda"];
    "Spec + Showcases\ndefined?" [shape=diamond];
    "Phase 2a:\nBuild Demo Shell\n(Zig scaffold)" [shape=box, style=filled, fillcolor="#cce5ff"];
    "Phase 2b:\nSubagent Testing\n(no --help)" [shape=box, style=filled, fillcolor="#fff3cd"];
    "Synthesis:\nRead JSONL, report" [shape=box];
    "Revise spec +\nrebuild demo" [shape=box];
    "Phase 2c:\nHuman Testing\n(native + WASM)" [shape=box, style=filled, fillcolor="#fff3cd"];
    "Synthesis 2:\nRead all JSONL" [shape=box];
    "Design stable?" [shape=diamond];
    "Phase 3:\nDefine ACs from showcases" [shape=box, style=filled, fillcolor="#f8d7da"];
    "Invoke writing-plans\nfor real implementation" [shape=doublecircle];
    "Verify semantic\nequivalence" [shape=box];
    "Ship showcase +\ntutorial subcommands" [shape=box, style=filled, fillcolor="#d4edda"];

    "CLI exists?" -> "Phase 0:\nDiscover Existing CLI" [label="yes"];
    "CLI exists?" -> "Phase 1:\nSocratic Brainstorm" [label="no"];
    "Phase 0:\nDiscover Existing CLI" -> "Spec + Showcases\ndefined?";
    "Phase 1:\nSocratic Brainstorm" -> "Spec + Showcases\ndefined?";
    "Spec + Showcases\ndefined?" -> "Phase 1:\nSocratic Brainstorm" [label="no, keep talking"];
    "Spec + Showcases\ndefined?" -> "Phase 2a:\nBuild Demo Shell\n(Zig scaffold)" [label="yes"];
    "Phase 2a:\nBuild Demo Shell\n(Zig scaffold)" -> "Phase 2b:\nSubagent Testing\n(no --help)";
    "Phase 2b:\nSubagent Testing\n(no --help)" -> "Synthesis:\nRead JSONL, report";
    "Synthesis:\nRead JSONL, report" -> "Revise spec +\nrebuild demo";
    "Revise spec +\nrebuild demo" -> "Phase 2c:\nHuman Testing\n(native + WASM)";
    "Phase 2c:\nHuman Testing\n(native + WASM)" -> "Synthesis 2:\nRead all JSONL";
    "Synthesis 2:\nRead all JSONL" -> "Design stable?";
    "Design stable?" -> "Revise spec +\nrebuild demo" [label="no"];
    "Design stable?" -> "Phase 3:\nDefine ACs from showcases" [label="yes"];
    "Phase 3:\nDefine ACs from showcases" -> "Invoke writing-plans\nfor real implementation";
    "Invoke writing-plans\nfor real implementation" -> "Verify semantic\nequivalence";
    "Verify semantic\nequivalence" -> "Ship showcase +\ntutorial subcommands";
}

Phase 0: Discover Existing CLI

When: A CLI already exists (binary is installed, source code is available, or both).

Goal: Understand what the CLI already does, how it's structured, and what workflows it supports — then generate showcases from real commands instead of brainstorming from scratch.

Step 0a: Discovery

Run an exploration agent (or do it yourself) to map the existing CLI:

1. Find the binary name and check it runs: which <cli> and <cli> --version
2. Capture top-level help: <cli> --help — extract all subcommands and global flags
3. Capture subcommand help: <cli> <subcommand> --help for every subcommand found
4. Read CLAUDE.md / README: Look for documented usage examples, workflows, and pipelines
5. Read the CLI source code: Find argument parser definitions (clap derives, subcommand enums, etc.) to discover flags/options that help text might not fully explain
6. Check for example data: Look for test fixtures, example files, or example subcommands that provide inputs for showcases

Output: A CLI inventory document listing:

• Binary name and all subcommands
• For each subcommand: positional args, flags, options, stdin/stdout behavior
• Known workflows and pipelines (multi-command sequences)
• Example data files available

Step 0b: Exercise the CLI

Run real commands to verify the inventory and capture actual outputs:

1. Run each subcommand with example data — capture both terminal and piped (JSON) output
2. Run documented pipelines end-to-end — verify they work, note any that fail
3. Test edge cases: stdin (-), missing args, invalid inputs — see what error messages look like
4. Note surprises: Commands that don't do what you'd expect, confusing flag names, missing features

This is NOT testing discoverability yet — you're building ground truth. Save actual command+output pairs.

Step 0c: Generate Showcases from Real Commands

Convert the exercised workflows into showcase definitions. Key differences from greenfield:

• expected_sequence comes from real commands that already work, plus plausible aliases a user might try
• expected_output is captured from actual CLI output, not invented
• context reflects real scenarios the CLI is designed for
• Showcases should cover the documented workflows from CLAUDE.md/README, plus the most common single-command uses

Showcase selection heuristic:

1. The single most common task (happy path) → first showcase
2. Each major subcommand → one showcase
3. Each documented pipeline → one showcase
4. Any command with non-obvious flags → one showcase

After generating showcases, skip Phase 1 and go directly to Phase 2a (build demo shell). The showcases serve the same role as Phase 1's output — they define what to test.

When Phase 0 Leads Back to Phase 1

If discovery reveals the CLI is poorly structured, incomplete, or the user explicitly wants to redesign it, Phase 0 findings become input to Phase 1 (Socratic brainstorm). The inventory doc grounds the conversation in what exists, so brainstorming focuses on what to change rather than starting from nothing.

---

Red Flags — STOP If You're Doing These

Thought	Reality
"Let me just design the commands first"	Commands emerge from conversation, not from reading code
"I'll validate workflows after the design"	Workflows ARE the design — showcases come first
"Let me think about clap/argparse structure"	Implementation details come in phase 3, not before
"I can skip the demo shell, the spec is clear"	You cannot evaluate feel from a spec. Build the demo.
"Subagent testing is enough, skip humans"	Agents guess differently than humans. Both are required.
"Let me propose a full command structure"	Propose nothing until you've explored the user's mental model
"Subagents can use --help"	No. --help makes tests trivial. Subagents must guess cold.
"I'll pre-generate all inputs and pipe them"	No. Subagents must interact step-by-step, reacting to hints.
"CLI exists, so skip straight to testing"	Phase 0 discovers first. You need the inventory before showcases.
"CLI exists, so I'll brainstorm from scratch"	Phase 0 grounds you in what exists. Don't ignore working commands.
"I know what the CLI does from the README"	Run the commands. README may be stale. Actual output is ground truth.

Phase 1: Socratic Brainstorm

Goal: Co-discover the CLI structure through conversation. The command design EMERGES — you do not propose it.

Method: Invoke superpowers:brainstorming with CLI-specific guidance. You have an internal checklist (NEVER exposed as questions):

• User's current workflow and vocabulary
• Natural verbs and nouns they use
• The single most common task (happy path)
• Input/output formats
• Who else will use this

How to converse:

• Ask about their workflow, listen for natural language
• Generate options based on what they say, let them react
• Do NOT propose command structures early
• Reflecting back vocabulary is fine — mapping to subcommands is premature. Stay in the user's language until THEY reach for command syntax.
• Gradually converge: workflow talk -> verb/noun discovery -> user reaches for syntax -> command structure crystallizes -> showcases co-designed

Output:

1. CLI spec (markdown): subcommands, flags, arguments
2. Showcase definitions (JSON files in showcases/ directory)

Showcase Definition Format

{
  "id": "qft-overlap",
  "persona": "quantum researcher familiar with QFT circuits",
  "goal": "Compute overlap with zero state",
  "context": "You have a tool called `yaosim` installed. Your colleague sent you QFT.json. Have a try.",
  "expected_sequence": [
    "yaosim run QFT.json --overlap-with-zero",
    "yaosim run QFT.json --task overlap"
  ],
  "hint_threshold": 4,
  "hints": [
    "Use `run` to simulate. Try an overlap flag.",
    "The flag is --overlap-with-zero.",
    "Try: yaosim run QFT.json --overlap-with-zero"
  ],
  "expected_output": "Overlap <0|U|0>:\n  Amplitude: 0.2500+0.0000i\n  |Amplitude|^2: 0.062500",
  "rating_prompt": "How natural did this command feel?"
}

Design rules for expected_sequence (learned from testing):

• Include common aliases users will try (hyphenated, short flags, both notation styles)
• Prefer hyphenated flags (--overlap-with-zero) over camelCase (--overlapwithzero)
• Provide shortcut flags that collapse multi-flag patterns (--expect <obs> for --task expect --obs <obs>)
• Accept both --to and --format style flags for conversions
• Add short forms for common flags (--nq for --nqubits)

First showcase MUST introduce the CLI root name in context.

Phase 2: Demo Shell

2a. Build

Generate a Zig project from the scaffold. See scaffold/ for template files.

Key properties:

• Zero external deps — zig build only (user manages toolchain via mise, etc.)
• Two build targets: native binary + WASM with HTML wrapper
• All outputs are hardcoded strings — pure mock, no real software logic
• Not a prototype — different language (Zig) reinforces this

Two Interaction Modes (CRITICAL)

The demo shell MUST support two modes:

Interactive mode (for humans): ./cli-demo — modern shell experience with line editing.

Stateless CLI mode (for subagents): Each call = one action, returns immediately.

./cli-demo list                          # List showcases
./cli-demo context <N>                   # Show scenario N
./cli-demo try <N> <attempt#> <command>  # Try a command → JSON response
./cli-demo shell <N> <command>           # Shell cmd (ls, cat — NOT help)
./cli-demo rate <N> <score> <comment>    # Submit rating

Why stateless mode is required: Subagents call the binary via Bash, one invocation per attempt. They see the JSON response, think, call again. Pre-generating all inputs and piping them defeats the purpose — the agent never reacts to hints.

Example subagent interaction:

$ ./cli-demo try 1 1 yaosim simulate bell.json
{"status":"hint","showcase":"basic-simulate","attempt":1,"hint":"The main verb is `run`."}

$ ./cli-demo try 1 2 yaosim run bell.json
{"status":"success","showcase":"basic-simulate","attempt":2}
Circuit: 2 qubits | ...

Interactive Shell Features

The interactive mode must feel like a real modern shell. Use scaffold/src/line_editor.zig (pure Zig, no deps):

• Line editing — cursor movement, backspace, home/end (Ctrl-A/E), word-jump (Ctrl-Left/Right), Ctrl-K/U/W
• Tab completion — context-aware: completes subcommands after yaosim, file names after run, flags after --, task names after --task. Single match fills in; multiple matches display vertically below prompt then redraw.
• History — Up/Down arrow recalls previous attempts within the session
• Inline hints — greyed-out suggestions as you type (e.g., type yaosim and see run|generate|convert|show|info dimmed)
• ANSI colors — green prompt $, yellow hints, green output, red threshold hits, magenta showcase headers, blue rating prompts
• Ctrl-L — clear screen and redraw

Tab completion is word-level: yaosim run Q<tab> completes to yaosim run QFT.json, not just matching full command strings.

Shell Commands

Mock basic shell commands (don't count as attempts, logged as explore events):

• ls — lists mock files (also tab-completable)
• cat <file> — shows mock file contents (file names tab-complete)
• Help is available in interactive mode for humans but BANNED for subagent testing

JSONL Logger

Append-only feedback.jsonl. Event types: attempt, success, threshold_hit, rating, explore, quit.

The explore event separates shell/help usage from real attempts — critical for synthesis to distinguish "discovered via help" from "guessed cold."

WASM Build for Sharing

Zig → wasm32-freestanding + minimal HTML wrapper. JSONL in browser memory, download button at end. Shareable via static hosting.

2b. Subagent Testing (FIRST)

CRITICAL RULES:

1. NO --help. Subagents guess cold. With help, everything is 1-shot and the test is useless. Validated in practice: same agent scored 5/5 with help, 2-3/5 without.
2. Step-by-step interaction via stateless mode. Agent reads hint, thinks, tries again. NOT pre-generated input pipes.
3. Foreground dispatch. Background subagents may fail to get Bash permissions. Run in foreground or ensure Bash is pre-approved.
4. CAN use shell N ls and shell N cat <file> — viewing files is exploratory, not cheating.

Dispatch guidance (adapt per project):

• Give domain knowledge generously — they should understand the vocabulary, just not the CLI
• Vary personas: senior expert, grad student, adjacent-field researcher, tool-specific user
• Instruct honesty: "Type your real first instinct, not what you think the right answer is"
• 3-5 subagents. Each renames feedback.jsonl to feedback-agent-{N}.jsonl

2c. Human Testing (SECOND)

After subagent feedback incorporated and demo rebuilt:

• Humans CAN use --help — tests real-world discoverability
• Share WASM build with colleagues — minimal onboarding: "type what feels natural, ~5 min"
• Do NOT explain CLI commands beforehand

Synthesis & Iteration

Synthesis agent reads ALL JSONL and reports:

• Per-showcase: avg attempts, threshold hits, common wrong guesses
• Pattern analysis: what did testers type? Natural vocabulary reveals better names
• Concrete proposals: "add --format alias because 3/4 tried it"

Revision patterns that commonly surface:

• CamelCase/mashed flags → hyphenated (--overlapwithzero → --overlap-with-zero)
• Missing short flags → add them (--nq for --nqubits)
• Missing convention aliases (--format for --to)
• Multi-flag combos → shortcuts (--expect <obs> for --task expect --obs <obs>)
• Implementation-specific names → discoverable aliases (easybuild → generate alias)

Cycle: build → subagent test (no help) → revise → human test (help OK) → revise → repeat.

Phase 3: Implement

For Greenfield CLIs (came through Phase 1)

3a. Acceptance Criteria

Each showcase → AC with semantically equivalent output (not exact string match).

3b. Plan & Build

Invoke superpowers:writing-plans. Target language = real software's language.

3c. Verify

Run showcase scenarios against real CLI. Judge semantic equivalence.

3d. Ship Built-in Subcommands

• <cli> showcase — interactive demo, same hints/ratings/JSONL
• <cli> tutorial — guided learning for non-domain-experts, built on showcase infrastructure

For Existing CLIs (came through Phase 0)

The CLI already works. Phase 3 focuses on improvements surfaced by testing:

3a. Triage Findings

From synthesis, categorize changes:

• Aliases/shortcuts — add flag aliases, short forms (low risk, high value)
• Better defaults — change default output format, add missing - stdin support
• Structural changes — rename subcommands, merge/split commands (higher risk)
• New features — commands testers expected but don't exist yet

3b. Implement Iteratively

Start with aliases/shortcuts (quick wins), then defaults, then structural changes. Each change should make at least one showcase succeed in fewer attempts.

3c. Re-run Showcases Against Real CLI

After changes, run the same showcase scenarios against the updated real CLI. Compare attempt counts before/after.

3d. Ship Showcase Subcommand

Add <cli> showcase as a built-in subcommand using the tested showcase definitions — doubles as onboarding tutorial.

Common Mistakes

Mistake	Fix
Proposing commands before exploring mental model	Phase 1 is Socratic — commands emerge, not proposed
Skipping demo shell ("spec is enough")	You cannot evaluate feel from text. Build the demo.
Letting subagents use --help	Ban it. With help = 1-shot. Without = real issues surface.
Pre-generating subagent inputs	They must react to hints step-by-step via stateless mode.
Only building interactive mode	Stateless CLI mode is required for subagent interaction.
Using CamelCase mashed flags	Hyphenated: --overlap-with-zero not --overlapwithzero
No short flags or aliases	Add them. --nq, --format, --expect.
Raw stdin readline for interactive mode	Use line_editor.zig: tab completion, history, colors, cursor. Demo must feel like a real shell.
Full-command tab completion only	Word-level: complete files after run, flags after --, task names after --task
Treating demo as throwaway	Demo becomes <cli> showcase + <cli> tutorial
Existing CLI: inventing showcases without running it	Phase 0b: exercise the real CLI first, capture actual outputs
Existing CLI: trusting README over actual behavior	Run every command. README may be stale or aspirational.
Existing CLI: skipping Phase 0 inventory	You need the full subcommand/flag map before generating showcases
Existing CLI: redesigning what already works	Phase 0 showcases test discoverability of current design first