cli-setup-verify

name

cli-setup-verify

description

Verify the Omnigent CLI's setup/onboarding flow, terminal UI/UX, and critical user journeys in a completely isolated, reproducible loop. Drives the real `omnigent` binary through a PTY (pexpect) inside a throwaway OMNIGENT_CONFIG_HOME / OMNIGENT_DATA_DIR sandbox that never touches the user's real ~/.omnigent, captures ANSI-stripped frames for UX inspection, and proves a change is verifiable via a before→fix→after baseline diff. Load when developing or reviewing a CLI setup/onboarding/REPL/picker change (omnigent/cli.py, omnigent/onboarding/*, omnigent/repl/*, scripts/install_oss.sh), reproducing a cold-start/first-run UX bug, or confirming a fix actually lands. Several agents can run it concurrently on separate worktrees.

Verifying the Omnigent CLI setup & UX in a closed loop

The Omnigent CLI's first impression is: curl | sh → run omnigent → pick a model credential → start a session. This skill lets an agent enter that flow, examine the UI/UX, and prove whether a change is verifiable — without a browser, without real credentials, and without ever touching the developer's real ~/.omnigent.

The engine is verify_cli.py (next to this file). It drives the real omnigent binary through a pseudo-terminal (pexpect) inside a throwaway sandbox, captures what renders, runs assertions, and prints one machine-readable SUMMARY {json} line.

The whole point is a verifiable loop, not a one-shot check:

1. Run a scenario on the unfixed code → baseline (--label before).
2. Make the change.
3. Run the same scenario → --label after.
4. Diff the two SUMMARY lines. A fix is "verifiable" only if a concrete check or note flips between the two runs. If it doesn't flip, you can't prove the fix did anything — go back to step 2.

Why this is safe (read first)

The real ~/.omnigent here can be many GB (chat DB, runner logs, native harness state). The sandbox isolates every write three ways:

• HOME is redirected into the sandbox by default. This is the load-bearing one. OMNIGENT_CONFIG_HOME / OMNIGENT_DATA_DIR (omnigent/cli.py _CONFIG_HOME_ENV_VAR / _DATA_DIR_ENV_VAR) redirect config + data — but the CLI's diagnostics logger ignores them: it writes a per-invocation cli-*.log under state_dir(), hardcoded to Path.home()/.omnigent/logs (omnigent_ui_sdk/terminal/_config.py). So only redirecting HOME keeps a non-help command (config list, the setup PTY spawns, server stop) from writing into the real home. The driver does this for you.
• --strip-path reduces PATH so node/tmux/claude/codex read as "not installed" → the true fresh-machine cold start.
• Ambient model keys (ANTHROPIC_API_KEY, …) are stripped from the child env unless you pass --keep-env-creds.

--inherit-home opts out of HOME isolation — use it only to reach a real credentialed REPL via ambient ~/.claude / ~/.databrickscfg auth. It is less safe: a non-help command then writes cli-*.log into the real ~/.omnigent/logs.

Every run fingerprints the real ~/.omnigent before and after (stat-only, no content reads): the top-level config files and the set of logs/cli-*.log diagnostic files. A new config file/mtime or a new cli-*.log basename trips real_config_untouched: false. With the default isolation that never happens; under --inherit-home it correctly does — which is exactly the violation the guard is meant to catch. If that check is ever false, stop and investigate. Run check-isolation first to confirm the loop is safe on your machine.

Prerequisites

• You're in the worktree whose code you want to test (each parallel agent on its own worktree). The driver runs omnigent from --repo's checkout.
• A Python with pexpect — the project's .venv/bin/python bundles it (pexpect>=4.9 in pyproject.toml). Run the driver with that interpreter.
• An omnigent binary: the driver auto-finds <repo>/.venv/bin/omnigent, or pass --omnigent <path>.
• The setup / picker / help / cold-start scenarios need no credentials and no harness. Only repl-commands needs a working harness + credential: pass --inherit-home (ambient ~/.claude auth) and/or --keep-env-creds (env API key) with --agent. It reports skipped, never a false pass, when the prompt isn't reachable.

Quick start

REPO=/path/to/your/worktree
PY=$REPO/.venv/bin/python
DRV=$REPO/.claude/skills/cli-setup-verify/verify_cli.py

# 0. Prove the sandbox is safe on this machine (do this once).
#    HOME is isolated by default — no flag needed.
$PY $DRV --scenario check-isolation --repo "$REPO"

# 1. See exactly what a brand-new user sees on a fresh machine.
$PY $DRV --scenario cold-start --strip-path --keep-sandbox --repo "$REPO"
#    → reads the printed `artifacts` path, then `cat <that>/cold_start.txt`

# 2. Lint the top-level help (and any subcommand's).
$PY $DRV --scenario help-snapshot --repo "$REPO"
$PY $DRV --scenario help-snapshot --subcommand server --repo "$REPO"

Each run prints SUMMARY {…} and exits non-zero if any check failed (a skipped scenario exits 0). Pipe to … | grep '^SUMMARY' | python -m json.tool to read it.

Scenario catalog

Scenario	What it drives	Key checks / notes	Maps to findings
check-isolation	omnigent config list in the sandbox (no PTY)	config_list_ran, sandbox_config_home_used, real_config_untouched	safety gate for everything
cold-start	omnigent setup via PTY on a simulated fresh machine	onboarding_rendered, harness_menu_present; note guided_default_affordance	cold-start dead-end; missing "recommended start here"
setup-snapshot	omnigent setup, optional --nav-down N arrow steps	menu_rendered; saves a frame per step	picker markers/footer/alignment; narrow-terminal at 80×24
help-snapshot	omnigent [--subcommand] --help (no PTY)	help_rendered, no_param_leak, no_update_dup; note top_level_command_count	:param leak, duplicate update/upgrade, command sprawl
repl-commands	omnigent run <agent> REPL, sends /help + /quit	help_lists_commands; note quit_advertised	REPL discoverability (/help, /quit)

--list-scenarios prints them too. Captured frames land in the printed artifacts dir as both <name>.txt (ANSI-stripped, for reading/asserting) and <name>.ansi.txt (raw, to see real colors with less -R).

The verifiable loop — a worked example

Finding: "server --help leaks Sphinx :param/:returns into user help."

# BEFORE the fix (on the unfixed code):
$PY $DRV --scenario help-snapshot --subcommand server --label before --repo "$REPO"
#   → "no_param_leak": {"ok": false, ...}      ← bug reproduced (the baseline)

# ... make the change (move :param docs into # comments) ...

# AFTER the fix:
$PY $DRV --scenario help-snapshot --subcommand server --label after --repo "$REPO"
#   → "no_param_leak": {"ok": true, "detail": "clean"}   ← flipped → fix is verifiable

The same shape proves the update/upgrade duplicate (no_update_dup), the cold-start dead-end (guided_default_affordance note flips absent→present), or REPL /quit discoverability (quit_advertised note flips no→yes). If the check/note doesn't flip, the fix isn't proven — that is the signal to keep working, and it's exactly the judgment the loop exists to force.

If a finding has no machine check yet, add one (see "Adding a scenario") so the fix becomes provable instead of asserted.

Examining UI/UX deliberately

• Narrow terminal is the default. The driver uses 80×24 — the size a new user's window actually is, and where banner overflow and picker redraw-past-the-bottom bugs appear. Re-run with --cols 120 --rows 40 to compare the roomy layout; diff the two frames.
• Read the frame, don't just trust the check. cat <artifacts>/cold_start.txt shows the literal screen — the all-✗ menu, the footer hint (Esc back at the root), the marker (❯), alignment of the status gutter. The frame is the UX evidence.
• Compare pickers for consistency. setup-snapshot --nav-down 3 captures the harness menu as you move; eyeball marker/footer/highlight drift against the theme and resume pickers (different engines render differently).

Covering all critical user journeys

This skill owns the setup / onboarding / first-run / TUI journeys. The repo already has complementary CUJ coverage — use both:

• Live setup/UX journeys → this skill's scenarios (cold-start, setup, pickers, help, REPL discoverability).
• Deeper end-to-end journeys → tests/e2e/test_journey_*.py (first session to code, resume/disconnect, fork/explore, file upload, collaboration, …). Run a slice with the project's gated runner, e.g. uv run --frozen --group test python -m pytest tests/e2e/test_journey_first_session_to_code.py -q.
• Reusable PTY helpers live in tests/e2e/omnigent/_pexpect_harness.py (spawn_omnigent_run, wait_for_ready, submit_prompt, await_turn_complete, clean_exit) and the snapshot comparator in tests/e2e/omnigent/_snapshot.py — prefer extending those over re-inventing.

To drive a surface this skill doesn't script yet, spawn it by hand with the sandbox env and the keys the driver exports (KEY_UP/KEY_DOWN/KEY_ENTER/ KEY_ESC), then drain() and save_frame() the result.

Teardown — non-negotiable

• The driver force-kills the PTY child and its descendants, and runs omnigent server stop against the sandbox to reap any spawned background server. After a run, confirm nothing leaked: pgrep -af "omnigent.*(server|runner|host._daemon)" — anything bound to your sandbox's data dir is yours to kill.
• The sandbox temp dir is deleted unless --keep-sandbox. If you keep one for inspection, rm -rf it when done.
• Always drive the CLI through the driver (which redirects HOME + the config/data knobs), never a bare omnigent setup — that would write to the real ~/.omnigent. If you pass --inherit-home, expect cli-*.log writes to the real ~/.omnigent/logs and a real_config_untouched: false — that's the guard working, not a bug.

Honesty

If you can't reach the surface under test (no harness, no credential, headless limit), the scenario must report skipped — do not claim a CUJ passed. The strongest evidence for a fix is a reproduced baseline (before) plus the flipped after; report both SUMMARY lines, not a summary of a summary.

Adding a scenario

Write scenario_<name>(args, sandbox, result) in verify_cli.py: drive the CLI (reuse pexpect.spawn(... env=sandbox.env, dimensions=(args.rows, args.cols)), drain(), save_frame(), the KEY_* constants), record findings with result.add(name, ok, detail) (fails the run) or result.notes.append(...) (informational, for before/after flips), register it in SCENARIOS, and add a row to the catalog above. Keep one assertion per real, observable behavior so a fix is provable as a single check flip.

Code under test

• First-run dispatch / no-arg routing: omnigent/cli.py (run, the first-run plan, _run_configure_harnesses_interactive).
• Onboarding: omnigent/onboarding/* (setup.py, interactive.py, configure_models.py, provider_selection.py, detected.py).
• TUI / REPL & pickers: omnigent/repl/* (_repl.py, _theme_picker.py, _resume_picker.py), omnigent/_terminal_picker_theme.py.
• Installer: scripts/install_oss.sh.