cli-setup-verify

# Verifying the Omnigent CLI setup & UX in a closed loop The Omnigent CLI's first impression is: `curl | sh` → run `omnigent` → pick a model credential → start a session. This skill lets an agent **enter that flow, examine the UI/UX, and prove whether a change is verifiable** — without a browser, without real credentials, and **without ever touching the developer's real `~/.omnigent`**. The engine is `verify_cli.py` (next to this file). It drives the real `omnigent` binary through a pseudo-terminal (`pexpect`) inside a throwaway sandbox, captures what renders, runs assertions, and prints one machine-readable `SUMMARY {json}` line. > **The whole point is a verifiable loop**, not a one-shot check: > 1. Run a scenario on the **unfixed** code → baseline (`--label before`). > 2. Make the change. > 3. Run the **same** scenario → `--label after`. > 4. Diff the two `SUMMARY` lines. A fix is "verifiable" only if a concrete > check or note **flips** between the two runs. If it doesn't flip, you > can't prove the fix did anything — go back to step 2. ## Why this is safe (read first) The real `~/.omnigent` here can be **many GB** (chat DB, runner logs, native harness state). The sandbox isolates every write three ways: - **`HOME` is redirected into the sandbox by default.** This is the load-bearing one. `OMNIGENT_CONFIG_HOME` / `OMNIGENT_DATA_DIR` (`omnigent/cli.py` `_CONFIG_HOME_ENV_VAR` / `_DATA_DIR_ENV_VAR`) redirect config + data — but the CLI's **diagnostics logger ignores them**: it writes a per-invocation `cli-*.log` under `state_dir()`, hardcoded to `Path.home()/.omnigent/logs` (`omnigent_ui_sdk/terminal/_config.py`). So only redirecting `HOME` keeps a non-help command (`config list`, the setup PTY spawns, `server stop`) from writing into the real home. The driver does this for you. - `--strip-path` reduces `PATH` so `node`/`tmux`/`claude`/`codex` read as "not installed" → the true fresh-machine cold start. - Ambient model keys (`ANTHROPIC_API_KEY`, …) are stripped from the child env unless you pass `--keep-env-creds`. **`--inherit-home` opts out** of `HOME` isolation — use it only to reach a real credentialed REPL via ambient `~/.claude` / `~/.databrickscfg` auth. It is **less safe**: a non-help command then writes `cli-*.log` into the real `~/.omnigent/logs`. Every run **fingerprints the real `~/.omnigent` before and after** (stat-only, no content reads): the top-level config files **and** the set of `logs/cli-*.log` diagnostic files. A new config file/mtime *or* a new `cli-*.log` basename trips `real_config_untouched: false`. With the default isolation that never happens; under `--inherit-home` it correctly does — which is exactly the violation the guard is meant to catch. If that check is ever `false`, stop and investigate. Run `check-isolation` first to confirm the loop is safe on your machine. ## Prerequisites - You're in the **worktree whose code you want to test** (each parallel agent on its own worktree). The driver runs `omnigent` from `--repo`'s checkout. - A Python with `pexpect` — the project's `.venv/bin/python` bundles it (`pexpect>=4.9` in `pyproject.toml`). Run the driver with that interpreter. - An `omnigent` binary: the driver auto-finds `<repo>/.venv/bin/omnigent`, or pass `--omnigent <path>`. - The setup / picker / help / cold-start scenarios need **no credentials and no harness**. Only `repl-commands` needs a working harness + credential: pass `--inherit-home` (ambient `~/.claude` auth) and/or `--keep-env-creds` (env API key) with `--agent`. It reports `skipped`, never a false pass, when the prompt isn't reachable. ## Quick start ```bash REPO=/path/to/your/worktree PY=$REPO/.venv/bin/python DRV=$REPO/.claude/skills/cli-setup-verify/verify_cli.py # 0. Prove the sandbox is safe on this machine (do this once). # HOME is isolated by default — no flag needed. $PY $DRV --scenario check-isolation --repo "$REPO" # 1. See exactly what a brand-new user sees on a fresh machine. $PY $DRV --scenario cold-start --strip-path --keep-sandbox --repo "$REPO" # → reads the printed `artifacts` path, then `cat <that>/cold_start.txt` # 2. Lint the top-level help (and any subcommand's). $PY $DRV --scenario help-snapshot --repo "$REPO" $PY $DRV --scenario help-snapshot --subcommand server --repo "$REPO" ``` Each run prints `SUMMARY {…}` and exits non-zero if any check failed (a `skipped` scenario exits 0). Pipe to `… | grep '^SUMMARY' | python -m json.tool` to read it. ## Scenario catalog | Scenario | What it drives | Key checks / notes | Maps to findings | |---|---|---|---| | `check-isolation` | `omnigent config list` in the sandbox (no PTY) | `config_list_ran`, `sandbox_config_home_used`, `real_config_untouched` | safety gate for everything | | `cold-start` | `omnigent setup` via PTY on a simulated fresh machine | `onboarding_rendered`, `harness_menu_present`; note `guided_default_affordance` | cold-start dead-end; missing "recommended start here" | | `setup-snapshot` | `omnigent setup`, optional `--nav-down N` arrow steps | `menu_rendered`; saves a frame per step | picker markers/footer/alignment; narrow-terminal at 80×24 | | `help-snapshot` | `omnigent [--subcommand] --help` (no PTY) | `help_rendered`, `no_param_leak`, `no_update_dup`; note `top_level_command_count` | `:param` leak, duplicate `update`/`upgrade`, command sprawl | | `repl-commands` | `omnigent run <agent>` REPL, sends `/help` + `/quit` | `help_lists_commands`; note `quit_advertised` | REPL discoverability (`/help`, `/quit`) | `--list-scenarios` prints them too. Captured frames land in the printed `artifacts` dir as both `<name>.txt` (ANSI-stripped, for reading/asserting) and `<name>.ansi.txt` (raw, to see real colors with `less -R`). ## The verifiable loop — a worked example Finding: *"`server --help` leaks Sphinx `:param`/`:returns` into user help."* ```bash # BEFORE the fix (on the unfixed code): $PY $DRV --scenario help-snapshot --subcommand server --label before --repo "$REPO" # → "no_param_leak": {"ok": false, ...} ← bug reproduced (the baseline) # ... make the change (move :param docs into # comments) ... # AFTER the fix: $PY $DRV --scenario help-snapshot --subcommand server --label after --repo "$REPO" # → "no_param_leak": {"ok": true, "detail": "clean"} ← flipped → fix is verifiable ``` The same shape proves the `update`/`upgrade` duplicate (`no_update_dup`), the cold-start dead-end (`guided_default_affordance` note flips `absent`→`present`), or REPL `/quit` discoverability (`quit_advertised` note flips `no`→`yes`). **If the check/note doesn't flip, the fix isn't proven** — that is the signal to keep working, and it's exactly the judgment the loop exists to force. If a finding has no machine check yet, add one (see "Adding a scenario") so the fix becomes provable instead of asserted. ## Examining UI/UX deliberately - **Narrow terminal is the default.** The driver uses **80×24** — the size a new user's window actually is, and where banner overflow and picker redraw-past-the-bottom bugs appear. Re-run with `--cols 120 --rows 40` to compare the roomy layout; diff the two frames. - **Read the frame, don't just trust the check.** `cat <artifacts>/cold_start.txt` shows the literal screen — the all-`✗` menu, the footer hint (`Esc back` at the root), the marker (`❯`), alignment of the status gutter. The frame *is* the UX evidence. - **Compare pickers for consistency.** `setup-snapshot --nav-down 3` captures the harness menu as you move; eyeball marker/footer/highlight drift against the theme and resume pickers (different engines render differently). ## Covering all critical user journeys This skill owns the **setup / onboarding / first-run / TUI** journeys. The repo already has complementary CUJ coverage — use both: - **Live setup/UX journeys → this skill's scenarios** (cold-start, setup, pickers, help, REPL discoverability). - **Deeper end-to-end journeys → `tests/e2e/test_journey_*.py`** (first session to code, resume/disconnect, fork/explore, file upload, collaboration, …). Run a slice with the project's gated runner, e.g. `uv run --frozen --group test python -m pytest tests/e2e/test_journey_first_session_to_code.py -q`. - **Reusable PTY helpers** live in `tests/e2e/omnigent/_pexpect_harness.py` (`spawn_omnigent_run`, `wait_for_ready`, `submit_prompt`, `await_turn_complete`, `clean_exit`) and the snapshot comparator in `tests/e2e/omnigent/_snapshot.py` — prefer extending those over re-inventing. To drive a surface this skill doesn't script yet, spawn it by hand with the sandbox env and the keys the driver exports (`KEY_UP`/`KEY_DOWN`/`KEY_ENTER`/ `KEY_ESC`), then `drain()` and `save_frame()` the result. ## Teardown — non-negotiable - The driver force-kills the PTY child and its descendants, and runs `omnigent server stop` against the sandbox to reap any spawned background server. After a run, confirm nothing leaked: `pgrep -af "omnigent.*(server|runner|host._daemon)"` — anything bound to your sandbox's data dir is yours to kill. - The sandbox temp dir is deleted unless `--keep-sandbox`. If you keep one for inspection, `rm -rf` it when done. - Always drive the CLI through the driver (which redirects `HOME` + the config/data knobs), never a bare `omnigent setup` — that would write to the real `~/.omnigent`. If you pass `--inherit-home`, expect `cli-*.log` writes to the real `~/.omnigent/logs` and a `real_config_untouched: false` — that's the guard working, not a bug. ## Honesty If you can't reach the surface under test (no harness, no credential, headless limit), the scenario must report `skipped` — **do not claim a CUJ passed**. The strongest evidence for a fix is a reproduced baseline (`before`) plus the flipped `after`; report both `SUMMARY` lines, not a summary of a summary. ## Adding a scenario Write `scenario_<name>(args, sandbox, result)` in `verify_cli.py`: drive the CLI (reuse `pexpect.spawn(... env=sandbox.env, dimensions=(args.rows, args.cols))`, `drain()`, `save_frame()`, the `KEY_*` constants), record findings with `result.add(name, ok, detail)` (fails the run) or `result.notes.append(...)` (informational, for before/after flips), register it in `SCENARIOS`, and add a row to the catalog above. Keep one assertion per real, observable behavior so a fix is provable as a single check flip. ## Code under test - First-run dispatch / no-arg routing: `omnigent/cli.py` (`run`, the first-run plan, `_run_configure_harnesses_interactive`). - Onboarding: `omnigent/onboarding/*` (`setup.py`, `interactive.py`, `configure_models.py`, `provider_selection.py`, `detected.py`). - TUI / REPL & pickers: `omnigent/repl/*` (`_repl.py`, `_theme_picker.py`, `_resume_picker.py`), `omnigent/_terminal_picker_theme.py`. - Installer: `scripts/install_oss.sh`.