| name | verify-byok |
| description | Verify an omp change end-to-end — static gate (build/tsc/tests/lint/catalog) plus a live BYOK run that drives Copilot/teams on a real model and captures evidence. Use before merging any PR that touches hooks, comms, team, launch, or skills. |
| argument-hint | <branch-or-PR to verify> |
verify-byok — evidence-based verification for omp changes
Invocation: /verify-byok <branch-or-PR>
Prove a change actually works, not just that it compiles. Two gates: a static gate (cheap, always) and a live BYOK gate (drives real Copilot CLI sessions on a Bring-Your-Own-Key model so no GitHub quota is needed). Report real command output, never claims. Be honest about what you did NOT test.
When to use
Before merging any PR — especially ones touching hooks/, scripts/*.mjs, src/comms, src/team, src/copilot/launch, src/copilot/trust, or .github/skills. Anything whose behavior only appears at runtime (tmux submit keys, hook firing, trust dialog, model tool-calls) needs the live gate.
Prerequisites (one-time, persistent)
~/.omp/.env holds BYOK: COPILOT_PROVIDER_BASE_URL=https://openrouter.ai/api/v1, COPILOT_PROVIDER_TYPE=openai, COPILOT_PROVIDER_API_KEY=…, COPILOT_MODEL=…, COPILOT_PROVIDER_MODEL_ID=…. omp auto-loads it; ~/.zshrc sourcing it makes direct copilot BYOK too.- Free model
openai/gpt-oss-120b:free exercises plumbing but botches structured edits/apply_patch and narrates instead of acting — use a capable model (anthropic/claude-sonnet-4.5 + COPILOT_PROVIDER_MODEL_ID=claude-sonnet-4) when verifying that real tasks complete.
- Folder trust: a session only skips the "Do you trust this folder?" dialog if the cwd is in
~/.copilot/config.json#trustedFolders (--yolo does NOT skip it). omp auto-adds it on launch; /private/tmp is pre-trusted.
Static gate (run from the branch worktree)
npm run build
npx tsc -p tsconfig.json --noEmit
npx vitest run
node dist/src/cli.js lint:skills --root .
node dist/src/cli.js catalog validate
A merged-overlap PR? Confirm the test count rose (new tests survived the rebase) and old behavior's tests still pass.
Live BYOK gate
Build the branch CLI first (npm run build); installed omp may be older. Drive Copilot inside tmux (each pane is a pty). Submit with the Enter key name, never C-m — Copilot ≥1.0.61 ignores C-m.
- Launch + reach a model:
tmux new-session -d -s vbyok -c <dir> → send omp (or omp --madmax for bypass) → wait for / commands. Confirm the model line shows your BYOK model and Session: 0 AIC used (0 AI Credits = not on GitHub quota). Send a prompt, send Enter, confirm a real reply.
- Exercise the changed surface — pick the user-visible behavior the change claims:
- hooks → tail
<cwd>/.omp/state/hooks.log for the events firing;
- team → run a 2-worker team, confirm files/artifacts on disk and the script's
🎉 All N agents completed! (read the raw stdout, not the leader's summary);
- cost/minify → pipe a postToolUse payload to
scripts/post-tool-use.mjs and check modifiedResult + raw preserved + ledger savedTokens + omp cost.
- Capture evidence:
tmux capture-pane -p -t <session> plus the on-disk artifact. Keep transcripts.
Reporting (mandatory)
- PASS/FAIL table with the actual captured output (model line, reply, artifact contents, ledger numbers).
- State the model used and that no GitHub quota was consumed.
- Honest limitations: heuristics (e.g. pane-scrape completion detection), model-quality effects (free model botching edits), anything gated/conditional, and anything you did NOT exercise.
- Never report "works" without exercising the user-visible surface — verify before asserting.
Anti-patterns
- Trusting a subagent's "Complete." — re-run the gate yourself.
C-m to submit to Copilot (use Enter).- Concluding from a green build alone — runtime bugs (env propagation, trust dialog, submit keys) pass the build and fail live.
