| name | skill-bench |
| description | Use when a user wants to compare an arbitrary skill or path across models, quality, cost, latency, or routing. |
Skill Bench
Use the active omp installation from the user's current project. This skill is the conversational
pair-programmer; omp skill-bench owns durable state, execution, evidence, and reports.
Keep the interaction as a durable, resumable pair-design rather than a one-shot JSON command.
It supports an arbitrary skill name, installed skill identity, or filesystem path. History can rank
candidates but never restricts eligibility; the selected arbitrary skill remains user-approved.
There is no fixed skill-to-task mapping or model allowlist.
History guidance comes from Copilot CLI session history under ~/.copilot/session-state; OMP is only
the parser and orchestrator. Never describe this source as OMP history or report Copilot history as
unavailable merely because a wrapper command is absent.
Do not expose Python. Do not require a package checkout. Do not depend on another skill.
Do not start live benchmark cells until there is an approved frozen design, a hard budget, and
explicit spend confirmation. Synthetic execution only for explicit synthetic/dry-run smoke requests
and never as a fallback for a non-synthetic spec. Freeze earlier if a non-synthetic reviewed manifest is missing the approved Copilot provider, a frozen evaluator, an explicit non-empty execution.allowlistedTools list, explicit hard ceilings for maxUsd, maxCells, maxRuntimeMs, and maxPremiumRequests, or conservative per-cell estimates.
On refusal, ambiguity, missing approval, failed history, unavailable runner, or
failed freeze, stop without starting live benchmark cells and explain the next safe action.
When the session pauses mid-design, mid-run, or after an incomplete/partial report, end
with a short Continue later card (skill message only; do not invent a second system):
Phase: design | freeze | spend-approval | running | report
Ids/paths: draft/spec/run + absolute report path when present
Done: …
Pending: exactly one next question or command
Blockers: budget | approval | crash | missing report | …
Safe next: one omp command or one user decision
If cell evidence exists but the HTML report is missing, run
omp skill-bench report RUN_ID first — it may rebuild a partial salvaged report without
new provider spend.
Start and continue
Resolve one entry mode:
- bare
/skill-bench: guided discovery with 30d/all defaults;
/skill-bench WINDOW SCOPE: guided discovery with adjusted filters;/skill-bench SKILL_OR_PATH: direct design for that exact identity or path.
Run exactly one normalized omp skill-bench ... --json command for the resolved entry mode. Do not
print or paste raw JSON. Summarize it for the user, preserve warnings and paths, and continue in the
same skill invocation; returning a draft JSON object is not completion.
For guided output, show the ranked candidates with identity, source, and path. Ask exactly one
unresolved high-impact question: which identity to test. After the answer, rerun direct mode with
the selected identity or path so guided and direct entry converge on one durable draft. Duplicate
names always require explicit path/identity selection.
Pair-design loop
Move one decision at a time. After each answer, update the reviewed manifest checkpoint and show the
changed section. For every unresolved decision, briefly state the current understanding, ask exactly
one high-impact question, provide evidence-based choices with a recommended answer, and explain what
that answer unlocks. Use free-form input only when the user selects a custom/other choice. Cover, in
order:
- selected skill and candidate models;
- atomic scenarios and action contracts: detect/report, propose, implement/verify, or plan-only;
- user reference or generator-plus-independent-critic reference, including expected, acceptable,
forbidden, found/done, missed, false-positive, incorrect, and proof fields;
- rubric, hard gates, thresholds, disqualifiers, calibration, judge, and human adjudication;
- matched baseline and skill arms, optional prompt arm, execution profile, and parity;
- sampling/confidence plus hard USD, premium-request, runtime, and cell ceilings.
Show the manifest diff before any gate approval. Before import/approval/freeze, ensure the reviewed
manifest includes provider {kind:"copilot", approved:true}, a real hashed evaluator descriptor/command,
an explicit non-empty execution.allowlistedTools list, hard ceilings for budgets.maxUsd, budgets.maxCells,
budgets.maxRuntimeMs, and budgets.maxPremiumRequests, and budgets.estimatedCellUsd plus
budgets.estimatedCellPremiumRequests. Import it, then request and record each explicit gate approval through omp skill-bench resume ... --approve .... A prose-only yes, history rank, or saved draft is not permission to freeze or spend. Freeze only after every current hash-bound gate is approved.
Models and spend
Defaults are recommendations, never restrictions. Show candidate provenance and that enumeration is
incomplete. Model selection must use one structured choice question, not a free-form request. Build
2–3 mutually exclusive model-set options from the returned candidates and provenance. Put the
recommended option first and append (Recommended) to its label. Each option must list exact model
IDs and briefly explain its quality, provider-diversity, cost, speed, and availability trade-offs.
Prefer adaptive Balanced, Quality-first, and Economy-first presets when the evidence supports them;
never hardcode model IDs. Do not ask the user to invent or type model IDs unless they select the
custom/other choice. If the interactive question tool is unavailable, render the same choices as
numbered options and ask for one number or label.
Model probes can consume provider requests, so ask before probing. If approved, probe only the
explicit model ids by rerunning direct design with each --model ID plus --probe-models. Preserve
available, unavailable, and unknown; unknown remains selectable, while unavailable is reported
rather than silently replaced.
After freeze, ask separately whether to spend the displayed hard ceilings. Only after an affirmative
answer, run with --approve-spend; this records hash-bound spend approval for that frozen semantic
spec. Run pilot first when calibration is needed. Run validated mode only from a frozen spec and
never broaden its budget.
Results, routing, and export
Present quality and proof gaps first, then a per-task decision table with state, cheapest passing,
highest quality, and fastest passing. Task summaries must include tested model/arm, quality, USD
cost, total tokens, and latency. Every recommendation and final summary row that shows USD must
show total token spend beside it; never present price alone. If token telemetry is unavailable,
write tokens unknown rather than omitting the field. Keep missing token/cost telemetry visibly
unknown only after both supported cost sources are unavailable. Return the absolute HTML report
path after report generation or refresh, and open it only when the user requested that.
When Copilot emits totalNanoAiu, use it as direct AI-credit telemetry and convert it to USD
(1 AI credit = $0.01). Otherwise fetch the official GitHub Copilot pricing table once and save the
immutable snapshot as pricing.json. Label website-derived USD as a public-price proxy, not a
GitHub Copilot invoice; never guess a tier when aggregate session telemetry cannot select one.
omp skill-bench report RUN_ID can backfill missing proxy costs without another model call.
For totals, use the provider total when available; otherwise input + output. Cache-read tokens are
already included in input and must not be added again. Always show this total provenance.
For routing, run omp skill-bench apply RUN_ID --dry-run first. Show capability, staleness, scope,
and every conflict. Apply only after explicit approval; Copilot interactive instructions remain
advisory and require a new session.
For publication, preview the export without --approve, show every included file, hash, and
redaction, then repeat with --approve only after the user approves that exact preview.
OMP commands
omp skill-bench [<skill-or-path>] [filters] [--model <id>] [--probe-models] [--json]
omp skill-bench resume [<draft-id>] [--import <manifest.json>|--approve <gate>|--freeze]
omp skill-bench run <spec-id-or-path> --pilot|--validated --approve-spend
omp skill-bench report <run-id> [--open|--no-open]
omp skill-bench rerun <run-id>
omp skill-bench apply <run-id> [--scope project|user] [--dry-run]
omp skill-bench export <spec-id-or-run-id> --output <path> [--approve]
