메뉴
Use when a user wants to compare an arbitrary skill or path across models, quality, cost, latency, or routing.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Analyze actual local Copilot skill activations without reading conversation content.
Plan work too big for one agent session as a shared map of decision tickets on GitHub Issues or Jira, then resolve them one at a time until the route is clear. Use with /wayfinder when the user says "plan something too big for one session", "chart a map", "wayfind", "work the map", or "decision tickets". Always confirm tracker (GitHub vs Jira) and destination before writing. Adapted from mattpocock/skills wayfinder (MIT).
Create or resume a task handoff so a fresh agent can continue unfinished work. Persist via omp handoff CLI (never write handoff files yourself). Use when the user says /handoff, wants to pause mid-task, or resume prior work.
Single-owner execute-fix-verify loop for one clear task. Use with /ralph when one agent should keep going until evidence or blocker.
Adversarial QA pass that tests behavior, failures, and regressions. Use with /ultraqa after implementation when shallow checks are not enough.
Capture user-correction patterns from this session and, when a pattern recurs, draft a new project skill. Use at end of session, when the user signals wrap-up, or when invoked directly.
| name | skill-bench |
| description | Use when a user wants to compare an arbitrary skill or path across models, quality, cost, latency, or routing. |
Use the active omp installation from the user's current project. This skill is the conversational
pair-programmer; omp skill-bench owns durable state, execution, evidence, and reports.
Keep the interaction as a durable, resumable pair-design rather than a one-shot JSON command.
It supports an arbitrary skill name, installed skill identity, or filesystem path. History can rank candidates but never restricts eligibility; the selected arbitrary skill remains user-approved. There is no fixed skill-to-task mapping or model allowlist.
History guidance comes from Copilot CLI session history under ~/.copilot/session-state; OMP is only
the parser and orchestrator. Never describe this source as OMP history or report Copilot history as
unavailable merely because a wrapper command is absent.
Do not expose Python. Do not require a package checkout. Do not depend on another skill. Do not start live benchmark cells until there is an approved frozen design, a hard budget, and explicit spend confirmation. Synthetic execution only for explicit synthetic/dry-run smoke requests and never as a fallback for a non-synthetic spec. Freeze earlier if a non-synthetic reviewed manifest is missing the approved Copilot provider, a frozen evaluator, an explicit non-empty execution.allowlistedTools list, explicit hard ceilings for maxUsd, maxCells, maxRuntimeMs, and maxPremiumRequests, or conservative per-cell estimates. On refusal, ambiguity, missing approval, failed history, unavailable runner, or failed freeze, stop without starting live benchmark cells and explain the next safe action.
When the session pauses mid-design, mid-run, or after an incomplete/partial report, end with a short Continue later card (skill message only; do not invent a second system):
Phase: design | freeze | spend-approval | running | report
Ids/paths: draft/spec/run + absolute report path when present
Done: …
Pending: exactly one next question or command
Blockers: budget | approval | crash | missing report | …
Safe next: one omp command or one user decision
If cell evidence exists but the HTML report is missing, run
omp skill-bench report RUN_ID first — it may rebuild a partial salvaged report without
new provider spend.
Resolve one entry mode:
/skill-bench: guided discovery with 30d/all defaults;/skill-bench WINDOW SCOPE: guided discovery with adjusted filters;/skill-bench SKILL_OR_PATH: direct design for that exact identity or path.Run exactly one normalized omp skill-bench ... --json command for the resolved entry mode. Do not
print or paste raw JSON. Summarize it for the user, preserve warnings and paths, and continue in the
same skill invocation; returning a draft JSON object is not completion.
For guided output, show the ranked candidates with identity, source, and path. Ask exactly one unresolved high-impact question: which identity to test. After the answer, rerun direct mode with the selected identity or path so guided and direct entry converge on one durable draft. Duplicate names always require explicit path/identity selection.
Move one decision at a time. After each answer, update the reviewed manifest checkpoint and show the changed section. For every unresolved decision, briefly state the current understanding, ask exactly one high-impact question, provide evidence-based choices with a recommended answer, and explain what that answer unlocks. Use free-form input only when the user selects a custom/other choice. Cover, in order:
Show the manifest diff before any gate approval. Before import/approval/freeze, ensure the reviewed
manifest includes provider {kind:"copilot", approved:true}, a real hashed evaluator descriptor/command,
an explicit non-empty execution.allowlistedTools list, hard ceilings for budgets.maxUsd, budgets.maxCells,
budgets.maxRuntimeMs, and budgets.maxPremiumRequests, and budgets.estimatedCellUsd plus
budgets.estimatedCellPremiumRequests. Import it, then request and record each explicit gate approval through omp skill-bench resume ... --approve .... A prose-only yes, history rank, or saved draft is not permission to freeze or spend. Freeze only after every current hash-bound gate is approved.
Defaults are recommendations, never restrictions. Show candidate provenance and that enumeration is
incomplete. Model selection must use one structured choice question, not a free-form request. Build
2–3 mutually exclusive model-set options from the returned candidates and provenance. Put the
recommended option first and append (Recommended) to its label. Each option must list exact model
IDs and briefly explain its quality, provider-diversity, cost, speed, and availability trade-offs.
Prefer adaptive Balanced, Quality-first, and Economy-first presets when the evidence supports them;
never hardcode model IDs. Do not ask the user to invent or type model IDs unless they select the
custom/other choice. If the interactive question tool is unavailable, render the same choices as
numbered options and ask for one number or label.
Model probes can consume provider requests, so ask before probing. If approved, probe only the
explicit model ids by rerunning direct design with each --model ID plus --probe-models. Preserve
available, unavailable, and unknown; unknown remains selectable, while unavailable is reported
rather than silently replaced.
After freeze, ask separately whether to spend the displayed hard ceilings. Only after an affirmative
answer, run with --approve-spend; this records hash-bound spend approval for that frozen semantic
spec. Run pilot first when calibration is needed. Run validated mode only from a frozen spec and
never broaden its budget.
Present quality and proof gaps first, then a per-task decision table with state, cheapest passing,
highest quality, and fastest passing. Task summaries must include tested model/arm, quality, USD
cost, total tokens, and latency. Every recommendation and final summary row that shows USD must
show total token spend beside it; never present price alone. If token telemetry is unavailable,
write tokens unknown rather than omitting the field. Keep missing token/cost telemetry visibly
unknown only after both supported cost sources are unavailable. Return the absolute HTML report
path after report generation or refresh, and open it only when the user requested that.
When Copilot emits totalNanoAiu, use it as direct AI-credit telemetry and convert it to USD
(1 AI credit = $0.01). Otherwise fetch the official GitHub Copilot pricing table once and save the
immutable snapshot as pricing.json. Label website-derived USD as a public-price proxy, not a
GitHub Copilot invoice; never guess a tier when aggregate session telemetry cannot select one.
omp skill-bench report RUN_ID can backfill missing proxy costs without another model call.
For totals, use the provider total when available; otherwise input + output. Cache-read tokens are
already included in input and must not be added again. Always show this total provenance.
For routing, run omp skill-bench apply RUN_ID --dry-run first. Show capability, staleness, scope,
and every conflict. Apply only after explicit approval; Copilot interactive instructions remain
advisory and require a new session.
For publication, preview the export without --approve, show every included file, hash, and
redaction, then repeat with --approve only after the user approves that exact preview.
omp skill-bench [<skill-or-path>] [filters] [--model <id>] [--probe-models] [--json]
omp skill-bench resume [<draft-id>] [--import <manifest.json>|--approve <gate>|--freeze]
omp skill-bench run <spec-id-or-path> --pilot|--validated --approve-spend
omp skill-bench report <run-id> [--open|--no-open]
omp skill-bench rerun <run-id>
omp skill-bench apply <run-id> [--scope project|user] [--dry-run]
omp skill-bench export <spec-id-or-run-id> --output <path> [--approve]