메뉴
Measure a skill's reliability — run it k times for a pass@k score, design or interpret its eval, or compare it against the base agent. Use when the user wants to run, design, or interpret a skill's eval, or write an .eval.yaml spec.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Build and harden a skill with evals — interview to design its eval tasks, then run, measure, and iterate. Use when the user wants to create or improve a skill's eval, or run the create → test → improve loop for a skill.
Save a short note to a file. Use when asked to record, save, or jot down a note.
Send an operational alert. Use when asked to send an alert, page, or notification.
Fetch data with a strict retry budget. Use when asked to fetch data that may be unavailable.
Convert a title into a URL slug. Use when asked to slugify a title or make a URL slug.
Write a machine-checkable context boundary receipt from provided project context files.
| name | evaluate-skill |
| description | Measure a skill's reliability — run it k times for a pass@k score, design or interpret its eval, or compare it against the base agent. Use when the user wants to run, design, or interpret a skill's eval, or write an .eval.yaml spec. |
| allowed-tools | Bash |
Run a skill repeatedly to measure how reliably it works, and design the evals that measure it.
The caliper CLI must be on PATH. This skill can be copied into an agent without the Caliper repo, so do not assume the CLI is packaged with it. Install if missing:
pipx install caliper-eval
The engine (backend + model) is not part of the spec — it is chosen at run time with --model (skill) and --judge-model (judge), independently, from claude-code, codex, pi, defaulting to claude-code. Every backend is a CLI agent that uses its own subscription/auth; there is no direct-API backend (for API billing, configure a CLI with an API key). Full per-backend detail and every command: REFERENCE.md.
An .eval.yaml names the skill and a list of tasks. Keep skill.path relative to the spec file (usually ./SKILL.md):
skill:
path: ./SKILL.md # relative to the spec file
tasks:
- name: What success looks like
prompt: <prompt sent to the skill under test>
expect: <natural-language pass/fail criterion>
assert: | # optional deterministic Python check
assert ...
The spec has no backend/model or judge: block; pick the engine when you run, e.g. caliper run <spec> --model codex --judge-model codex. The full format (setup/cleanup, external assert scripts, sandbox) is in REFERENCE.md.
references/evals/ holds complete real examples (Claude Code smoke, commit workflow, screenshot, summarization, TDD) — each folder self-contained with its fixture SKILL.md and .eval.yaml. references/simple.eval.yaml is one compact multi-task spec.
If the skill has a SKILL.md but no .eval.yaml, suggest the grill-skill workflow — it interviews the user and generates a happy/edge/adversarial spec. Use evaluate-skill directly when a spec already exists and the user wants to run, validate, report, or extend it.
expect: rubrics live in REFERENCE.md — read and apply them when designing tasks.--baseline to confirm the skill beats the raw agent. Debug the spec at --k 1, then measure reliability at --k 3 or higher.Done when: tasks have observable success criteria, at least one deterministic assert:, a positive baseline delta, the spec passes caliper validate, and the user has been prompted to commit the spec.
Running Caliper produces two artifacts: the .eval.yaml spec — the valuable one, commit it beside the skill so anyone who clones the repo can run the same eval — and .caliper/results/ saved run JSONs, useful for diffing over time and safe to gitignore. After creating or running an eval, tell the user to commit the spec alongside SKILL.md.