菜单
Measure a skill's reliability — run it k times for a pass@k score, design or interpret its eval, or compare it against the base agent. Use when the user wants to run, design, or interpret a skill's eval, or write an .eval.yaml spec.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Build and harden a skill with evals — interview to design its eval tasks, then run, measure, and iterate. Use when the user wants to create or improve a skill's eval, or run the create → test → improve loop for a skill.
Save a short note to a file. Use when asked to record, save, or jot down a note.
Send an operational alert. Use when asked to send an alert, page, or notification.
Fetch data with a strict retry budget. Use when asked to fetch data that may be unavailable.
Convert a title into a URL slug. Use when asked to slugify a title or make a URL slug.
Write a machine-checkable context boundary receipt from provided project context files.
| name | evaluate-skill |
| description | Measure a skill's reliability — run it k times for a pass@k score, design or interpret its eval, or compare it against the base agent. Use when the user wants to run, design, or interpret a skill's eval, or write an .eval.yaml spec. |
| allowed-tools | Bash |
Run a skill repeatedly to measure how reliably it works, and design the evals that measure it.
The caliper CLI must be on PATH. This skill can be copied into an agent without the Caliper repo, so do not assume the CLI is packaged with it. Install if missing:
pipx install caliper-eval
The engine (backend + model) is not part of the spec — it is chosen at run time with --model (skill) and --judge-model (judge), independently, from claude-code, codex, pi, defaulting to claude-code. Every backend is a CLI agent that uses its own subscription/auth; there is no direct-API backend (for API billing, configure a CLI with an API key). Full per-backend detail and every command: REFERENCE.md.
An .eval.yaml names the skill and a list of tasks. Keep skill.path relative to the spec file (usually ./SKILL.md):
skill:
path: ./SKILL.md # relative to the spec file
tasks:
- name: What success looks like
prompt: <prompt sent to the skill under test>
expect: <natural-language pass/fail criterion>
assert: | # optional deterministic Python check
assert ...
The spec has no backend/model or judge: block; pick the engine when you run, e.g. caliper run <spec> --model codex --judge-model codex. The full format (setup/cleanup, external assert scripts, sandbox) is in REFERENCE.md.
references/evals/ holds complete real examples (Claude Code smoke, commit workflow, screenshot, summarization, TDD) — each folder self-contained with its fixture SKILL.md and .eval.yaml. references/simple.eval.yaml is one compact multi-task spec.
If the skill has a SKILL.md but no .eval.yaml, suggest the grill-skill workflow — it interviews the user and generates a happy/edge/adversarial spec. Use evaluate-skill directly when a spec already exists and the user wants to run, validate, report, or extend it.
expect: rubrics live in REFERENCE.md — read and apply them when designing tasks.--baseline to confirm the skill beats the raw agent. Debug the spec at --k 1, then measure reliability at --k 3 or higher.Done when: tasks have observable success criteria, at least one deterministic assert:, a positive baseline delta, the spec passes caliper validate, and the user has been prompted to commit the spec.
Running Caliper produces two artifacts: the .eval.yaml spec — the valuable one, commit it beside the skill so anyone who clones the repo can run the same eval — and .caliper/results/ saved run JSONs, useful for diffing over time and safe to gitignore. After creating or running an eval, tell the user to commit the spec alongside SKILL.md.