eval-skills
Behavioral evaluation of skills using fixture-based test cases. Runs eval.json fixtures per skill to verify correct output, exit codes, latency budgets, and compose chain pipelines. Produces transparent Markdown and JSON reports with diffs for failures. Complements sanity.sh (smoke tests) and skills-ci (static analysis) with behavioral correctness.
ソース情報
- リポジトリ
- grahama1970/agent-stack-public
- ソースの最終更新活動
- 2026年9月24日 15:51
- 検出された SKILL.md の言語
- 英語
- スター
- 0
- フォーク
- 0
インストール方法
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
ソースファイルを確認
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
ファイルエクスプローラー
6 ファイルSKILL.md を表示中
SKILL.md
ソースの指示 · 読み取り専用プレビュー- name
- eval-skills
- description
- Behavioral evaluation of skills using fixture-based test cases. Runs eval.json fixtures per skill to verify correct output, exit codes, latency budgets, and compose chain pipelines. Produces transparent Markdown and JSON reports with diffs for failures. Complements sanity.sh (smoke tests) and skills-ci (static analysis) with behavioral correctness.
- internal
- true
- triggers
- ["evaluate skill output","behavioral test skills","eval skills","test skill correctness","check skill output","run skill fixtures","skill regression test"]
- provides
- ["skill-evaluation","behavioral-testing"]
- composes
- []
- metadata
- {"short-description":"Behavioral skill evaluation via fixtures"}
- taxonomy
- ["precision","resilience"]
- disciplines
- ["evaluation-quality"]
# eval-skills
Behavioral evaluation for Embry OS skills. Fills the gap between `sanity.sh` ("does it run?") and `skills-ci` ("does it follow rules?") by answering **"does it produce correct output?"**.
## Usage
```bash
# Evaluate all skills that have fixtures/eval.json
./run.sh eval
# Specific skills
./run.sh eval --skill taxonomy,normalize
# Filter by tag
./run.sh eval --tags fast
# Output reports
./run.sh eval --report-json /tmp/eval.json
./run.sh eval --report-md /tmp/eval.md
```
## Per-Skill Opt-In
Skills opt in by adding `fixtures/eval.json`:
```
my-skill/
└── fixtures/
├── eval.json # Test declarations
├── inputs/sample.txt # Test inputs (optional)
└── expected/sample.json # Golden outputs (optional)
```
## eval.json Schema
```json
{
"version": 1,
"skill": "my-skill",
"defaults": { "latency_budget_ms": 5000 },
"cases": [
{
"name": "basic-test",
"command": ["subcommand", "--flag"],
"input_inline": "test input",
"expected_exit_code": 0,
"expected_stdout_contains": ["expected_string"],
"latency_budget_ms": 3000,
"tags": ["fast"]
}
]
}
```
## Composition
| Consumer | How |
|----------|-----|
| `skills-ci apply` | Runs after fixes to verify behavioral correctness |
| `monitor-skill-health` | Nightly behavioral audit |
| Human | `/eval-skills eval --skill X` |
GitHubで見る