agent-eval
Use when installing or using @primer/agent-eval, designing agent evaluations, creating benchmarks, experiments, scenarios, checks, judges, or treatments, and running, planning, sharding, or interpreting evaluations with the agent-eval CLI.
来源信息
- 仓库
- primer/agent-eval
- 最近来源活动
- 2026年10月8日 22:14
- 检测到的 SKILL.md 语言
- 英语
- 星标
- 1
- 分支
- 0
安装方式
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
检查来源文件
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
文件资源管理器
15 个文件正在显示 SKILL.md
SKILL.md
来源说明 · 只读预览- name
- agent-eval
- description
- Use when installing or using @primer/agent-eval, designing agent evaluations, creating benchmarks, experiments, scenarios, checks, judges, or treatments, and running, planning, sharding, or interpreting evaluations with the agent-eval CLI.
# agent-eval
Use agent-eval to measure how agents perform on real tasks and whether a change
to their environment improves the result.
## Choose a task
Read only the references needed for the current task. The overview is context,
not a prerequisite for every operation.
| Task | Read |
| :------------------------------------------------------ | :--------------------------------------------------------------------------------- |
| Install and complete a first evaluation | [Getting started](references/getting-started.md) |
| Understand project structure and evaluation methodology | [Overview](references/overview.md) |
| Establish a capability baseline | [Benchmarks](references/benchmarks.md), [capabilities](references/capabilities.md) |
| Compare instructions, skills, agents, MCP, or plugins | [Experiments](references/experiments.md), [treatments](references/treatments.md) |
| Create a task, starting workspace, or custom image | [Scenarios](references/scenarios.md) |
| Add deterministic grading or a subjective rubric | [Checks](references/checks.md) or [judges](references/judges.md) |
| Select models, effort, or execution backends | [Models and runners](references/models-and-runners.md) |
| Install resources or run commands in a trial | [Sandbox](references/sandbox.md) |
| Inspect a run matrix, shard, or merge | [Plans](references/plans.md) |
| Read results or diagnose agent behavior | [Trials and results](references/trials-and-results.md) |
| Find a command, flag, or runtime failure | [CLI](references/cli.md) |
## Execution rules
- For a first comparison, use one scenario, one model/effort, and one treatment
beyond the automatic control.
- Keep prompts and fixtures treatment-blind. Put the knowledge being tested in
treatment setup, not shared setup.
- Withhold grader files using `files`. Verify the starter fails for the intended
reasons and a correct solution passes; restore the starter before execution.
- Inspect the plan's combinations before running. Use a fresh output directory.
- Finish by reading trial results and reporting outcomes and errors separately.
Command completion alone does not establish quality.
- Interpret session AI credits using [Trials and results](references/trials-and-results.md);
missing usage is not the same as zero cost.
These references describe the source API at authoring time. For an installed
version, verify its package types and `npx agent-eval <command> --help`
before using an unfamiliar option. Do not substitute legacy CLI flags.
在 GitHub 查看