Menu
Style guide for writing behavioral eval criteria that produce consistent LLM-judged scores. Use when creating, reviewing, or modifying eval definitions.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Dispatch Codex to review all unreviewed repository work, then verify its correlated findings ticket or clean result. Use when requesting an independent Codex review, auditing all changes, checking work since the last review, or waiting for Codex findings. Trigger: dispatch codex review, independent review, review all changes, check codex findings, run review agent.
Always-on: when a project has tickets, work the frontier (first unblocked open ticket), mark done, update plan status.
Incremental and full-history repository review with resumable adoption batches. Use when reviewing new work, initializing review tracking, reviewing all work to date, resuming historical review, inspecting unreviewed commits, or reviewing changed tickets. Trigger: review new work, adopt repository, review all history, resume review, review marker, unreviewed commits.
End-of-session handoff that captures current state for the next session. Use when ending a work session, switching context, or before a long break.
Consolidate project artifacts — promote scratch to memory, deduplicate memory, process decisions, organize scripts, update steering/skills accuracy. Use periodically or when the project feels cluttered.
Run and interpret crew-research evals — run.sh flags, scores.jsonl fields, activation TPR/FPR verdicts, resuming interrupted runs. Use when running evals, reading eval results, diagnosing a failing definition, or resuming a dead run. Trigger: run the evals, eval results, scores.jsonl, activation test, TPR, delta threshold, resume the run, skip-completed, known gap.
| name | eval-criteria |
| description | Style guide for writing behavioral eval criteria that produce consistent LLM-judged scores. Use when creating, reviewing, or modifying eval definitions. |
| metadata | {"type":"reference","invocation":"both","practice":null} |
- name: agent-verb-noun
input: "realistic user message"
criteria: |
PRIMARY: ...
AUTOMATIC FAIL: ...
tags: [category]
threshold: 4
PRIMARY: The ONE thing being tested. One sentence.
AUTOMATIC FAIL (score 1): Condition that means instant failure.
Score 3: What partial credit looks like.
Score 4: What "good" looks like.
BONUS (score 5): What excellence looks like.
| ❌ Bad | ✅ Good |
|---|---|
| "Should handle correctly" | "PRIMARY: Reads the file before proposing changes" |
| "Should not do bad things" | "AUTOMATIC FAIL: Implements feature (scope is bugs only)" |
| Testing routing + style in one eval | Separate eval per concern |
| No ideal on routing eval | Ideal showing correct delegation |
| Scoring vocabulary over reasoning | Score correct resolution regardless of terminology used |
Score reasoning quality, not vocabulary. An agent that correctly resolves a source conflict using natural language ("official docs beat community answers because the maintainers control the source") should score the same as one using formal framework terms ("L4 > L6"). The framework makes reasoning visible; the reasoning is what matters.
Negative tasks (expect_activation: false) must stay OUT of the skill's legitimate downstream territory, not just its trigger vocabulary. A task that instructs the agent to produce changes can legitimately end in commit territory — loading git-protocol there is correct behavior, and the "negative" flakes at the FPR gate. Use read-only/Q&A negatives for skills whose triggers include post-change workflow (incident: activation-git-protocol, ticket 27 — two verified-genuine FPs from change-producing negatives).
{skill}-{verb}-{noun} — e.g., planning-cycles-produces-phases, code-review-checks-security
threshold: 4 — wrong answer = broken system (routing, scope, safety)threshold: 3 — partial credit acceptable (identity, style, narration)