메뉴
Style guide for writing behavioral eval criteria that produce consistent LLM-judged scores. Use when creating, reviewing, or modifying eval definitions.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Dispatch Codex to review all unreviewed repository work, then verify its correlated findings ticket or clean result. Use when requesting an independent Codex review, auditing all changes, checking work since the last review, or waiting for Codex findings. Trigger: dispatch codex review, independent review, review all changes, check codex findings, run review agent.
Always-on: when a project has tickets, work the frontier (first unblocked open ticket), mark done, update plan status.
Incremental and full-history repository review with resumable adoption batches. Use when reviewing new work, initializing review tracking, reviewing all work to date, resuming historical review, inspecting unreviewed commits, or reviewing changed tickets. Trigger: review new work, adopt repository, review all history, resume review, review marker, unreviewed commits.
End-of-session handoff that captures current state for the next session. Use when ending a work session, switching context, or before a long break.
Consolidate project artifacts — promote scratch to memory, deduplicate memory, process decisions, organize scripts, update steering/skills accuracy. Use periodically or when the project feels cluttered.
Run and interpret crew-research evals — run.sh flags, scores.jsonl fields, activation TPR/FPR verdicts, resuming interrupted runs. Use when running evals, reading eval results, diagnosing a failing definition, or resuming a dead run. Trigger: run the evals, eval results, scores.jsonl, activation test, TPR, delta threshold, resume the run, skip-completed, known gap.
| name | eval-criteria |
| description | Style guide for writing behavioral eval criteria that produce consistent LLM-judged scores. Use when creating, reviewing, or modifying eval definitions. |
| metadata | {"type":"reference","invocation":"both","practice":null} |
- name: agent-verb-noun
input: "realistic user message"
criteria: |
PRIMARY: ...
AUTOMATIC FAIL: ...
tags: [category]
threshold: 4
PRIMARY: The ONE thing being tested. One sentence.
AUTOMATIC FAIL (score 1): Condition that means instant failure.
Score 3: What partial credit looks like.
Score 4: What "good" looks like.
BONUS (score 5): What excellence looks like.
| ❌ Bad | ✅ Good |
|---|---|
| "Should handle correctly" | "PRIMARY: Reads the file before proposing changes" |
| "Should not do bad things" | "AUTOMATIC FAIL: Implements feature (scope is bugs only)" |
| Testing routing + style in one eval | Separate eval per concern |
| No ideal on routing eval | Ideal showing correct delegation |
| Scoring vocabulary over reasoning | Score correct resolution regardless of terminology used |
Score reasoning quality, not vocabulary. An agent that correctly resolves a source conflict using natural language ("official docs beat community answers because the maintainers control the source") should score the same as one using formal framework terms ("L4 > L6"). The framework makes reasoning visible; the reasoning is what matters.
Negative tasks (expect_activation: false) must stay OUT of the skill's legitimate downstream territory, not just its trigger vocabulary. A task that instructs the agent to produce changes can legitimately end in commit territory — loading git-protocol there is correct behavior, and the "negative" flakes at the FPR gate. Use read-only/Q&A negatives for skills whose triggers include post-change workflow (incident: activation-git-protocol, ticket 27 — two verified-genuine FPs from change-producing negatives).
{skill}-{verb}-{noun} — e.g., planning-cycles-produces-phases, code-review-checks-security
threshold: 4 — wrong answer = broken system (routing, scope, safety)threshold: 3 — partial credit acceptable (identity, style, narration)