Menu
How to evaluate implementation against holdout scenarios. Run each scenario and assess whether the system behaves as expected.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Complete reference for the factory CLI — all commands, arguments, options, and usage. Use when an agent needs to invoke factory commands, check correct syntax, or understand available options.
When and how to write a human-action-needed entry to memory/{agent}/needs.md. Agents write these entries when they hit blockers that only a human can resolve.
When you encounter a failure, correction, or new pattern, write a learning and optionally propose a skill.
How the factory workspace is organized and what each directory means.
What to examine during a reflection pass and how to write observations to memory/{agent}/needs.md.
When and how to write scenario holdouts for factory verification.
| name | scenario-evaluation |
| description | How to evaluate implementation against holdout scenarios. Run each scenario and assess whether the system behaves as expected. |
Holdout scenarios in scenarios/{project-name}/ are end-to-end user stories that
Builder never sees. They describe concrete situations: a user wants to do X, they
invoke the system in way Y, the expected outcome is Z.
Scenarios test whether the system works for real user intent, not just whether it satisfies the spec's stated criteria. A system can satisfy every spec criterion and still fail a scenario if the spec missed something.
For each scenario file:
"Pass" requires the output to match the expected outcome, not just to be close. "Partial" means the system does something reasonable but not what the scenario expected. "Fail" means the system does something wrong, crashes, or does nothing.
Run when the scenario involves CLI invocation with specific inputs and is safe to execute. Trace when running requires external setup or tests an unsafe error path. Prefer running — tracing introduces your own reasoning errors.
Write results to scenarios/{project-name}/satisfaction.md with one entry per
scenario: scenario name, Result (PASS/PARTIAL/FAIL), and specific evidence observed.
Aggregate results factor into the satisfaction score.
