aec-bench

Create a new benchmark task seed from an expert description. Use when the user wants to add a new task, describe a task idea, or create a seed file for the generation framework.

Build an experiment.yaml config interactively. Discovers available tasks, guides agent and model selection, validates the manifest, and previews the trial plan with a dry run. Use when the user wants to set up, configure, or plan a benchmark run.

Guide an agent through creating or revising a task-world harness, running it against a baseline, reviewing evidence, and preparing a comparison. Use when the user wants to create a harness from prose, compare a candidate harness to an existing one, or investigate verifier/schema gaps.

Create a versioned benchmark dataset interactively. Discovers templates and tasks, guides selection and configuration, generates instances if needed, freezes into an immutable manifest, and verifies integrity. Use when the user wants to create, build, or publish a dataset.

Create a new task generation template from a source_task.json seed file. Use when the user wants to add a new benchmark template, convert a seed task into a template, or asks to create/build a template.

Review and harden a task template or task instance for correctness, robustness, and benchmark readiness. Use when a template or task has been created and needs quality-gating before use in real benchmarks. Also trigger when the user asks to "harden", "review a template", "check a task", or "validate before benchmarking".

Verify that code changes respect aec-bench architectural invariants and domain boundaries. Use when modifying files under src/aec_bench/, adding new modules, changing contracts or data shapes, working across multiple domains, or before committing changes. Also trigger when the user asks to "check domains", "verify architecture", "validate invariants", or "review boundaries".