| name | eval-corpus |
| description | Sweep the evaluation corpus (synthetic stress shapes by default, curated meshes opt-in) through the placement strategies, produce an aggregate scorecard in ./legolization-eval/, and diff it against the committed baseline to catch regressions or confirm improvements. Use before or after placement or pipeline changes, when asked how the project is doing overall, or to find the worst current case to improve next. |
| license | GPL-3.0-or-later |
Evaluate the corpus and diff the baseline
One model tells you about one shape; the corpus scorecard tells you
whether the project got better or worse. This skill materializes the
corpus, sweeps it through the placement strategies, and diffs the
aggregate scorecard against the committed baseline. Outputs land in
./legolization-eval/runs/.
Requirements
Needs legolization 0.6.0 or newer — check with legolization --version.
If it is missing or older, install the latest stable release with
uv tool install legolization@latest
(or pip install --upgrade legolization). Only if neither package
manager is available, fall back to the standalone installer:
curl -LsSf https://uvx.sh/legolization/install.sh | sh
(PowerShell: irm https://uvx.sh/legolization/install.ps1 | iex).
If installation fails, report the failure and stop.
Conversation contract
- Explain the likely outcome and rough cost first: synthetics take
minutes; meshes take tens of minutes to hours — run those in the
background.
- Ask only when a choice materially changes the result and always
recommend one option; a synthetic-scope sweep is the right default.
- Run the CLI with
--json; present the scorecard verdict, regressions,
and the next useful action.
- Never pass
--write-baseline unless the user explicitly confirms
replacing the committed baseline in this conversation — and never to
silence an unexplained regression. That is the one move that defeats
the whole harness.
Workflow
-
Materialize and check the corpus (idempotent; inputs live in
user-data storage):
legolization corpus list --json
legolization corpus generate --json
legolization corpus download --json
legolization corpus verify --json
Synthetics regenerate locally and are never stale; meshes join the
sweep only after download, and an unavailable mesh is reported as
skipped, not failed.
-
Evaluate everything currently available:
legolization corpus evaluate --json
Scope controls: --kind synthetic|mesh, --strategies NAME,...,
--seeds N,N,..., --jobs 0 (one worker per candidate, CPU-capped).
-
For finer control, split the phases: corpus collect runs
strategies into resumable candidate artifacts (--models,
--traits, --timeout, --fresh to ignore cached successes), then
corpus assemble builds the scorecard and diffs the baseline
(--runs, --baseline, --tolerance).
-
Read the scorecard from the newest run. HARD regressions (a
buildable-strategy count dropped, an expectation newly failing, a
winner objective worsened beyond tolerance) must be explained or
fixed before merging. note: lines (winner identity, brick drift)
are context, not failures — but a winner flip on many models at once
deserves a look. Expected-unbuildable models pin physics verdicts: a
pass there means the pipeline correctly refused them.
-
Drill into the worst model with the other skills: render-ldraw for
the visual field, inspect-instructions when the weakness is in
sequencing. Only after an intentional, explained improvement — and
explicit user confirmation in this conversation — refresh the
baseline with legolization corpus assemble --write-baseline --json
(requires the full-kind, every-strategy, seed-0 scope and a
failure-free evaluation); commit the refreshed baseline afterwards.
Presenting results
The --json envelope (legolization.result/v1) carries status,
exit_code, artifacts, warnings, data, and error on failure.
| Exit | Status | Meaning |
|---|
| 0 | complete | sweep clean; no hard regression against the baseline |
| 1 | error | operational error or incomplete collection — say what is missing |
| 2 | unbuildable | an expectation failed or a HARD regression vs the baseline |
| 130 | interrupted | resumable — rerun to continue collection |
Skipped unavailable meshes surface as note: warnings with exit 0, not
as failures.
