blender-agent-benchmark

name	blender-agent-benchmark
description	Benchmark Blender modeling agents, skills, prompts, scripts, or MCP tools with paired isolated runs. Use for baseline-versus-plugin comparisons, regression suites, skill forward-testing, MCP usefulness evaluation, score calibration, or claims that a Blender workflow improves mesh, visual, structural, animation, cost, or completion quality.

Blender Agent Benchmark

Measure changes with the same tasks, model, effort, limits, Blender build, and evaluator. Preserve natural agent behavior.

Protect benchmark integrity

1. Create isolated directories for every condition and repetition.
2. Do not leave the other condition's code, renders, metrics, or expected fixes where the agent can discover them.
3. Keep the user-facing task prompt identical except for explicit skill invocation in the plugin condition.
4. Use codex exec --ignore-user-config for the no-plugin baseline.
5. Use the installed plugin in a fresh invocation for the plugin condition.
6. Record CLI version, model, effort, Blender build, duration, tool calls, failures, and output hashes.
7. Evaluate outputs after generation. Do not leak hidden rubric details to the agent.

Read references/methodology.md before changing fixtures, scoring, or comparison claims. Read references/validated-results.md only when reviewing the plugin's recorded validation result, not while generating a benchmark submission.

Run the suites

Use scripts/run_benchmark.ts:

bun "<skill-root>\scripts\run_benchmark.ts" `
  --suite quick `
  --mode baseline `
  --output "<run-root>\baseline"

bun "<skill-root>\scripts\run_benchmark.ts" `
  --suite quick `
  --mode plugin `
  --output "<run-root>\plugin"

Start with a smoke task to validate the harness. Use at least three representative tasks and repeated runs before claiming a general capability gain.

Use scripts/compare_runs.ts for randomized blinded multiview judging. Use scripts/rescore_run.ts to recompute deterministic scores after a scorer change without rerunning agents.

Noninteractive Codex cancels MCP tool calls that require approval. If testing an MCP condition, pass --bypass-approvals to every compared condition and use isolated benchmark directories. Do not give only the MCP condition broader permissions.

Use scripts/benchmark_mcp.ts --asset <path> --output <new-dir> to verify that the MCP transport returns the same deterministic metrics as direct CLI evaluation. Equivalent results prove transport correctness, not a modeling quality gain.

Compare conditions

Keep these dimensions separate:

• execution/export validity;
• prompt and structural compliance;
• geometry/game-readiness;
• multiview visual quality;
• physical plausibility;
• animation quality when applicable;
• context/export correctness;
• time, turns, tool failures, and cost.

Use hard gates before the weighted score. Prefer blinded pairwise visual review over uncalibrated absolute aesthetic scores.

Iterate

1. Inspect failed metrics, renders, videos, and traces.
2. Identify one transferable workflow defect.
3. Change the smallest relevant skill, deterministic tool, or MCP surface.
4. Rerun the same slice.
5. Run a holdout task before retaining the change.

Do not retain benchmark-specific instructions that reveal fixture answers or damage ordinary modeling behavior.