skill-eval

name	skill-eval
description	Use to orchestrate and evaluate a multi-agent TDD skill benchmark. Trigger when the user says start eval, wants junior/mid/senior coder agents on separate model tiers, parallel git worktrees, adversarial evaluation of result bundles, deterministic coverage/mutation scoring, or a consolidated parity report.

Skill Eval

Purpose

Evaluate whether the same TDD skill lets lower-capability coder agents perform comparably to higher-capability coder agents.

This skill is separate from tdd and eval-result. Coder agents use tdd to implement and eval-result to package evidence; the evaluator uses this skill to orchestrate and score them.

Agent Files

Use these repo-level agent definitions as the source of truth:

• .codex/agents/junior-coder.yaml: lower-capability coder, model gpt-5.4-mini with low reasoning effort, workspace worktrees/junior
• .codex/agents/mid-coder.yaml: coding-optimized mid coder, model gpt-5.3-codex, workspace worktrees/mid
• .codex/agents/senior-coder.yaml: senior baseline coder, model gpt-5.5, workspace worktrees/senior
• .codex/agents/tdd-evaluator.yaml: adversarial evaluator, model gpt-5.5

Required Inputs

Before running eval, require:

• a root-level prompt file produced by env-setup or equivalent task context
• .codex/tdd-setup.json or equivalent setup metadata
• a clean baseline commit or workspace snapshot
• deterministic test/coverage/test-strength commands or recorded fallbacks

If the prompt file or setup metadata is missing, run or request env-setup first.

Start Eval Protocol

When the user says start eval:

1. Read prompt, .codex/tdd-setup.json, and the four .codex/agents/*.yaml files.
2. Create three isolated git worktrees from the same baseline:
   • worktrees/junior
   • worktrees/mid
   • worktrees/senior
3. Spawn the three coder agents in parallel using their agent files.
4. Give each coder identical task context, setup metadata, commands, and acceptance criteria.
5. Require each coder to return the eval-result bundle.
6. After all coders finish, inspect each worktree directly.
7. Run deterministic verification commands from setup metadata where available.
8. Score each result using references/rubric.md.
9. Produce the consolidated report using references/report-format.md.

If git worktrees are unavailable, use isolated directories copied from the same baseline and record the fallback.

Generated worktree code is run evidence only. Do not preserve it by default.

Adversarial Evaluation Rules

Do not trust coder self-report. Score only from artifacts:

• final files
• tests
• command outputs
• coverage reports
• mutation reports or mutation-style proxy checks
• diffs against baseline
• chronological logs/checkpoints where available

Penalize:

• production code before failing test evidence
• missing RED evidence
• tests that pass trivially or do not assert behavior
• broad rewrites unrelated to the prompt
• claims unsupported by worktree artifacts

Show Report Protocol

When the user says show report:

1. Re-read the latest worktree artifacts and result bundles.
2. Re-run deterministic commands if needed and safe.
3. Recompute weighted scores.
4. Compare junior and mid aggregates against senior using the +/-5 percentage-point parity band.
5. Return the consolidated report.

Tune Skill Protocol

When the user wants to improve the skill after a report:

1. Use the latest report to identify the most likely general weakness in tdd.
2. Propose one focused edit to tdd; do not overfit to the benchmark solution.
3. Do not edit the skill unless the user asks to apply the proposed change.
4. After accepted edits, commit or otherwise snapshot only reusable skill changes.
5. Treat worktrees/, prompt, .codex/tdd-setup.json, reports, coverage output, mutation output, and result bundles as disposable run state.
6. Rerun env-setup and skill-eval from a clean baseline for the next iteration.

References

• references/rubric.md: weights, scoring rules, fallbacks, parity calculation.
• references/report-format.md: output format.
• references/subagent-prompts.md: exact prompts derived from .codex/agents/*.yaml.