spec-ensemble-driver

name

spec-ensemble-driver

description

Drive spec-driven development section-by-section using a multi-model ensemble. For each spec requirement (§), fan the same design prompt out to OpenAI, Gemini, and OpenRouter, score every proposal with the Weighted Ensemble Quality Score (WEQS — six SDD dimensions, 0–100), implement the highest-scoring proposal, present the ranked results, and checkpoint for the user's approval before advancing to the next §. Use this whenever the user wants to work through a spec/PRD/requirements doc with multiple AI models, get an ensemble or "second opinion" from GPT/Gemini/OpenRouter on a spec section, rank competing design proposals by quality, run WEQS (or AQS) scoring on a spec, or "implement the best proposal for this section and move on." Trigger on mentions of ensemble review, multi-model spec work, WEQS/AQS scoring, OpenRouter/OpenAI/Gemini consultation on a spec, or section-by-section spec implementation — even if the user doesn't name the tool explicitly.

Spec Ensemble Driver

Work through a spec one section (§) at a time. For each section, get genuinely different design proposals from three model families, score them on a fixed quality rubric, implement the winner, show your work, and stop for approval before the next section. The value is twofold: diversity (three families disagree in useful ways) and an auditable ranking (WEQS makes "this one is better" a number with evidence, not a vibe).

When this applies

The user is iterating on a spec in specs/ (or any SDD/PRD doc) and wants multiple models to weigh in on how to implement each section, with the best proposal chosen objectively and implemented before moving on. If they just want one model's take, or there's no spec section to anchor on, this skill is overkill — say so and proceed normally.

Prerequisites

The scripts need three API keys: OPENAI_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY. They are loaded from a .env file at the skill root (copy .env.example → .env, chmod 600), so they don't have to live in the shell profile. A real exported env var still overrides .env. Point elsewhere with WEQS_ENV_FILE=/path/to/.env.

The scripts fail loudly if a required key is missing from both .env and the environment — they name the exact variable and stop. If that happens, relay which key to set; do not try to work around it (no fabricated proposals, no silent skips). For testing without spending tokens, pass --mock-dir; see evals/.

Security: .env is chmod 600 and gitignored. Never echo key values, never commit .env, and never paste a key into a response or a log.

Default models (each overridable via env var or CLI flag — --openai-model, --gemini-model, --openrouter-model): gpt-5.1, gemini-3-pro, anthropic/claude-opus-4.1. OpenRouter intentionally runs a different family so the ensemble isn't three near-identical answers.

The loop (one cycle per §)

Run this for the current section, then checkpoint. Do not auto-advance — the user approves each section before the next.

1. Identify the target section

Find the current § the user is working on. Sections are ##/### markdown headings numbered like ## 4. or ### 4.6 or ### 7.2. If the user didn't name one, list the numbered headings and ask which to start from. Work strictly one § per cycle — fine-grained is the point.

2. Fan out to the ensemble

cd ~/.claude/skills/spec-ensemble-driver/scripts
python3 ensemble_review.py \
  --spec <path-to-spec.md> --section "4.6" \
  --out /tmp/ensemble/<section>/proposals.json

This sends the same section-prompt to all three providers and collects their proposals. A single dead provider degrades the ensemble (a warning) rather than killing the run; if all fail, it stops — surface that to the user.

3. Score with WEQS (pointwise first pass)

python3 weqs_score.py \
  --proposals /tmp/ensemble/<section>/proposals.json \
  --out /tmp/ensemble/<section>/scored.json

WEQS is a hybrid scorer (details: references/weqs.md):

• Heuristic (deterministic, free): completeness, testability, specificity.
• LLM-judged: clarity, consistency, feasibility (default judge: openai, override with --judge-provider / --judge-model).

Every dimension carries a score, a rationale, and the evidence behind it. This is a pointwise pass — each proposal judged in isolation. It's a noisy first ranking (pointwise LLM judging under-discriminates, and heuristics can be gamed by keyword density), so it is the input to re-ranking, not the final word.

4. Re-rank and refine (the quality stage)

python3 rerank.py \
  --proposals /tmp/ensemble/<section>/proposals.json \
  --scored   /tmp/ensemble/<section>/scored.json \
  --out      /tmp/ensemble/<section>/reranked.json

This second stage is what lifts spec quality (details: references/reranking.md):

1. Listwise re-rank — one model sees all proposals together with their WEQS breakdowns and ranks them comparatively, recovering the relative signal that pointwise scoring loses.
2. RRF fusion — the pointwise and listwise rankings are fused with Reciprocal Rank Fusion to choose the leader.
3. Critique-revise — one improved candidate is generated from the leader, folding in the losing proposals' best ideas and targeting the leader's weakest WEQS dimensions.
4. WEQS-gated adoption — the revision is re-scored and adopted only if it strictly beats the leader on weighted WEQS and does not regress consistency (the no-conflict guard). Otherwise the leader stands. Quality is therefore monotonic: a worse revision can never win.

Read reranked.json → final_choice. Its adopted field is either "revision" (the refined candidate raised the score) or "leader" (refinement didn't help or was rejected by the guard). The refinement block shows the before/after WEQS and the reason — surface that to the user so the adoption decision is auditable.

If the re-rank or revision call fails, the script degrades to the pointwise leader (a warning), never crashes. To skip refinement entirely (selection only), pass --no-refine.

5. Implement the final choice

Implement final_choice.text from reranked.json against the real codebase — this is the adopted proposal (refined or leader), not necessarily the pointwise winner. Follow the project's conventions and the spec's constitutional constraints. Normal implementation work — TDD, run the tests. Re-ranking chose what to build; you still build it carefully, and if the adopted design conflicts with an invariant despite a high WEQS, say so — a high score on a wrong design is still wrong.

6. Present and checkpoint

Show the user a compact summary, then stop and wait:

## §<section> — <title>

**WEQS ranking (pointwise)**
| Provider   | Model              | WEQS  | Notable |
|------------|--------------------|-------|---------|
| openai     | gpt-5.1            | 84.6  | strongest testability |
| openrouter | claude-opus-4.1    | 78.9  | best completeness, weaker scope |
| gemini     | gemini-3-pro       | 69.6  | clear but verification = TBD |

**Re-rank:** RRF leader = openai.
**Refinement:** leader 84.6 → revision 94.6 — **adopted revision** (strictly
improved, consistency held at 90). Targeted weak dims: testability, feasibility.
**Why this is the choice:** <one or two lines on what the refinement fixed>
**Implemented:** <what you changed — files, tests added, result>

Approve to advance to §<next>, or tell me what to adjust.

When adopted is "leader", say so plainly ("refinement didn't beat the leader, so the original proposal stands") — a no-op refinement is a valid, honest outcome.

Do not start the next section until the user approves. If they want changes, revise within this section and re-present.

Notes on judgment

• WEQS ranks proposals; it does not absolve you of reading them. If the winning proposal conflicts with the spec's constitution or an existing invariant, say so — a high score on a wrong design is still wrong.
• Keep per-section artifacts under /tmp/ensemble/<section>/ so each cycle is inspectable and you can diff proposals across runs.
• The scripts are dependency-free (stdlib only). If you change provider APIs, edit scripts/llm_providers.py — that's the single seam.