multi-model-validation

# Multi-Model Validation **Version:** 3.3.0 **Purpose:** Patterns for running multiple AI models in parallel via Claudish proxy with **context-aware preferences**, dynamic model discovery, session-based workspaces, and performance statistics **Status:** Production Ready ## Overview Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves: - **3-5x speedup** via parallel execution (15 minutes → 5 minutes) - **Consensus-based prioritization** (issues flagged by all models are CRITICAL) - **Diverse perspectives** (different models catch different issues) - **Cost transparency** (know before you spend) - **Free model discovery** (NEW v3.0) - find high-quality free models from trusted providers - **Performance tracking** - identify slow/failing models for future exclusion - **Data-driven recommendations** - optimize model shortlist based on historical performance **Key Innovations:** 1. **Context-Aware Preferences** (NEW v3.3.0) - Automatically use saved model preferences per task type (debug/research/coding/review) from `.claude/multimodel-team.json` 2. **Dynamic Model Discovery** (v3.0) - Read the live catalog (`list_models`) for current available models (live, 24h cache) 3. **Session-Based Workspaces** (v3.0) - Each validation session gets a unique directory to prevent conflicts 4. **4-Message Pattern** - Ensures true parallel execution by using only Agent tool calls in a single message 5. **Pattern 7-8** - Statistics collection and data-driven model recommendations This skill is extracted from the `/review` command and generalized for use in any multi-model workflow. --- ## ⚠️ MANDATORY: Learn and Reuse User Preferences > **Model preferences are learned per context and reused automatically.** > > - First time a context is used → ASK user → SAVE to that context > - Next time same context → VALIDATE the saved IDs against `list_models`, then use > the survivors automatically (no asking). "No asking" applies to the *selection*, > never to the catalog check — saved IDs go stale and must be re-checked every run. > - User explicitly says "change models" or "different models" → ASK and UPDATE ```bash # FIRST STEP - Read preferences file cat .claude/multimodel-team.json 2>/dev/null ``` **Flow:** ``` 1. Detect context from task keywords - "debug", "error", "bug", "fix" → debug - "research", "analyze", "investigate" → research - "implement", "build", "create", "code" → coding - "review", "audit", "check" → review 2. Check if contextPreferences[context] exists and is non-empty IF EXISTS (has models saved): → Call: list_models (claudish MCP) and KEEP ONLY the saved IDs it still lists Saved preferences are user policy, not a catalog snapshot — they go stale silently. This applies to defaultModels and contextPreferences alike; see claudish:claudish-usage → "Every field of the preferences file is untrusted" → Name every dropped ID in your reply → DO NOT ask the user to re-pick while at least one saved ID survives → If NOTHING survives, say so and offer live alternatives IF EMPTY/MISSING (first time for this context): → Call: list_models (claudish MCP — current models, pricing, capabilities) → Ask user to select models (AskUserQuestion) → Save to contextPreferences[context] → Proceed with validation 3. User override triggers (explicit request to change): - "use different models" - "change models" - "update model preferences" → Ask user to select new models → Update contextPreferences[context] ``` **Example - Learning Flow:** ``` # First debug task ever: Task: "Debug this authentication error" → Context: debug → contextPreferences.debug is empty → ASK: "Which models for debug tasks?" → User selects: grok, glm, minimax → SAVE to contextPreferences.debug → Run with those models # Second debug task: Task: "Debug the API timeout" → Context: debug → contextPreferences.debug = ["grok", "glm", "minimax"] → USE directly (no asking) → Run with saved models # User wants to change: Task: "Debug this error, use different models" → Detected: "different models" override trigger → ASK: "Which models for debug tasks?" → User selects: gemini, <model-c> → UPDATE contextPreferences.debug → Run with new models ``` --- ## Related Skills **Cross-References:** - **multimodel:error-recovery** - Handling failures: diagnosis, reporting, user-chosen re-runs --- ## Core Patterns ### Pattern 0: Session Setup and Model Discovery (NEW v3.0) **Purpose:** Create isolated session workspace and discover available models dynamically. **Why Session-Based Workspaces:** Using a fixed directory like `ai-docs/reviews/` causes problems: - ❌ Multiple sessions overwrite each other's files - ❌ Stale data from previous sessions pollutes results - ❌ Hard to track which files belong to which session Instead, create a **unique session directory** for each validation: ```bash # Generate unique session ID TARGET_SLUG=$(echo "${TASK_NAME:-review}" | tr '[:upper:] ' '[:lower:]-' | sed 's/[^a-z0-9-]//g' | head -c20) SESSION_ID="review-${TARGET_SLUG}-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)" SESSION_DIR="ai-docs/sessions/${SESSION_ID}" # Create session workspace mkdir -p "$SESSION_DIR" echo "Session: $SESSION_ID" echo "Directory: $SESSION_DIR" # Example output: # Session: review-auth-impl-20251212-143052-a3f2 # Directory: ai-docs/sessions/review-auth-impl-20251212-143052-a3f2 ``` **Benefits:** - ✅ Each session is isolated (no cross-contamination) - ✅ Traceable - can associate files with a specific session - ✅ Session ID can be used for tracking in statistics - ✅ Parallel sessions don't conflict - ✅ Aligned with the `dev:dev` session pattern - ✅ Git-ignored session scratch: it dies with the session and is never committed > Use `ai-docs/sessions/`, not `/tmp/`: a `/tmp/` path is not tied to the project, and > the `/team` hook denies `/tmp/` paths in agent prompts. --- **Dynamic Model Discovery:** **NEVER hardcode model lists.** Models change frequently — new ones appear, old ones deprecate, pricing updates. Instead, read the live catalog (`list_models`) for current available models: Call the `list_models` MCP tool (claudish). It returns the current recommended set — model IDs, pricing, context window, capabilities, and the `provider@model` access prefixes — served from claudish’s catalog with a 24-hour cache. For every live variant in one family, call `search_models` with the family name. **Free models:** filter the `list_models` result by pricing. Keep no list here — free tiers churn faster than any other part of the catalog. **Model Selection Flow (Learn and Reuse):** ``` 1. Read Preferences File → cat .claude/multimodel-team.json → If file NOT exists → create empty one 2. Detect Task Context → Parse task for keywords (case-insensitive): - "debug", "error", "bug", "fix", "trace", "issue" → debug - "research", "investigate", "analyze", "explore", "find" → research - "implement", "build", "create", "code", "develop", "feature" → coding - "review", "audit", "check", "validate", "verify" → review → If no keywords match → context = "default" 3. Check for Override Triggers in User Message → "use different models", "change models", "update preferences" → If found → force_ask = true 4. Load or Learn Models → models = contextPreferences[context] IF models exist AND NOT force_ask: → USE models directly (no asking) → Go to step 6 IF models empty OR force_ask: → Read: the live catalog (list_models) → AskUserQuestion with multiSelect → Save user selection to contextPreferences[context] → Go to step 6 5. Save Updated Preferences → Write .claude/multimodel-team.json → Update lastUpdated timestamp 6. Execute with Models → Launch parallel validation → No further confirmation needed ``` **Context Keywords:** | Context | Keywords | |---------|----------| | debug | debug, error, bug, fix, trace, issue | | research | research, investigate, analyze, explore, find | | coding | implement, build, create, code, develop, feature | | review | review, audit, check, validate, verify | **Override Triggers (force re-selection):** - "use different models" - "change models" - "update model preferences" - "select new models" ### Routing is Claudish's **Send the `id` from `list_models`. Never build an address.** Claudish owns backend selection, credentials and fallback; this repo implements none of it. A prefix/backend/key table used to sit here. It is deleted — it had drifted to the wrong separator (`/` where claudish uses `@`), listed alias env vars as if canonical, covered a third of the providers, and marked models "collision-free" that had since gained a direct provider. A second copy in `claudish-usage` had drifted differently, which is the point: restating claudish's routing anywhere in this repo guarantees two versions of the truth and no way to tell which is stale. If a model will not route, that is a claudish bug — report it with `report_error`. Do not work around it by choosing a different prefix here. **Interactive Model Selection (AskUserQuestion with multiSelect):** **CRITICAL:** Use AskUserQuestion tool with `multiSelect: true` to let users choose models interactively. This provides a better UX than just showing recommendations. ```typescript // Use AskUserQuestion to let user select models AskUserQuestion({ questions: [{ question: "Which external models should validate your code? (Internal Claude reviewer always included)", header: "Models", multiSelect: true, options: [ // One option per model, built from this run's list_models result plus // ai-docs/llm-performance.json history when it exists. Illustrative shape: { label: "<id from list_models>", description: "<price from list_models> | Quality: <history %, or 'new'> | Avg: <history s>" }, { label: "<free id from list_models> 🆓", description: "FREE | <context from list_models> | Quality: <history %, or 'new'>" } ] }] }) ``` **Remember Selection for Session:** Store the user's model selection in the session directory so it persists throughout the validation: ```bash # After user selects models, save to session save_session_models() { local session_dir="$1" shift local models=("$@") # Always include internal reviewer echo "claude-embedded" > "$session_dir/selected-models.txt" # Add user-selected models for model in "${models[@]}"; do echo "$model" >> "$session_dir/selected-models.txt" done echo "Session models saved to $session_dir/selected-models.txt" } # Load session models for subsequent operations load_session_models() { local session_dir="$1" cat "$session_dir/selected-models.txt" } # Usage: # After AskUserQuestion returns selected models save_session_models "$SESSION_DIR" "grok" "<free-model-a>" # Later in the session, retrieve the selection MODELS=$(load_session_models "$SESSION_DIR") ``` **Session Model Memory Structure:** ``` $SESSION_DIR/ ├── selected-models.txt # User's model selection (persists for session) ├── claude-review.md # Internal review ├── grok-review.md # External review (if selected) ├── free-model-a-review.md # External review (if selected) └── consolidated-review.md # Final consolidated review ``` **Why Remember the Selection:** 1. **Re-runs**: If validation needs to be re-run, use same models 2. **Consistency**: All phases of validation use identical model set 3. **Audit trail**: Know which models produced which results 4. **Cost tracking**: Accurate cost attribution per session **Always Include Internal Reviewer:** ``` BEST PRACTICE: Always run internal Claude reviewer alongside external models. Why? ✓ FREE (embedded Claude, no API costs) ✓ Fast baseline (usually fastest) ✓ Provides comparison point ✓ Works even if ALL external models fail ✓ Consistent behavior (same model every time) The internal reviewer should NEVER be optional - it's your safety net. In the code-review panels `dev` dispatches — where claudish is optional — it is not a claudish slot. Launch it as its own `Agent(subagent_type: "dev:reviewer", run_in_background: false, …)` in the SAME message as the `team` call, carrying the contract lines — `TARGET: BRANCH`, `FOCUS:`, its own `OUTPUT:` path, and `MODELS:` naming the externals. Never seat it in that team's `models` list. Its return is in your hands: list its OUTPUT file on REVIEWS: whether or not it carries a `**Verdict**:` line — the aggregator counts a file with no verdict as no-verdict, never as one more approval. (`/team` itself, where claudish is a hard dependency, seats it as the `internal` slot instead — Pattern 3 states the rule.) ``` --- ### Pattern 1: The 4-Message Pattern (MANDATORY) **Why This Pattern Exists:** independent calls issued in one message run together; calls split across messages wait for each other. So preparation comes first, every reviewer starts in one message, and consolidation waits until the panel settles. What decides whether two calls may share a message is data dependency, not tool type — Message 2 below issues an `Agent` call and a `team` MCP call together, and that is correct. **The Pattern:** ``` Message 1: Preparation (Bash Only) - Create workspace directories - Validate inputs (check if claudish installed) - Write the brief (input.md) — never a pre-computed diff: every reviewer is handed `TARGET: BRANCH` and captures its own surfaces through dev's `capture-review-surfaces.ts` - NO Agent calls Message 2: Parallel Execution (the internal Agent call and ONE team call, same message) - `Agent(subagent_type: "dev:reviewer", run_in_background: false, …)` — the internal reviewer, always present, carrying the contract lines - Every external model in a single `team` MCP call; the tool parallelises them internally. In a dev-dispatched panel the internal reviewer is never a `models` entry (Pattern 3 — `/team` itself seats it as the `internal` slot) - Pass require_pattern whenever the prompt mandates an output shape Message 3: Auto-Consolidation (one Agent call) - Automatically triggered when the panel settles with N ≥ 2 reviews. At N = 1 there is nothing to consolidate: use that review's own verdict directly (Pattern 5) - Launch `dev:aggregator` — the only consolidator; it reads reviews, never code - Pass every review file path on REVIEWS:, the three lines under 'Apply verdict thresholds' in `dev:reviewer`'s agent file on THRESHOLDS: — read at dispatch time, never recalled — and the consolidated file on OUTPUT: (Pattern 5) - Consensus analysis is the aggregator's; never send the reviews to dev:reviewer Message 4: Present Results - Show user prioritized issues - Include consensus levels (unanimous, strong, majority) - Link to detailed reports - Cost summary (if applicable) ``` **Example: 5-Model Parallel Code Review** ``` Message 1: Preparation (Session Setup + Model Discovery) # Create unique session workspace Bash: SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)" Bash: SESSION_DIR="ai-docs/sessions/${SESSION_ID}" && mkdir -p "$SESSION_DIR" # No code capture here. Every reviewer is handed TARGET: BRANCH and runs dev's # capture-review-surfaces.ts itself, in BRANCH mode. A range computed here would # be one more hand-rolled diff, which is the one thing no dispatcher may do. # Discover available models MCP: list_models # current models, pricing, capabilities # User selects models via AskUserQuestion (see Pattern 0) Message 2: Start the panel (the internal Agent call and ONE team call, same message) Bash: write the brief to "$SESSION_DIR/input.md" — the contract lines every external gets: TARGET: BRANCH / FOCUS: code / MODELS: none Agent( subagent_type: "dev:reviewer",