multi-model-validation

# Multi-Model Validation **Version:** 3.3.0 **Purpose:** Patterns for running multiple AI models in parallel via Claudish proxy with **context-aware preferences**, dynamic model discovery, session-based workspaces, and performance statistics **Status:** Production Ready ## Overview Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves: - **3-5x speedup** via parallel execution (15 minutes → 5 minutes) - **Consensus-based prioritization** (issues flagged by all models are CRITICAL) - **Diverse perspectives** (different models catch different issues) - **Cost transparency** (know before you spend) - **Free model discovery** (NEW v3.0) - find high-quality free models from trusted providers - **Performance tracking** - identify slow/failing models for future exclusion - **Data-driven recommendations** - optimize model shortlist based on historical performance **Key Innovations:** 1. **Context-Aware Preferences** (NEW v3.3.0) - Automatically use saved model preferences per task type (debug/research/coding/review) from `.claude/multimodel-team.json` 2. **Dynamic Model Discovery** (v3.0) - Read the live catalog (`list_models`) for current available models (live, 24h cache) 3. **Session-Based Workspaces** (v3.0) - Each validation session gets a unique directory to prevent conflicts 4. **4-Message Pattern** - Ensures true parallel execution by using only Agent tool calls in a single message 5. **Pattern 7-8** - Statistics collection and data-driven model recommendations This skill is extracted from the `/review` command and generalized for use in any multi-model workflow. --- ## ⚠️ MANDATORY: Learn and Reuse User Preferences > **Model preferences are learned per context and reused automatically.** > > - First time a context is used → ASK user → SAVE to that context > - Next time same context → VALIDATE the saved IDs against `list_models`, then use > the survivors automatically (no asking). "No asking" applies to the *selection*, > never to the catalog check — saved IDs go stale and must be re-checked every run. > - User explicitly says "change models" or "different models" → ASK and UPDATE ```bash # FIRST STEP - Read preferences file cat .claude/multimodel-team.json 2>/dev/null ``` **Flow:** ``` 1. Detect context from task keywords - "debug", "error", "bug", "fix" → debug - "research", "analyze", "investigate" → research - "implement", "build", "create", "code" → coding - "review", "audit", "check" → review 2. Check if contextPreferences[context] exists and is non-empty IF EXISTS (has models saved): → Call: list_models (claudish MCP) and KEEP ONLY the saved IDs it still lists Saved preferences are user policy, not a catalog snapshot — they go stale silently. This applies to defaultModels and contextPreferences alike; see claudish:claudish-usage → "Every field of the preferences file is untrusted" → Name every dropped ID in your reply → DO NOT ask the user to re-pick while at least one saved ID survives → If NOTHING survives, say so and offer live alternatives IF EMPTY/MISSING (first time for this context): → Call: list_models (claudish MCP — current models, pricing, capabilities) → Ask user to select models (AskUserQuestion) → Save to contextPreferences[context] → Proceed with validation 3. User override triggers (explicit request to change): - "use different models" - "change models" - "update model preferences" → Ask user to select new models → Update contextPreferences[context] ``` **Example - Learning Flow:** ``` # First debug task ever: Task: "Debug this authentication error" → Context: debug → contextPreferences.debug is empty → ASK: "Which models for debug tasks?" → User selects: grok, glm, minimax → SAVE to contextPreferences.debug → Run with those models # Second debug task: Task: "Debug the API timeout" → Context: debug → contextPreferences.debug = ["grok", "glm", "minimax"] → USE directly (no asking) → Run with saved models # User wants to change: Task: "Debug this error, use different models" → Detected: "different models" override trigger → ASK: "Which models for debug tasks?" → User selects: gemini, <model-c> → UPDATE contextPreferences.debug → Run with new models ``` --- ## Related Skills **Cross-References:** - **multimodel:error-recovery** - Handling failures: diagnosis, reporting, user-chosen re-runs --- ## Core Patterns ### Pattern 0: Session Setup and Model Discovery (NEW v3.0) **Purpose:** Create isolated session workspace and discover available models dynamically. **Why Session-Based Workspaces:** Using a fixed directory like `ai-docs/reviews/` causes problems: - ❌ Multiple sessions overwrite each other's files - ❌ Stale data from previous sessions pollutes results - ❌ Hard to track which files belong to which session Instead, create a **unique session directory** for each validation: ```bash # Generate unique session ID TARGET_SLUG=$(echo "${TASK_NAME:-review}" | tr '[:upper:] ' '[:lower:]-' | sed 's/[^a-z0-9-]//g' | head -c20) SESSION_ID="review-${TARGET_SLUG}-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)" SESSION_DIR="ai-docs/sessions/${SESSION_ID}" # Create session workspace mkdir -p "$SESSION_DIR" echo "Session: $SESSION_ID" echo "Directory: $SESSION_DIR" # Example output: # Session: review-auth-impl-20251212-143052-a3f2 # Directory: ai-docs/sessions/review-auth-impl-20251212-143052-a3f2 ``` **Benefits:** - ✅ Each session is isolated (no cross-contamination) - ✅ Traceable - can associate files with a specific session - ✅ Session ID can be used for tracking in statistics - ✅ Parallel sessions don't conflict - ✅ Aligned with the `dev:dev` session pattern - ✅ Git-ignored session scratch: it dies with the session and is never committed > Use `ai-docs/sessions/`, not `/tmp/`: a `/tmp/` path is not tied to the project, and > the `/team` hook denies `/tmp/` paths in agent prompts. --- **Dynamic Model Discovery:** **NEVER hardcode model lists.** Models change frequently — new ones appear, old ones deprecate, pricing updates. Instead, read the live catalog (`list_models`) for current available models: Call the `list_models` MCP tool (claudish). It returns the current recommended set — model IDs, pricing, context window, capabilities, and the `provider@model` access prefixes — served from claudish’s catalog with a 24-hour cache. For every live variant in one family, call `search_models` with the family name. **Free models:** filter the `list_models` result by pricing. Keep no list here — free tiers churn faster than any other part of the catalog. **Model Selection Flow (Learn and Reuse):** ``` 1. Read Preferences File → cat .claude/multimodel-team.json → If file NOT exists → create empty one 2. Detect Task Context → Parse task for keywords (case-insensitive): - "debug", "error", "bug", "fix", "trace", "issue" → debug - "research", "investigate", "analyze", "explore", "find" → research - "implement", "build", "create", "code", "develop", "feature" → coding - "review", "audit", "check", "validate", "verify" → review → If no keywords match → context = "default" 3. Check for Override Triggers in User Message → "use different models", "change models", "update preferences" → If found → force_ask = true 4. Load or Learn Models → models = contextPreferences[context] IF models exist AND NOT force_ask: → USE models directly (no asking) → Go to step 6 IF models empty OR force_ask: → Read: the live catalog (list_models) → AskUserQuestion with multiSelect → Save user selection to contextPreferences[context] → Go to step 6 5. Save Updated Preferences → Write .claude/multimodel-team.json → Update lastUpdated timestamp 6. Execute with Models → Launch parallel validation → No further confirmation needed ``` **Context Keywords:** | Context | Keywords | |---------|----------| | debug | debug, error, bug, fix, trace, issue | | research | research, investigate, analyze, explore, find | | coding | implement, build, create, code, develop, feature | | review | review, audit, check, validate, verify | **Override Triggers (force re-selection):** - "use different models" - "change models" - "update model preferences" - "select new models" ### Routing is Claudish's **Send the `id` from `list_models`. Never build an address.** Claudish owns backend selection, credentials and fallback; this repo implements none of it. A prefix/backend/key table used to sit here. It is deleted — it had drifted to the wrong separator (`/` where claudish uses `@`), listed alias env vars as if canonical, covered a third of the providers, and marked models "collision-free" that had since gained a direct provider. A second copy in `claudish-usage` had drifted differently, which is the point: restating claudish's routing anywhere in this repo guarantees two versions of the truth and no way to tell which is stale. If a model will not route, that is a claudish bug — report it with `report_error`. Do not work around it by choosing a different prefix here. **Interactive Model Selection (AskUserQuestion with multiSelect):** **CRITICAL:** Use AskUserQuestion tool with `multiSelect: true` to let users choose models interactively. This provides a better UX than just showing recommendations. ```typescript // Use AskUserQuestion to let user select models AskUserQuestion({ questions: [{ question: "Which external models should validate your code? (Internal Claude reviewer always included)", header: "Models", multiSelect: true, options: [ // One option per model, built from this run's list_models result plus // ai-docs/llm-performance.json history when it exists. Illustrative shape: { label: "<id from list_models>", description: "<price from list_models> | Quality: <history %, or 'new'> | Avg: <history s>" }, { label: "<free id from list_models> 🆓", description: "FREE | <context from list_models> | Quality: <history %, or 'new'>" } ] }] }) ``` **Remember Selection for Session:** Store the user's model selection in the session directory so it persists throughout the validation: ```bash # After user selects models, save to session save_session_models() { local session_dir="$1" shift local models=("$@") # Always include internal reviewer echo "claude-embedded" > "$session_dir/selected-models.txt" # Add user-selected models for model in "${models[@]}"; do echo "$model" >> "$session_dir/selected-models.txt" done echo "Session models saved to $session_dir/selected-models.txt" } # Load session models for subsequent operations load_session_models() { local session_dir="$1" cat "$session_dir/selected-models.txt" } # Usage: # After AskUserQuestion returns selected models save_session_models "$SESSION_DIR" "grok" "<free-model-a>" # Later in the session, retrieve the selection MODELS=$(load_session_models "$SESSION_DIR") ``` **Session Model Memory Structure:** ``` $SESSION_DIR/ ├── selected-models.txt # User's model selection (persists for session) ├── claude-review.md # Internal review ├── grok-review.md # External review (if selected) ├── free-model-a-review.md # External review (if selected) └── consolidated-review.md # Final consolidated review ``` **Why Remember the Selection:** 1. **Re-runs**: If validation needs to be re-run, use same models 2. **Consistency**: All phases of validation use identical model set 3. **Audit trail**: Know which models produced which results 4. **Cost tracking**: Accurate cost attribution per session **Always Include Internal Reviewer:** ``` BEST PRACTICE: Always run internal Claude reviewer alongside external models. Why? ✓ FREE (embedded Claude, no API costs) ✓ Fast baseline (usually fastest) ✓ Provides comparison point ✓ Works even if ALL external models fail ✓ Consistent behavior (same model every time) The internal reviewer should NEVER be optional - it's your safety net. It runs in a session of its own, never as a subagent of the session that wrote the code — except on the nested route below, `cli-sessions.md` §3, the one place a subagent is allowed: seat it as the `internal` slot in the `models` list of the ONE `team` call, beside the externals, so it gets the same `require_pattern` shape check as every one of them. After the panel settles, copy `response-<slots.internal>.md` to `claude-review.md` — only when that slot is COMPLETED and matched the pattern — and list `claude-review.md` on REVIEWS:, never `response-<slots.internal>.md` beside it: that is the same review counted twice. Without the `team` tool, pick the route by §1 of `skills/core/team-gate/cli-sessions.md` inside the dev plugin, never by "is claudish installed": inside a claudish child `team` is absent too, and a CLI launched from there rebuilds the nesting claudish refuses. When §1 picks the nested route, follow its §3 — the one foreground subagent, no externals. When it picks cli-sessions, run the internal reviewer alone as a `claude` CLI session; this skill launches no external CLIs, and that file's §4.1 says the same. dev's root is the `installPath` of its `dev@` entry in `claude plugin list --json`; with no such entry dev is not installed, and the route is unavailable. Pattern 3 states the rule. ``` --- ### Pattern 1: The 4-Message Pattern (MANDATORY) **Why This Pattern Exists:** independent calls issued in one message run together; calls split across messages wait for each other. So preparation comes first, every reviewer starts in one message, and consolidation waits until the panel settles. What decides whether two calls may share a message is data dependency, not tool type — and every reviewer, the internal one included, is a slot of the one `team` call in Message 2. **The Pattern:** ``` Message 1: Preparation (Bash Only) - Create workspace directories - Validate inputs (check the claudish `team` tool is registered) - Write the brief (input.md) — never a pre-computed diff: every reviewer is handed `TARGET: BRANCH` and captures its own surfaces through dev's `capture-review-surfaces.ts`. No `OUTPUT:` line; it ends `Return the full report as your answer.` - NO Agent calls Message 2: Parallel Execution (ONE team call) - `internal` and every external model in a single `team` MCP call; the tool parallelises them internally. `internal` is always present (Pattern 3) - Pass require_pattern whenever the prompt mandates an output shape Message 3: Auto-Consolidation (one Agent call) - Automatically triggered when the panel settles with N ≥ 2 reviews. At N = 1 there is nothing to consolidate: use that review's own verdict directly (Pattern 5) - Launch `dev:aggregator` — the only consolidator; it reads reviews, never code - Pass every review file path on REVIEWS: — `claude-review.md`, then one `response-<slot>.md` per external slot, never `slots.internal` — the three lines under 'Apply verdict thresholds' in `dev:reviewer`'s agent file on THRESHOLDS: — read at dispatch time, never recalled — and the consolidated file on OUTPUT: (Pattern 5) - Consensus analysis is the aggregator's; never send the reviews to dev:reviewer Message 4: Present Results - Show user prioritized issues - Include consensus levels (unanimous, strong, majority) - Link to detailed reports - Cost summary (if applicable) ``` **Example: 5-Model Parallel Code Review** ``` Message 1: Preparation (Session Setup + Model Discovery) # Create unique session workspace Bash: SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)" Bash: SESSION_DIR="ai-docs/sessions/${SESSION_ID}" && mkdir -p "$SESSION_DIR" # No code capture here. Every reviewer is handed TARGET: BRANCH and runs dev's # capture-review-surfaces.ts itself, in BRANCH mode. A range computed here would # be one more hand-rolled diff, which is the one thing no dispatcher may do. # Discover available models MCP: list_models # current models, pricing, capabilities