Run multiple AI models in parallel for 3-5x speedup with ENFORCED performance statistics tracking. Use when validating with Grok, Gemini, GPT-5, DeepSeek, MiniMax, Kimi, GLM, or Claudish proxy for code review, consensus analysis, or multi-expert validation. NEW in v3.2.0 - Direct API prefixes (mmax/, kimi/, glm/) for cost savings. Includes dynamic model discovery via `claudish --top-models` and `claudish --free`, session-based workspaces, and Pattern 7-8 for tracking model performance. Trigger keywords - "grok", "gemini", "gpt-5", "deepseek", "minimax", "kimi", "glm", "claudish", "multiple models", "parallel review", "external AI", "consensus", "multi-model", "model performance", "statistics", "free models".
Run multiple AI models in parallel for 3-5x speedup with ENFORCED performance statistics tracking. Use when validating with Grok, Gemini, GPT-5, DeepSeek, MiniMax, Kimi, GLM, or Claudish proxy for code review, consensus analysis, or multi-expert validation. NEW in v3.2.0 - Direct API prefixes (mmax/, kimi/, glm/) for cost savings. Includes dynamic model discovery via `claudish --top-models` and `claudish --free`, session-based workspaces, and Pattern 7-8 for tracking model performance. Trigger keywords - "grok", "gemini", "gpt-5", "deepseek", "minimax", "kimi", "glm", "claudish", "multiple models", "parallel review", "external AI", "consensus", "multi-model", "model performance", "statistics", "free models".
Version: 3.3.0
Purpose: Patterns for running multiple AI models in parallel via Claudish proxy with context-aware preferences, dynamic model discovery, session-based workspaces, and performance statistics
Status: Production Ready
Overview
Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT-5, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves:
3-5x speedup via parallel execution (15 minutes → 5 minutes)
Consensus-based prioritization (issues flagged by all models are CRITICAL)
Diverse perspectives (different models catch different issues)
Cost transparency (know before you spend)
Free model discovery (NEW v3.0) - find high-quality free models from trusted providers
Performance tracking - identify slow/failing models for future exclusion
Data-driven recommendations - optimize model shortlist based on historical performance
Key Innovations:
Context-Aware Preferences (NEW v3.3.0) - Automatically use saved model preferences per task type (debug/research/coding/review) from .claude/multimodel-team.json
Dynamic Model Discovery (v3.0) - Use claudish --top-models and claudish --free to get current available models with pricing
Session-Based Workspaces (v3.0) - Each validation session gets a unique directory to prevent conflicts
4-Message Pattern - Ensures true parallel execution by using only Task tool calls in a single message
Pattern 7-8 - Statistics collection and data-driven model recommendations
This skill is extracted from the /review command and generalized for use in any multi-model workflow.
⚠️ MANDATORY: Learn and Reuse User Preferences
Model preferences are learned per context and reused automatically.
First time a context is used → ASK user → SAVE to that context
Next time same context → USE saved models automatically (no asking)
User explicitly says "change models" or "different models" → ASK and UPDATE
# FIRST STEP - Read preferences filecat .claude/multimodel-team.json 2>/dev/null
Flow:
1. Detect context from task keywords
- "debug", "error", "bug", "fix" → debug
- "research", "analyze", "investigate" → research
- "implement", "build", "create", "code" → coding
- "review", "audit", "check" → review
2. Check if contextPreferences[context] exists and is non-empty
IF EXISTS (has models saved):
→ Use those models directly
→ DO NOT ask user
→ Proceed with validation
IF EMPTY/MISSING (first time for this context):
→ Run: claudish --top-models
→ Ask user to select models (AskUserQuestion)
→ Save to contextPreferences[context]
→ Proceed with validation
3. User override triggers (explicit request to change):
- "use different models"
- "change models"
- "update model preferences"
→ Ask user to select new models
→ Update contextPreferences[context]
Example - Learning Flow:
# First debug task ever:
Task: "Debug this authentication error"
→ Context: debug
→ contextPreferences.debug is empty
→ ASK: "Which models for debug tasks?"
→ User selects: grok, glm, minimax
→ SAVE to contextPreferences.debug
→ Run with those models
# Second debug task:
Task: "Debug the API timeout"
→ Context: debug
→ contextPreferences.debug = ["grok", "glm", "minimax"]
→ USE directly (no asking)
→ Run with saved models
# User wants to change:
Task: "Debug this error, use different models"
→ Detected: "different models" override trigger
→ ASK: "Which models for debug tasks?"
→ User selects: gemini, gpt-5-codex
→ UPDATE contextPreferences.debug
→ Run with new models
Related Skills
CRITICAL: Tracking Protocol Required
Before using any patterns in this skill, ensure you have completed the
pre-launch setup from orchestration:model-tracking-protocol.
Launching models without tracking setup = INCOMPLETE validation.
Cross-References:
orchestration:model-tracking-protocol - MANDATORY tracking templates and protocols (NEW in v0.6.0)
Pre-launch checklist (8 required items)
Tracking table templates
Failure documentation format
Results presentation template
orchestration:quality-gates - Approval gates and severity classification
orchestration:task-orchestration - Progress tracking during execution
orchestration:error-recovery - Handling failures and retries
Skill Integration:
This skill (multi-model-validation) defines execution patterns (how to run models in parallel).
The model-tracking-protocol skill defines tracking infrastructure (how to collect and present results).
Pattern 0: Session Setup and Model Discovery (NEW v3.0)
Purpose: Create isolated session workspace and discover available models dynamically.
Why Session-Based Workspaces:
Using a fixed directory like ai-docs/reviews/ causes problems:
❌ Multiple sessions overwrite each other's files
❌ Stale data from previous sessions pollutes results
❌ Hard to track which files belong to which session
Instead, create a unique session directory for each validation:
# Generate unique session ID
TARGET_SLUG=$(echo"${TASK_NAME:-review}" | tr'[:upper:] ''[:lower:]-' | sed 's/[^a-z0-9-]//g' | head -c20)
SESSION_ID="review-${TARGET_SLUG}-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="ai-docs/sessions/${SESSION_ID}"# Create session workspacemkdir -p "$SESSION_DIR"echo"Session: $SESSION_ID"echo"Directory: $SESSION_DIR"# Example output:# Session: review-auth-impl-20251212-143052-a3f2# Directory: ai-docs/sessions/review-auth-impl-20251212-143052-a3f2
Benefits:
✅ Each session is isolated (no cross-contamination)
✅ Traceable - can associate files with a specific session
✅ Session ID can be used for tracking in statistics
✅ Parallel sessions don't conflict
✅ Aligned with dev:feature session pattern
✅ Committed to git for audit trail (unlike /tmp/)
⚠️ Do NOT use /tmp/ for session directories. Files in /tmp/ are not
traceable, not committable, and parallel runs will overwrite each other.
Dynamic Model Discovery:
NEVER hardcode model lists. Models change frequently - new ones appear, old ones deprecate, pricing updates. Instead, use claudish to get current available models:
# Get top paid models (best value for money)
claudish --top-models
# Example output:# google/gemini-3-pro-preview Google $7.00/1M 1048K 🔧 🧠 👁️# openai/gpt-5.2-codex Openai $5.63/1M 400K 🔧 🧠 👁️# x-ai/grok-code-fast-1 X-ai $0.85/1M 256K 🔧 🧠# minimax/minimax-m2.5 Minimax $0.64/1M 262K 🔧 🧠# z-ai/glm-4.7 Z-ai $1.07/1M 202K 🔧 🧠# qwen/qwen3-vl-235b-a22b-ins... Qwen $0.70/1M 262K 🔧 👁️# Get free models from trusted providers
claudish --free
# Example output:# google/gemini-2.0-flash-exp:free Google FREE 1049K ✓ · ✓# mistralai/devstral-2512:free Mistralai FREE 262K ✓ · ·# qwen/qwen3-coder:free Qwen FREE 262K ✓ · ·# qwen/qwen3-235b-a22b:free Qwen FREE 131K ✓ ✓ ·# openai/gpt-oss-120b:free Openai FREE 131K ✓ ✓ ·
Recommended Free Models for Code Review:
Model
Provider
Context
Capabilities
Why Good
qwen/qwen3-coder:free
Qwen
262K
Tools ✓
Coding-specialized, large context
mistralai/devstral-2512:free
Mistral
262K
Tools ✓
Dev-focused, excellent for code
qwen/qwen3-235b-a22b:free
Qwen
131K
Tools ✓ Reasoning ✓
Massive 235B model, reasoning
Model Selection Flow (Learn and Reuse):
1. Read Preferences File
→ cat .claude/multimodel-team.json
→ If file NOT exists → create empty one
2. Detect Task Context
→ Parse task for keywords (case-insensitive):
- "debug", "error", "bug", "fix", "trace", "issue" → debug
- "research", "investigate", "analyze", "explore", "find" → research
- "implement", "build", "create", "code", "develop", "feature" → coding
- "review", "audit", "check", "validate", "verify" → review
→ If no keywords match → context = "default"
3. Check for Override Triggers in User Message
→ "use different models", "change models", "update preferences"
→ If found → force_ask = true
4. Load or Learn Models
→ models = contextPreferences[context]
IF models exist AND NOT force_ask:
→ USE models directly (no asking)
→ Go to step 6
IF models empty OR force_ask:
→ Run: claudish --top-models
→ AskUserQuestion with multiSelect
→ Save user selection to contextPreferences[context]
→ Go to step 6
5. Save Updated Preferences
→ Write .claude/multimodel-team.json
→ Update lastUpdated timestamp
6. Execute with Models
→ Launch parallel validation
→ No further confirmation needed
Context Keywords:
Context
Keywords
debug
debug, error, bug, fix, trace, issue
research
research, investigate, analyze, explore, find
coding
implement, build, create, code, develop, feature
review
review, audit, check, validate, verify
Override Triggers (force re-selection):
"use different models"
"change models"
"update model preferences"
"select new models"
⚠️ Prefix Collision Awareness
CRITICAL: When using claudish, be aware of model ID prefix routing.
Claudish routes to different backends based on model ID prefix:
Prefix
Backend
Required Key
(none)
OpenRouter
OPENROUTER_API_KEY
g/gemini/
Google Gemini API
GEMINI_API_KEY
oai/
OpenAI Direct API
OPENAI_API_KEY
mmax/mm/
MiniMax Direct API
MINIMAX_API_KEY
kimi/moonshot/
Kimi Direct API
KIMI_API_KEY
glm/zhipu/
GLM Direct API
GLM_API_KEY
ollama/
Ollama (local)
None
lmstudio/
LM Studio (local)
None
vllm/
vLLM (local)
None
mlx/
MLX (local)
None
Collision-Free Models (safe for OpenRouter):
x-ai/grok-code-fast-1 ✅
google/gemini-* ✅ (use g/ for Gemini Direct)
deepseek/deepseek-chat ✅
minimax/* ✅ (use mmax/ for MiniMax Direct)
qwen/qwen3-coder:free ✅
mistralai/devstral-2512:free ✅
moonshotai/* ✅ (use kimi/ for Kimi Direct)
z-ai/glm-* ✅ (use glm/ for GLM Direct)
openai/* ✅ (use oai/ for OpenAI Direct)
anthropic/claude-* ✅
Direct API prefixes for cost savings:
OpenRouter Model
Direct API Prefix
API Key Required
openai/gpt-*
oai/gpt-*
OPENAI_API_KEY
google/gemini-*
g/gemini-*
GEMINI_API_KEY
minimax/*
mmax/*
MINIMAX_API_KEY
moonshotai/*
kimi/*
KIMI_API_KEY
z-ai/glm-*
glm/*
GLM_API_KEY
Rule: OpenRouter models work without prefix. Use direct API prefixes for cost savings when you have the corresponding API key.
Interactive Model Selection (AskUserQuestion with multiSelect):
CRITICAL: Use AskUserQuestion tool with multiSelect: true to let users choose models interactively. This provides a better UX than just showing recommendations.
// Use AskUserQuestion to let user select modelsAskUserQuestion({
questions: [{
question: "Which external models should validate your code? (Internal Claude reviewer always included)",
header: "Models",
multiSelect: true,
options: [
// Top paid (from claudish --top-models + historical data)
{
label: "x-ai/grok-code-fast-1 ⚡",
description: "$0.85/1M | Quality: 87% | Avg: 42s | Fast + accurate"
},
{
label: "google/gemini-3-pro-preview",
description: "$7.00/1M | Quality: 91% | Avg: 55s | High accuracy"
},
// Free models (from claudish --free)
{
label: "qwen/qwen3-coder:free 🆓",
description: "FREE | Quality: 82% | 262K context | Coding-specialized"
},
{
label: "mistralai/devstral-2512:free 🆓",
description: "FREE | 262K context | Dev-focused, new model"
}
]
}]
})
Remember Selection for Session:
Store the user's model selection in the session directory so it persists throughout the validation:
# After user selects models, save to sessionsave_session_models() {
local session_dir="$1"shiftlocal models=("$@")
# Always include internal reviewerecho"claude-embedded" > "$session_dir/selected-models.txt"# Add user-selected modelsfor model in"${models[@]}"; doecho"$model" >> "$session_dir/selected-models.txt"doneecho"Session models saved to $session_dir/selected-models.txt"
}
# Load session models for subsequent operationsload_session_models() {
local session_dir="$1"cat"$session_dir/selected-models.txt"
}
# Usage:# After AskUserQuestion returns selected models
save_session_models "$SESSION_DIR""x-ai/grok-code-fast-1""qwen/qwen3-coder:free"# Later in the session, retrieve the selection
MODELS=$(load_session_models "$SESSION_DIR")
Re-runs: If validation needs to be re-run, use same models
Consistency: All phases of validation use identical model set
Audit trail: Know which models produced which results
Cost tracking: Accurate cost attribution per session
Always Include Internal Reviewer:
BEST PRACTICE: Always run internal Claude reviewer alongside external models.
Why?
✓ FREE (embedded Claude, no API costs)
✓ Fast baseline (usually fastest)
✓ Provides comparison point
✓ Works even if ALL external models fail
✓ Consistent behavior (same model every time)
The internal reviewer should NEVER be optional - it's your safety net.
Pattern 1: The 4-Message Pattern (MANDATORY)
This pattern is CRITICAL for achieving true parallel execution with multiple AI models.
Why This Pattern Exists:
Claude Code executes tools sequentially by default when different tool types are mixed in the same message. To achieve true parallelism, you MUST:
Use ONLY one tool type per message
Ensure all Task calls are in a single message
Separate preparation (Bash) from execution (Task) from presentation
The Pattern:
Message 1: Preparation (Bash Only)
- Create workspace directories
- Validate inputs (check if claudish installed)
- Write context files (code to review, design reference, etc.)
- NO Task calls
- NO Tasks calls
Message 2: Parallel Execution (Task Only)
- Launch ALL AI models in SINGLE message
- ONLY Task tool calls
- Separate each Task with --- delimiter
- Each Task is independent (no dependencies)
- All execute simultaneously
Message 3: Auto-Consolidation (Task Only)
- Automatically triggered when N ≥ 2 models complete
- Launch consolidation agent
- Pass all review file paths
- Apply consensus analysis
Message 4: Present Results
- Show user prioritized issues
- Include consensus levels (unanimous, strong, majority)
- Link to detailed reports
- Cost summary (if applicable)
Example: 5-Model Parallel Code Review
Message 1: Preparation (Session Setup + Model Discovery)
# Create unique session workspace
Bash: SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
Bash: SESSION_DIR="ai-docs/sessions/${SESSION_ID}" && mkdir -p "$SESSION_DIR"
Bash: git diff > "$SESSION_DIR/code-context.md"
# Discover available models
Bash: claudish --top-models # See paid options
Bash: claudish --free # See free options
# User selects models via AskUserQuestion (see Pattern 0)
Message 2: Parallel Execution (ONLY Task calls - single message)
Task: senior-code-reviewer
Prompt: "Review $SESSION_DIR/code-context.md for security issues.
Write detailed review to $SESSION_DIR/claude-review.md
Return only brief summary."
---
Bash: claudish --model x-ai/grok-code-fast-1 --stdin --quiet
< $SESSION_DIR/review-prompt.md > $SESSION_DIR/grok-review.md 2>$SESSION_DIR/grok-stderr.log
---
Bash: claudish --model qwen/qwen3-coder:free --stdin --quiet
< $SESSION_DIR/review-prompt.md > $SESSION_DIR/qwen-coder-review.md 2>$SESSION_DIR/qwen-stderr.log
---
Bash: claudish --model openai/gpt-5.1-codex --stdin --quiet
< $SESSION_DIR/review-prompt.md > $SESSION_DIR/gpt5-review.md 2>$SESSION_DIR/gpt5-stderr.log
---
Bash: claudish --model mistralai/devstral-2512:free --stdin --quiet
< $SESSION_DIR/review-prompt.md > $SESSION_DIR/devstral-review.md 2>$SESSION_DIR/devstral-stderr.log
All 5 models execute simultaneously (5x parallelism!)
Message 3: Auto-Consolidation
(Automatically triggered - don't wait for user to request)
Task: senior-code-reviewer
Prompt: "Consolidate 5 code reviews from:
- $SESSION_DIR/claude-review.md
- $SESSION_DIR/grok-review.md
- $SESSION_DIR/qwen-coder-review.md
- $SESSION_DIR/gpt5-review.md
- $SESSION_DIR/devstral-review.md
Apply consensus analysis:
- Issues flagged by ALL 5 → UNANIMOUS (VERY HIGH confidence)
- Issues flagged by 4 → STRONG (HIGH confidence)
- Issues flagged by 3 → MAJORITY (MEDIUM confidence)
- Issues flagged by 1-2 → DIVERGENT (LOW confidence)
Prioritize by consensus level and severity.
Write to $SESSION_DIR/consolidated-review.md"
Message 4: Present Results + Update Statistics
# Track performance for each model (see Pattern 7)
track_model_performance "claude-embedded" "success" 32 8 95
track_model_performance "x-ai/grok-code-fast-1" "success" 45 6 87
track_model_performance "qwen/qwen3-coder:free" "success" 52 5 82
track_model_performance "openai/gpt-5.1-codex" "success" 68 7 89
track_model_performance "mistralai/devstral-2512:free" "success" 48 5 84
# Record session summary
record_session_stats 5 5 0 68 245 3.6
"Multi-model code review complete! 5 AI models analyzed your code.
Session: $SESSION_ID
Top 5 Issues (Prioritized by Consensus):
1. [UNANIMOUS] Missing input validation on POST /api/users
2. [UNANIMOUS] SQL injection risk in search endpoint
3. [STRONG] Weak password hashing (bcrypt rounds too low)
4. [MAJORITY] Missing rate limiting on authentication endpoints
5. [MAJORITY] Insufficient error handling in payment flow
Model Performance (this session):
| Model | Time | Issues | Quality | Cost |
|--------------------------------|------|--------|---------|--------|
| claude-embedded | 32s | 8 | 95% | FREE |
| x-ai/grok-code-fast-1 | 45s | 6 | 87% | $0.002 |
| qwen/qwen3-coder:free | 52s | 5 | 82% | FREE |
| openai/gpt-5.1-codex | 68s | 7 | 89% | $0.015 |
| mistralai/devstral-2512:free | 48s | 5 | 84% | FREE |
Parallel Speedup: 3.6x (245s sequential → 68s parallel)
See $SESSION_DIR/consolidated-review.md for complete analysis.
Performance logged to ai-docs/llm-performance.json"
Performance Impact:
Sequential execution: 5 models × 3 min = 15 minutes
Parallel execution: max(model times) ≈ 5 minutes
Speedup: 3x with perfect parallelism
Pattern 2: Parallel Execution Architecture
Single Message, Multiple Tasks:
The key to parallel execution is putting ALL Task calls in a single message with the --- delimiter:
❌ WRONG - Sequential Execution:
Message 1:
Task: agent1
Message 2:
Task: agent2
Message 3:
Task: agent3
Each task waits for previous to complete (3x slower).
Independent Tasks Requirement:
Each Task must be independent (no dependencies):
✅ CORRECT - Independent:
Task: review code for security
Task: review code for performance
Task: review code for style
All can run simultaneously (same input, different perspectives).
❌ WRONG - Dependent:
Task: implement feature
Task: write tests for feature (depends on implementation)
Task: review implementation (depends on tests)
Must run sequentially (each needs previous output).
Unique Output Files:
Each Task MUST write to a unique output file within the session directory:
✅ CORRECT - Wait for All:
Launch: Task1, Task2, Task3, Task4 (parallel)
Wait: All 4 complete
Check: results.filter(r => r.status === 'fulfilled').length
If >= 2: Proceed with consolidation
If < 2: Offer retry or abort
❌ WRONG - Premature Consolidation:
Launch: Task1, Task2, Task3, Task4
After 30s: Task1, Task2 done
Consolidate: Only Task1 + Task2 (Task3, Task4 still running!)
Pattern 3: External Model Invocation via Bash+claudish
How External Models Are Invoked:
External AI models are invoked deterministically via Bash+claudish CLI. The orchestrator
calls claudish directly — no LLM delegation needed. This is 100% reliable.
ALWAYS ask for user approval before expensive operations:
Present to user:
"You selected 5 AI models for code review:
- Claude Sonnet (embedded, free)
- Grok Code Fast (external, $0.002)
- Gemini 2.5 Flash (external, $0.001)
- GPT-5 Codex (external, $0.004)
- DeepSeek Coder (external, $0.001)
Estimated total cost: $0.008 ($0.005 - $0.010)
Proceed with multi-model review? (Yes/No)"
If user says NO:
Offer alternatives:
1. Use only free embedded Claude
2. Select fewer models
3. Cancel review
If user says YES:
Proceed with Message 2 (parallel execution)
Pattern 5: Auto-Consolidation Logic
Automatic Trigger:
Consolidation should happen automatically when N ≥ 2 reviews complete:
✅ CORRECT - Auto-Trigger:
const results = await Promise.allSettled([task1, task2, task3, task4, task5]);
const successful = results.filter(r => r.status === 'fulfilled');
if (successful.length >= 2) {
// Auto-trigger consolidation (DON'T wait for user to ask)
const consolidated = await Task({
subagent_type: "senior-code-reviewer",
description: "Consolidate reviews",
prompt: `Consolidate ${successful.length} reviews and apply consensus analysis`
});
return formatResults(consolidated);
} else {
// Too few successful reviews
notifyUser("Only 1 model succeeded. Retry failures or abort?");
}
❌ WRONG - Wait for User:
const results = await Promise.allSettled([...]);
const successful = results.filter(r => r.status === 'fulfilled');
// Present results to user
notifyUser("3 reviews complete. Would you like me to consolidate them?");
// Waits for user to request consolidation...
Why Auto-Trigger:
Better UX (no extra user prompt needed)
Faster workflow (no wait for user response)
Expected behavior (user assumes consolidation is part of workflow)
Minimum Threshold:
Require at least 2 successful reviews for meaningful consensus:
if (successful.length >= 2) {
// Proceed with consolidation
} else if (successful.length === 1) {
// Only 1 review succeeded
notifyUser("Only 1 model succeeded. No consensus available. See single review or retry?");
} else {
// All failed
notifyUser("All models failed. Check logs and retry?");
}
Pass All Review File Paths:
Consolidation agent needs paths to ALL review files within the session directory:
Task: senior-code-reviewer
Prompt: "Consolidate reviews from these files:
- $SESSION_DIR/claude-review.md
- $SESSION_DIR/grok-review.md
- $SESSION_DIR/qwen-coder-review.md
Apply consensus analysis and prioritize issues."
Don't Inline Full Reviews:
❌ WRONG - Inline Reviews (context pollution):
Prompt: "Consolidate these reviews:
Claude Review:
[500 lines of review content]
Grok Review:
[500 lines of review content]
Qwen Review:
[500 lines of review content]"
✅ CORRECT - File Paths in Session Directory:
Prompt: "Read and consolidate reviews from:
- $SESSION_DIR/claude-review.md
- $SESSION_DIR/grok-review.md
- $SESSION_DIR/qwen-coder-review.md"
Pattern 6: Consensus Analysis
Consensus Levels:
Classify issues by how many models flagged them:
Consensus Levels (for N models):
UNANIMOUS (100% agreement):
- All N models flagged this issue
- VERY HIGH confidence
- MUST FIX priority
STRONG CONSENSUS (67-99% agreement):
- Most models flagged this issue (⌈2N/3⌉ to N-1)
- HIGH confidence
- RECOMMENDED priority
MAJORITY (50-66% agreement):
- Half or more models flagged this issue (⌈N/2⌉ to ⌈2N/3⌉-1)
- MEDIUM confidence
- CONSIDER priority
DIVERGENT (< 50% agreement):
- Only 1-2 models flagged this issue
- LOW confidence
- OPTIONAL priority (may be model-specific perspective)
Algorithm:
1. Extract issues from each review
2. For each unique issue:
a. Identify keywords (e.g., "SQL injection", "input validation")
b. Check which other reviews mention same keywords
c. Count models that flagged this issue
d. Assign consensus level
Example:
Claude Review: "Missing input validation on POST /api/users"
Grok Review: "Input validation absent in user creation endpoint"
Gemini Review: "No validation for user POST endpoint"
Keywords: ["input validation", "POST", "/api/users", "user"]
Match: All 3 reviews mention these keywords
Consensus: UNANIMOUS (3/3 = 100%)
Instead of keyword matching, use semantic similarity:
- Embed issue descriptions with sentence-transformers
- Calculate cosine similarity between embeddings
- Issues with >0.8 similarity are "same issue"
- More accurate consensus detection
Pattern 7: Statistics Collection and Analysis
Purpose: Track model performance to help users identify slow or poorly-performing models for future exclusion.
Storage Location: ai-docs/llm-performance.json (persistent across all sessions)
When to Collect Statistics:
After each model completes (success, failure, or timeout)
Purpose: Use historical performance data to make intelligent model selection recommendations.
The Problem:
Users often select models arbitrarily or based on outdated information:
"I'll use GPT-5 because it's famous"
"Let me try this new model I heard about"
"I'll use the same 5 models every time"
The Solution:
Use accumulated performance data to recommend:
Top performers (highest quality scores)
Best value (quality/cost ratio)
Top free models (high quality, zero cost)
Models to avoid (slow, unreliable, or degrading)
Model Selection Algorithm:
1. Load historical data from ai-docs/llm-performance.json
2. Calculate metrics for each model:
- Success Rate = successfulRuns / totalRuns × 100
- Quality Score = avgQualityScore (from consensus analysis)
- Speed Score = avgExecutionTime relative to overall average
- Value Score = avgQualityScore / (totalCost / totalRuns)
3. Categorize models:
TOP PAID: Quality > 80%, Success > 90%, Speed <= avg
TOP FREE: Quality > 75%, Success > 90%, isFree = true
BEST VALUE: Highest Quality/Cost ratio among paid models
AVOID: Speed > 2x avg OR Success < 70% OR trend = "degrading"
4. Present recommendations with context:
- Show historical metrics
- Highlight trends (improving/stable/degrading)
- Flag new models with insufficient data
Interactive Model Selection with Recommendations:
Instead of just displaying recommendations, use AskUserQuestion with multiSelect to let users interactively choose:
// Build options from claudish output + historical dataconst paidModels = getTopModelsFromClaudish(); // claudish --top-modelsconst freeModels = getFreeModelsFromClaudish(); // claudish --freeconst history = loadPerformanceHistory(); // ai-docs/llm-performance.json// Merge and build AskUserQuestion optionsAskUserQuestion({
questions: [{
question: "Select models for validation (Claude internal always included). Based on 25 sessions across 8 models.",
header: "Models",
multiSelect: true,
options: [
// Top paid with historical data
{
label: "x-ai/grok-code-fast-1 ⚡ (Recommended)",
description: "$0.85/1M | Quality: 87% | Avg: 42s | Fast + accurate"
},
{
label: "google/gemini-3-pro-preview 🎯",
description: "$7.00/1M | Quality: 91% | Avg: 55s | High accuracy"
},
// Top free models
{
label: "qwen/qwen3-coder:free 🆓",
description: "FREE | Quality: 82% | 262K | Coding-specialized"
},
{
label: "mistralai/devstral-2512:free 🆓",
description: "FREE | Quality: 84% | 262K | Dev-focused"
}
// Note: Models to AVOID are simply not shown in options// Note: New models show "(new)" instead of quality score
]
}]
})
Key Principles for Model Selection UI:
Put recommended models first with "(Recommended)" suffix
Include historical metrics in description (Quality %, Avg time)
Mark free models with 🆓 emoji
Don't show models to avoid - just exclude them from options
Mark new models with "(new)" when no historical data
Remember selection - save to $SESSION_DIR/selected-models.txt
After Selection - Save to Session:
# User selected: grok-code-fast-1, qwen3-coder:free# Save for session persistence
save_session_models "$SESSION_DIR""${USER_SELECTED_MODELS[@]}"# Now $SESSION_DIR/selected-models.txt contains:# claude-embedded# x-ai/grok-code-fast-1# qwen/qwen3-coder:free
Warning Display (separate from selection):
If there are models to avoid, show a brief warning before the selection:
# Generate optimal shortlist based on criteriagenerate_shortlist() {
local criteria="${1:-balanced}"# balanced, quality, budget, free-onlylocal perf_file="ai-docs/llm-performance.json"case"$criteria"in"balanced")
# 1 internal + 1 fast paid + 1 freeecho"claude-embedded"
jq -r '.models | to_entries | map(select(.value.isFree == false and .value.avgQualityScore > 80)) | sort_by(.value.avgExecutionTime)[0].key'"$perf_file"
jq -r '.models | to_entries | map(select(.value.isFree == true and .key != "claude-embedded" and .value.avgQualityScore > 75)) | sort_by(-.value.avgQualityScore)[0].key'"$perf_file"
;;
"quality")
# Top 3 by quality regardless of costecho"claude-embedded"
jq -r '.models | to_entries | map(select(.value.avgQualityScore != null and .key != "claude-embedded")) | sort_by(-.value.avgQualityScore)[:2] | .[].key'"$perf_file"
;;
"budget")
# Internal + 2 cheapest performersecho"claude-embedded"
jq -r '.models | to_entries | map(select(.value.avgQualityScore > 75 and .value.isFree == true)) | sort_by(-.value.avgQualityScore)[:2] | .[].key'"$perf_file"
;;
"free-only")
# Only free modelsecho"claude-embedded"
jq -r '.models | to_entries | map(select(.value.isFree == true and .key != "claude-embedded" and .value.avgQualityScore != null)) | sort_by(-.value.avgQualityScore)[:2] | .[].key'"$perf_file"
;;
esac
}
# Usage:
generate_shortlist "balanced"# For most use cases
generate_shortlist "quality"# When accuracy is critical
generate_shortlist "budget"# When cost matters
generate_shortlist "free-only"# Zero-cost validation
Integration with Model Discovery:
Workflow:
1. Run `claudish --top-models` → Get current paid models
2. Run `claudish --free` → Get current free models
3. Load ai-docs/llm-performance.json → Get historical performance
4. Merge data:
- New models (no history): Mark as "🆕 New"
- Known models: Show performance metrics
- Deprecated models: Filter out (not in claudish output)
5. Generate recommendations
6. Present to user with AskUserQuestion
Why This Matters:
Selection Method
Outcome
Random/arbitrary
Hit-or-miss, may waste money on slow models
Always same models
Miss new better options, stuck with degrading ones
<phasename="External Review"><steps><step>Record start time: PHASE_START=$(date +%s)</step><step>Run external models in parallel (single message, multiple Task calls)</step><step>
After completion, track each model:
track_model_performance "{model}" "{status}" "{duration}" "{issues}" "{quality}"
</step><step>
Record session:
record_session_stats $TOTAL $SUCCESS $FAILED $PARALLEL $SEQUENTIAL $SPEEDUP
</step></steps></phase><phasename="Finalization"><steps><step>
Display Model Performance Statistics (read from ai-docs/llm-performance.json)
</step><step>Show recommendations for slow/failing models</step></steps></phase>
Plugins Using This Pattern
Plugin
Command
Usage
frontend
/review
Full implementation with historical tracking
agentdev
/develop
Plan review + quality review tracking
Integration with Other Skills
multi-model-validation + quality-gates:
Use Case: Cost approval before expensive multi-model review
Step 1: Cost Estimation (multi-model-validation)
Calculate input/output tokens
Estimate cost range
Step 2: User Approval Gate (quality-gates)
Present cost estimate
Ask user for approval
If NO: Offer alternatives or abort
If YES: Proceed with execution
Step 3: Parallel Execution (multi-model-validation)
Follow 4-Message Pattern
Launch all models simultaneously
multi-model-validation + error-recovery:
Use Case: Handling external model failures gracefully
Step 1: Parallel Execution (multi-model-validation)
Launch 5 external models
Step 2: Error Handling (error-recovery)
Model 1: Success
Model 2: Timeout after 30s → Skip, continue with others
Model 3: API 500 error → Retry once, then skip
Model 4: Success
Model 5: Success
Step 3: Partial Success Strategy (error-recovery)
3/5 models succeeded (≥ 2 threshold)
Proceed with consolidation using 3 reviews
Notify user: "2 models failed, proceeding with 3 reviews"
Step 4: Consolidation (multi-model-validation)
Consolidate 3 successful reviews
Apply consensus analysis
multi-model-validation + task-orchestration:
Use Case: Real-time progress tracking during parallel execution
Step 1: Initialize Tasks (task-orchestration)
Tasks:
1. Prepare workspace
2. Launch Claude review
3. Launch Grok review
4. Launch Gemini review
5. Launch GPT-5 review
6. Consolidate reviews
7. Present results
Step 2: Update Progress (task-orchestration)
Mark tasks complete as models finish:
- Claude completes → Mark task 2 complete
- Grok completes → Mark task 3 complete
- Gemini completes → Mark task 4 complete
- GPT-5 completes → Mark task 5 complete
Step 3: User Sees Real-Time Progress
"3/4 external models completed, 1 in progress..."
Best Practices
Do:
✅ Use 4-Message Pattern for true parallel execution
✅ Provide cost estimates BEFORE execution
✅ Ask user approval for costs >$0.01
✅ Auto-trigger consolidation when N ≥ 2 reviews complete
✅ Use blocking (synchronous) claudish execution
✅ Write full output to files, return brief summaries
Pattern 7: Statistics Collection - Track speed, cost, quality per model
Pattern 8: Data-Driven Selection (NEW v3.0) - Intelligent model recommendations
Master this skill and you can validate any implementation with multiple AI perspectives in minutes, while continuously improving your model shortlist based on actual performance data.
