Version: 1.0.0
Purpose: MANDATORY tracking protocol for multi-model validation to prevent incomplete reviews
Production Ready
Status:
Overview
This skill defines the MANDATORY tracking protocol for multi-model validation. It provides templates and procedures that make proper tracking unforgettable.
The Problem This Solves:
Agents often launch multiple external AI models but fail to:
Create structured tracking tables before launch
Collect timing and performance data during execution
Document failures with error messages
Perform consensus analysis comparing model findings
Present results in a structured format
The Solution:
This skill provides MANDATORY checklists, templates, and protocols that ensure complete tracking. Missing ANY of these steps = INCOMPLETE review.
DO NOT wait until all models finish. Update as each completes.
Per-Model Status Update Protocol
IMMEDIATELY After Each Model Completes
Do NOT wait until all models finish. Update tracking AS EACH COMPLETES.
Update Script
# Call this when each model completesupdate_model_status() {
local model="$1"local status="$2"local issues="${3:-0}"local quality="${4:-}"local error="${5:-}"local end_time=$(date +%s)
local start_time="${MODEL_START_TIMES[$model]}"local duration=$((end_time - start_time))
# Update arrays
MODEL_END_TIMES["$model"]=$end_time
MODEL_STATUS["$model"]="$status"# Log update to session tracking fileecho"$(date -u +%Y-%m-%dT%H:%M:%SZ) - Model: $model, Status: $status, Duration: ${duration}s" >> "$SESSION_DIR/execution.log"# Update tracking table (append to tracking.md)echo"| $model | $status | ${duration}s | $issues | ${quality:-N/A} | ${error:-} |" >> "$SESSION_DIR/tracking.md"# Track performance in global statisticsif [[ "$status" == "success" ]]; then
track_model_performance "$model""success""$duration""$issues""$quality"else
track_model_performance "$model""$status""$duration" 0 ""fi
}
# Usage examples:
update_model_status "claude-embedded""success" 8 95
update_model_status "x-ai/grok-code-fast-1""success" 6 87
update_model_status "some-model""timeout" 0 """Exceeded 120s limit"
update_model_status "other-model""failed" 0 """API 500 error"
Status Values
Status
Meaning
Action
pending
Not started
Wait
in_progress
Currently executing
Monitor
success
Completed successfully
Collect results
failed
Error during execution
Document error
timeout
Exceeded time limit
Note timeout
cancelled
User cancelled
Note cancellation
Real-Time Progress Display
Show user progress as models complete:
Model Status (3/5 complete):
✓ claude-embedded (32s, 8 issues)
✓ x-ai/grok-code-fast-1 (45s, 6 issues)
✓ qwen/qwen3-coder:free (52s, 5 issues)
⏳ openai/gpt-5.1-codex (in progress, 60s elapsed)
⏳ google/gemini-3-pro (in progress, 48s elapsed)
Failure Documentation Protocol
EVERY failed model MUST be documented with:
Model name
Failure type (timeout, API error, parse error, etc.)
Error message (exact or summarized)
Whether retry was attempted
Failure Report Template
## Failed Models Report### Model: x-ai/grok-code-fast-1-**Failure Type:** API Error
-**Error Message:** "500 Internal Server Error from OpenRouter"
-**Retry Attempted:** Yes, 1 retry, same error
-**Impact:** Review results based on 3/4 models instead of 4
-**Recommendation:** Check OpenRouter status, retry later
### Model: google/gemini-3-pro-**Failure Type:** Timeout
-**Error Message:** "Exceeded 120s limit, response incomplete"
-**Retry Attempted:** No, time constraints
-**Impact:** Lost Gemini perspective, consensus based on remaining models
-**Recommendation:** Extend timeout to 180s for this model
Symptom: Results presented as prose, not structured data
What went wrong:
"I ran 5 models. 3 succeeded and found various issues."
(No table, no structure)
Prevention:
Always run pre-launch script FIRST
Create $SESSION_DIR/tracking.md before Task calls
Populate table as models complete
Detection: SubagentStop hook warns if no tracking found
Failure 2: Timing Not Recorded
Symptom: "Duration: unknown" or missing speed stats
What went wrong:
# Launched models without recording start time
Task: reviewer1
Task: reviewer2
# No SESSION_START, cannot calculate duration!
Prevention:
# ALWAYS do this first
SESSION_START=$(date +%s)
MODEL_START_TIMES["model1"]=$SESSION_START
Detection: Hook checks for timing data in output
Failure 3: Failed Models Not Documented
Symptom: "2 of 8 succeeded" with no failure details
What went wrong:
"Launched 8 models. 2 succeeded."
(No info on why 6 failed)
Prevention:
# Immediately when model fails
document_failure "model-name""Timeout""Exceeded 120s""No"
Detection: Hook checks for failure section when success < total
Failure 4: No Consensus Analysis
Symptom: Individual model results listed without comparison
What went wrong:
"Model 1 found: A, B, C
Model 2 found: B, D, E"
(No comparison: which issues do they agree on?)
Prevention:
After all complete, ALWAYS run consolidation
Create consensus table comparing findings
Prioritize by agreement level
Detection: Hook checks for consensus keywords
Failure 5: Statistics Not Saved
Symptom: No record in ai-docs/llm-performance.json
What went wrong:
# Forgot to call tracking functions# No record of this session
Prevention:
# ALWAYS call these
track_model_performance "model""status" duration issues quality
record_session_stats total success failed parallel sequential speedup
Detection: Hook checks file modification time
Prevention Checklist
Before presenting results, verify:
[ ] Tracking table exists at $SESSION_DIR/tracking.md
[ ] Tracking table is populated with all model results
[ ] All model times recorded (or "timeout"/"failed" noted)
[ ] All failures documented in $SESSION_DIR/failures.md
[ ] Consensus analysis performed in $SESSION_DIR/consensus.md
[ ] Results match required output format
[ ] Statistics saved to ai-docs/llm-performance.json
[ ] Session directory contains all artifacts
Integration Examples
Example 1: Complete Multi-Model Review Workflow
#!/bin/bash# Full multi-model review with complete tracking# ============================================================================# PHASE 1: PRE-LAUNCH (MANDATORY)# ============================================================================# 1. Create unique session
SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="/tmp/${SESSION_ID}"mkdir -p "$SESSION_DIR"# 2. Record start time
SESSION_START=$(date +%s)
# 3. Create tracking tablecat > "$SESSION_DIR/tracking.md" << EOF
# Multi-Model Validation Tracking
## Session: $SESSION_ID
Started: $(date -u +%Y-%m-%dT%H:%M:%SZ)
## Model Status
| Model | Status | Duration | Issues | Quality |
|-------|--------|----------|--------|---------|
EOF# 4. Initialize timing arraysdeclare -A MODEL_START_TIMES
declare -A MODEL_END_TIMES
# 5. Create tracking markerecho"$SESSION_DIR" > /tmp/.claude-multi-model-active
# 6. Write code context
git diff > "$SESSION_DIR/code-context.md"echo"Pre-launch complete. Session: $SESSION_ID"# ============================================================================# PHASE 2: MODEL EXECUTION (Parallel Task calls)# ============================================================================# Record start times for each model
MODEL_START_TIMES["claude-embedded"]=$(date +%s)
MODEL_START_TIMES["x-ai/grok-code-fast-1"]=$(date +%s)
MODEL_START_TIMES["qwen/qwen3-coder:free"]=$(date +%s)
# Launch all models in single message (parallel execution)# (These would be actual Task calls in practice)echo"Launching 3 models in parallel..."# ============================================================================# PHASE 3: RESULTS COLLECTION (as each completes)# ============================================================================# Update status immediately after each completesupdate_model_status() {
local model="$1" status="$2" issues="${3:-0}" quality="${4:-}"local end_time=$(date +%s)
local duration=$((end_time - MODEL_START_TIMES["$model"]))
echo"| $model | $status | ${duration}s | $issues | ${quality:-N/A} |" >> "$SESSION_DIR/tracking.md"
track_model_performance "$model""$status""$duration""$issues""$quality"
}
# Example completions
update_model_status "claude-embedded""success" 8 95
update_model_status "x-ai/grok-code-fast-1""success" 6 87
update_model_status "qwen/qwen3-coder:free""timeout"# ============================================================================# PHASE 4: CONSENSUS ANALYSIS (MANDATORY)# ============================================================================# Consolidate and compare findingsecho"Performing consensus analysis..."# (Would launch consolidation agent here)# ============================================================================# PHASE 5: STATISTICS & PRESENTATION# ============================================================================# Calculate session stats
PARALLEL_TIME=52 # max of all durations
SEQUENTIAL_TIME=129 # sum of all durations
SPEEDUP=2.5
# Record session
record_session_stats 3 2 1 "$PARALLEL_TIME""$SEQUENTIAL_TIME""$SPEEDUP"# Present resultscat << RESULTS
## Multi-Model Review Complete
Session: $SESSION_ID
Directory: $SESSION_DIR
Models: 3 requested, 2 successful, 1 failed
See tracking table: $SESSION_DIR/tracking.md
See consensus: $SESSION_DIR/consensus.md
Statistics saved to: ai-docs/llm-performance.json
RESULTS# Cleanup markerrm -f /tmp/.claude-multi-model-active
# Multi-model with failure handling# Pre-launch (same as Example 1)# ... setup code ...# Launch 4 models# ... Task calls ...# Model 1: Success
update_model_status "claude""success" 32 8 95
# Model 2: Success
update_model_status "grok""success" 45 6 87
# Model 3: Timeout
update_model_status "gemini""timeout"
document_failure "gemini""Timeout""Exceeded 120s limit""No"# Model 4: API Error
update_model_status "gpt5""failed"
document_failure "gpt5""API Error""500 from OpenRouter""Yes, 1 retry"# Proceed with 2 successful modelsif [ "$SUCCESS_COUNT" -ge 2 ]; thenecho"Proceeding with $SUCCESS_COUNT successful models"# Consensus with partial dataelseecho"ERROR: Only $SUCCESS_COUNT succeeded, need minimum 2"fi
Integration with Other Skills
With multi-model-validation
The multi-model-validation skill defines the execution patterns (4-Message Pattern, parallel execution, proxy mode). This skill (model-tracking-protocol) defines the tracking infrastructure.
This skill provides MANDATORY tracking infrastructure for multi-model validation:
Pre-Launch Checklist - 8 items to complete before launching models
Tracking Tables - Templates for 3-5 models and 6+ models
Status Updates - Per-model completion tracking
Failure Documentation - Required format for all failures
Consensus Analysis - Comparing findings across models
Results Template - Required output format
Common Failures - Prevention strategies
Integration Examples - Complete workflows
Key Innovation: File-based tracking marker (/tmp/.claude-multi-model-active) allows hooks to detect active tracking without relying on environment variables.
Use this skill when: Running 2+ external AI models in parallel for validation, review, or consensus analysis.
