model-tracking-protocol

# Model Tracking Protocol **Version:** 1.0.0 **Purpose:** MANDATORY tracking protocol for multi-model validation to prevent incomplete reviews **Status:** Production Ready ## Overview This skill defines the MANDATORY tracking protocol for multi-model validation. It provides templates and procedures that make proper tracking unforgettable. **The Problem This Solves:** Agents often launch multiple external AI models but fail to: - Create structured tracking tables before launch - Collect timing and performance data during execution - Document failures with error messages - Perform consensus analysis comparing model findings - Present results in a structured format **The Solution:** This skill provides MANDATORY checklists, templates, and protocols that ensure complete tracking. Missing ANY of these steps = INCOMPLETE review. --- ## Table of Contents 1. [MANDATORY Pre-Launch Checklist](#mandatory-pre-launch-checklist) 2. [Tracking Table Templates](#tracking-table-templates) 3. [Per-Model Status Updates](#per-model-status-updates) 4. [Failure Documentation Protocol](#failure-documentation-protocol) 5. [Consensus Analysis Requirements](#consensus-analysis-requirements) 6. [Results Presentation Template](#results-presentation-template) 7. [Common Failures and Prevention](#common-failures-and-prevention) 8. [Integration Examples](#integration-examples) --- ## MANDATORY Pre-Launch Checklist **You MUST complete ALL items before launching ANY external models.** This is NOT optional. If you skip this, your multi-model validation is INCOMPLETE. ### Checklist (Copy and Complete) ``` PRE-LAUNCH VERIFICATION (complete before Task calls): [ ] 1. SESSION_ID created: ________________________ [ ] 2. SESSION_DIR created: ________________________ [ ] 3. Tracking table written to: $SESSION_DIR/tracking.md [ ] 4. Start time recorded: SESSION_START=$(date +%s) [ ] 5. Model list confirmed (comma-separated): ________________________ [ ] 6. Per-model timing arrays initialized [ ] 7. Code context written to session directory [ ] 8. Tracking marker created: /tmp/.claude-multi-model-active If ANY item is unchecked, STOP and complete it before proceeding. ``` ### Why Pre-Launch Matters Without pre-launch setup, you will: - Lose timing data (cannot calculate speed accurately) - Miss failed model details (no structured place to record) - Skip consensus analysis (no model list to compare) - Present incomplete results (no tracking table to populate) ### Pre-Launch Script Template **CRITICAL CONSENSUS FIX APPLIED:** Use file-based detection instead of environment variables. ```bash #!/bin/bash # Run this BEFORE launching any Task calls # 1. Create unique session SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)" SESSION_DIR="/tmp/${SESSION_ID}" mkdir -p "$SESSION_DIR" # 2. Record start time SESSION_START=$(date +%s) # 3. Create tracking table cat > "$SESSION_DIR/tracking.md" << EOF # Multi-Model Tracking ## Session Info - Session ID: ${SESSION_ID} - Started: $(date -u +%Y-%m-%dT%H:%M:%SZ) - Models Requested: [FILL] ## Model Status | Model | Agent ID | Status | Start | End | Duration | Issues | Quality | Notes | |-------|----------|--------|-------|-----|----------|--------|---------|-------| | [MODEL 1] | | pending | | | | | | | | [MODEL 2] | | pending | | | | | | | | [MODEL 3] | | pending | | | | | | | ## Failures | Model | Failure Type | Error Message | Retry? | |-------|--------------|---------------|--------| ## Consensus | Issue | Model 1 | Model 2 | Model 3 | Agreement | |-------|---------|---------|---------|-----------| EOF # 4. Initialize timing arrays declare -A MODEL_START_TIMES declare -A MODEL_END_TIMES declare -A MODEL_STATUS # 5. Create tracking marker file (CRITICAL FIX) # This allows hooks to detect that tracking is active echo "$SESSION_DIR" > /tmp/.claude-multi-model-active echo "Pre-launch setup complete. Session: $SESSION_ID" echo "Directory: $SESSION_DIR" echo "Tracking table: $SESSION_DIR/tracking.md" ``` ### Strict Mode (Optional) For stricter enforcement, set: ```bash export CLAUDE_STRICT_TRACKING=true ``` When enabled, hooks will BLOCK execution if tracking is not set up, rather than just warning. --- ## Tracking Table Templates ### Template A: Simple Model Tracking (3-5 models) ```markdown | Model | Status | Time | Issues | Quality | Cost | |-------|--------|------|--------|---------|------| | claude-embedded | pending | - | - | - | FREE | | x-ai/grok-code-fast-1 | pending | - | - | - | - | | qwen/qwen3-coder:free | pending | - | - | - | FREE | ``` **Update as each completes:** ```markdown | Model | Status | Time | Issues | Quality | Cost | |-------|--------|------|--------|---------|------| | claude-embedded | success | 32s | 8 | 95% | FREE | | x-ai/grok-code-fast-1 | success | 45s | 6 | 87% | $0.002 | | qwen/qwen3-coder:free | timeout | - | - | - | - | ``` ### Template B: Detailed Model Tracking (6+ models) ```markdown ## Model Execution Status ### Summary - Total Requested: 8 - Completed: 0 - In Progress: 0 - Failed: 0 - Pending: 8 ### Detailed Status | # | Model | Provider | Status | Start | Duration | Issues | Quality | Cost | Error | |---|-------|----------|--------|-------|----------|--------|---------|------|-------| | 1 | claude-embedded | Anthropic | pending | - | - | - | - | FREE | - | | 2 | x-ai/grok-code-fast-1 | X-ai | pending | - | - | - | - | - | - | | 3 | qwen/qwen3-coder:free | Qwen | pending | - | - | - | - | FREE | - | | 4 | google/gemini-3-pro | Google | pending | - | - | - | - | - | - | | 5 | openai/gpt-5.1-codex | OpenAI | pending | - | - | - | - | - | - | | 6 | mistralai/devstral | Mistral | pending | - | - | - | - | FREE | - | | 7 | deepseek/deepseek-r1 | DeepSeek | pending | - | - | - | - | - | - | | 8 | anthropic/claude-sonnet | Anthropic | pending | - | - | - | - | - | - | ``` ### Template C: Session-Based Tracking File Create this file at `$SESSION_DIR/tracking.md`: ```markdown # Multi-Model Validation Tracking Session: ${SESSION_ID} Started: ${TIMESTAMP} ## Pre-Launch Verification - [x] Session directory created: ${SESSION_DIR} - [x] Tracking table initialized - [x] Start time recorded: ${SESSION_START} - [x] Model list: ${MODEL_LIST} ## Model Status | Model | Status | Start | Duration | Issues | Quality | |-------|--------|-------|----------|--------|---------| | claude | pending | - | - | - | - | | grok | pending | - | - | - | - | | gemini | pending | - | - | - | - | ## Failures (populated as failures occur) ## Consensus (populated after all complete) ``` ### Update Protocol As each model completes, IMMEDIATELY update: 1. Status: `pending` -> `in_progress` -> `success`/`failed`/`timeout` 2. Duration: Calculate from start time 3. Issues: Number of issues found 4. Quality: Percentage if calculable 5. Error: If failed, brief error message **DO NOT wait until all models finish.** Update as each completes. --- ## Per-Model Status Update Protocol ### IMMEDIATELY After Each Model Completes Do NOT wait until all models finish. Update tracking AS EACH COMPLETES. ### Update Script ```bash # Call this when each model completes update_model_status() { local model="$1" local status="$2" local issues="${3:-0}" local quality="${4:-}" local error="${5:-}" local end_time=$(date +%s) local start_time="${MODEL_START_TIMES[$model]}" local duration=$((end_time - start_time)) # Update arrays MODEL_END_TIMES["$model"]=$end_time MODEL_STATUS["$model"]="$status" # Log update to session tracking file echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) - Model: $model, Status: $status, Duration: ${duration}s" >> "$SESSION_DIR/execution.log" # Update tracking table (append to tracking.md) echo "| $model | $status | ${duration}s | $issues | ${quality:-N/A} | ${error:-} |" >> "$SESSION_DIR/tracking.md" # Track performance in global statistics if [[ "$status" == "success" ]]; then track_model_performance "$model" "success" "$duration" "$issues" "$quality" else track_model_performance "$model" "$status" "$duration" 0 "" fi } # Usage examples: update_model_status "claude-embedded" "success" 8 95 update_model_status "x-ai/grok-code-fast-1" "success" 6 87 update_model_status "some-model" "timeout" 0 "" "Exceeded 120s limit" update_model_status "other-model" "failed" 0 "" "API 500 error" ``` ### Status Values | Status | Meaning | Action |