performance-tracking

# Performance Tracking **Version:** 1.0.0 **Purpose:** Track agent, skill, and model performance metrics for continuous optimization **Status:** Production Ready ## Overview Performance tracking transforms workflows from "fire and forget" to **data-driven optimization systems**. By measuring what actually works, you can route tasks more effectively, identify failing patterns early, and reduce costs. This skill provides battle-tested patterns for: - **Agent success tracking** (completion rates, confidence scores, task type affinity) - **Skill effectiveness** (activation counts, success correlation, usage patterns) - **Model performance** (latency, cost, quality, provider comparison) - **Routing optimization** (tier distribution, routing accuracy, cost efficiency) - **Historical analysis** (trend detection, degradation alerts, pattern discovery) Performance tracking enables **continuous improvement** by providing the data needed to make informed decisions about agent selection, model choice, and workflow routing. ### Why Track Performance **Optimize Routing:** - Identify which agents excel at specific task types - Route complex tasks to high-confidence agents - Avoid agents with low success rates for critical work **Identify Failing Agents:** - Detect agents with <70% success rate - Alert when agent performance degrades - Replace or retrain underperforming agents **Reduce Costs:** - Find cost-effective model alternatives - Identify expensive agents with low success rates - Optimize tier thresholds based on actual performance **Improve Quality:** - Track correlation between confidence scores and success - Identify patterns in successful implementations - Learn which models produce best results for task types ### What We Track **Agent Metrics:** - Total runs, success/failure counts - Average confidence scores - Task type distribution - Last used timestamp - Individual execution history **Skill Metrics:** - Activation counts per skill - Last activation timestamps - Success correlation (when skill active, what's success rate?) - Co-activation patterns **Model Metrics:** - Total runs, success/failure counts - Average latency (response time) - Total cost (cumulative spend) - Cost per successful task - Last used timestamp **Routing Metrics:** - Tier distribution (how often each tier selected) - Routing accuracy (did tier match complexity?) - Cost efficiency (tier1 vs tier4 cost ratio) - Decision history with outcomes ### Integration with task-complexity-router The performance tracker provides critical feedback to the task-complexity-router: ``` Routing Feedback Loop: 1. Router selects tier based on complexity → task-complexity-router analyzes task → Routes to tier2 (medium complexity) 2. Agent executes task → Records: tier=2, agent=ui-developer, result=success 3. Performance tracker updates metrics → tier2 usage +1 → ui-developer success +1 → Confidence in tier2 routing increases 4. Future routing decisions informed by history → Router sees tier2 has 85% success rate → Router sees ui-developer excels at UI tasks → Router confidently routes similar tasks to tier2 ``` --- ## Metrics Schema ### JSON Structure (Version 1.0.0) Store performance metrics in `.claude/agent-performance.json`: ```json { "schemaVersion": "1.0.0", "lastUpdated": "2026-01-28T15:30:00Z", "agents": { "ui-developer": { "totalRuns": 42, "successCount": 38, "failureCount": 4, "avgConfidence": 0.85, "lastUsed": "2026-01-28T15:30:00Z", "taskTypes": { "implement-component": 15, "fix-styling": 12, "refactor-ui": 8, "review-code": 7 }, "history": [ { "timestamp": "2026-01-28T15:30:00Z", "taskType": "implement-component", "result": "success", "confidence": 0.90, "duration": 45000, "tier": 2, "model": "claude-sonnet-4-5-20250929" }, { "timestamp": "2026-01-28T14:20:00Z", "taskType": "fix-styling", "result": "success", "confidence": 0.85, "duration": 30000, "tier": 1, "model": "claude-sonnet-4-5-20250929" } ] }, "backend-developer": { "totalRuns": 28, "successCount": 25, "failureCount": 3, "avgConfidence": 0.88, "lastUsed": "2026-01-28T14:00:00Z", "taskTypes": { "implement-api": 12, "fix-bug": 8, "database-migration": 5, "write-tests": 3 }, "history": [] } }, "skills": { "multi-model-validation": { "activations": 15, "lastActivated": "2026-01-28T15:00:00Z", "successCorrelation": 0.92, "coActivations": { "quality-gates": 12, "error-recovery": 8 } }, "task-complexity-router": { "activations": 68, "lastActivated": "2026-01-28T15:30:00Z", "successCorrelation": 0.85, "coActivations": { "multi-agent-coordination": 45, "hierarchical-coordinator": 30 } } }, "models": { "claude-sonnet-4-5-20250929": { "totalRuns": 120, "successCount": 108, "failureCount": 12, "avgLatency": 2500, "totalCost": 0.45, "lastUsed": "2026-01-28T15:30:00Z", "taskTypePerformance": { "code-review": { "success": 25, "failure": 2 }, "implementation": { "success": 40, "failure": 5 }, "testing": { "success": 20, "failure": 3 } } }, "x-ai/grok-code-fast-1": { "totalRuns": 35, "successCount": 30, "failureCount": 5, "avgLatency": 1800, "totalCost": 0.08, "lastUsed": "2026-01-28T13:00:00Z", "taskTypePerformance": { "code-review": { "success": 18, "failure": 2 }, "implementation": { "success": 12, "failure": 3 } } } }, "routing": { "tierDistribution": { "tier1": 45, "tier2": 30, "tier3": 15, "tier4": 8 }, "decisions": [ { "timestamp": "2026-01-28T15:30:00Z", "taskType": "implement-component", "complexity": "medium", "selectedTier": 2, "agent": "ui-developer", "result": "success", "cost": 0.003 }, { "timestamp": "2026-01-28T14:20:00Z", "taskType": "fix-styling", "complexity": "low", "selectedTier": 1, "agent": "ui-developer", "result": "success", "cost": 0.001 } ] } } ``` ### Schema Field Definitions **Agent Metrics:** - `totalRuns`: Total task executions - `successCount`: Tasks completed successfully - `failureCount`: Tasks that failed or required retry - `avgConfidence`: Rolling average of agent confidence scores (0.0-1.0) - `lastUsed`: ISO-8601 timestamp of last execution - `taskTypes`: Distribution of task types (understand agent specialization) - `history`: Array of recent executions (max 100 entries, FIFO) **Skill Metrics:** - `activations`: Total times skill was triggered - `lastActivated`: ISO-8601 timestamp - `successCorrelation`: Success rate when this skill is active (0.0-1.0) - `coActivations`: Skills frequently activated together (detect patterns) **Model Metrics:** - `totalRuns`: Total executions - `successCount`/`failureCount`: Outcome tracking - `avgLatency`: Average response time in milliseconds - `totalCost`: Cumulative spend in USD - `lastUsed`: ISO-8601 timestamp - `taskTypePerformance`: Success/failure breakdown by task type **Routing Metrics:** - `tierDistribution`: Count of tasks routed to each tier - `decisions`: Array of routing decisions with outcomes (max 100, FIFO) --- ## Tracking Patterns ### Pattern 1: Capturing Agent Performance **After Agent Completes Task:** ``` Execution Flow: 1. Agent executes task Task: ui-developer Input: "Implement login form component" Result: Success Confidence: 0.90 Duration: 45 seconds Tier: 2 Model: claude-sonnet-4-5-20250929 2. Update agent metrics Read: .claude/agent-performance.json Update: agents["ui-developer"].totalRuns += 1 agents["ui-developer"].successCount += 1 agents["ui-developer"].avgConfidence = rolling_avg(0.90) agents["ui-developer"].lastUsed = NOW agents["ui-developer"].taskTypes["implement-component"] += 1 agents["ui-developer"].history.push({ timestamp: NOW, taskType: "implement-component", result: "success", confidence: 0.90, duration: 45000, tier: 2, model: "claude-sonnet-4-5-20250929" }) Trim history if > 100 entries Write: .claude/agent-performance.json 3. Calculate derived metrics Success rate: successCount / totalRuns = 38/42 = 90.5% Avg duration: sum(history.duration) / history.length Task affinity: taskTypes sorted by count ``` **After Agent Fails:** ``` Failure Flow: 1. Agent fails task Task: backend-developer Input: "Implement complex payment flow" Result: Failure (error, timeout, or low quality) Confidence: 0.65 Tier: 3 2. Update failure metrics agents["backend-developer"].totalRuns += 1 agents["backend-developer"].failureCount += 1 agents["backend-developer"].avgConfidence = rolling_avg(0.65) agents["backend-developer"].history.push({ timestamp: NOW, taskType: "implement-api", result: "failure", confidence: 0.65, duration: 120000, tier: 3, error: "Exceeded max iterations" }) 3. Check for degradation If failureCount / totalRuns > 0.30: Alert: "backend-developer has >30% failure rate" Recommendation: "Review recent failures, retrain, or replace" ``` ### Pattern 2: Tracking Model Performance **After Model Execution:** ``` Execution Flow: 1. Model completes task Model: x-ai/grok-code-fast-1 Task: Code review Latency: 1800ms Cost: $0.002 Result: Success 2. Update model metrics models["x-ai/grok-code-fast-1"].totalRuns += 1 models["x-ai/grok-code-fast-1"].successCount += 1 models["x-ai/grok-code-fast-1"].avgLatency = rolling_avg(1800) models["x-ai/grok-code-fast-1"].totalCost += 0.002 models["x-ai/grok-code-fast-1"].lastUsed = NOW models["x-ai/grok-code-fast-1"].taskTypePerformance["code-review"].success += 1 3. Compare model performance Claude Sonnet: avgLatency=2500ms, cost=$0.45 (120 runs) Grok Fast: avgLatency=1800ms, cost=$0.08 (35 runs) Analysis: - Grok is 28% faster (1800ms vs 2500ms) - Grok is 82% cheaper per run ($0.0023 vs $0.0038) - Both have similar success rates (86% vs 90%) Recommendation: - Use Grok for cost-sensitive tasks - Use Claude for critical tasks (higher success rate) ``` ### Pattern 3: Recording Skill Activation **After Skill Activation:** ``` Activation Flow: 1. Skill triggers Skill: multi-model-validation Context: User requested /review with 3 models 2. Update skill metrics skills["multi-model-validation"].activations += 1 skills["multi-model-validation"].lastActivated = NOW 3. Track co-activation Active skills: ["multi-model-validation", "quality-gates"] skills["multi-model-validation"].coActivations["quality-gates"] += 1 4. Calculate success correlation Tasks with this skill active: 15 Successful tasks: 14 Success correlation: 14/15 = 93.3% 5. Pattern detection Observation: multi-model-validation + quality-gates = 100% success (12/12) Recommendation: Always pair these skills for high-quality reviews ``` ### Pattern 4: Routing Decision Tracking **After Routing Decision:** ``` Routing Flow: 1. Router selects tier Task: "Implement user profile page" Analysis: Medium complexity (multiple components, state management) Selected tier: 2 Agent: ui-developer Model: claude-sonnet-4-5-20250929 2. Record routing decision routing.tierDistribution["tier2"] += 1 routing.decisions.push({ timestamp: NOW, taskType: "implement-component", complexity: "medium", selectedTier: 2, agent: "ui-developer", result: "pending" }) 3. After task completes Update decision with result: routing.decisions[last].result = "success"