Track agent, skill, and model performance metrics for optimization. Use when measuring agent success rates, tracking model latency, analyzing routing effectiveness, or optimizing cost-per-task.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Track agent, skill, and model performance metrics for optimization. Use when measuring agent success rates, tracking model latency, analyzing routing effectiveness, or optimizing cost-per-task.
disable-model-invocation
true
Performance Tracking
Version: 1.0.0
Purpose: Track agent, skill, and model performance metrics for continuous optimization
Status: Production Ready
Overview
Performance tracking transforms workflows from "fire and forget" to data-driven optimization systems. By measuring what actually works, you can route tasks more effectively, identify failing patterns early, and reduce costs.
This skill provides battle-tested patterns for:
Agent success tracking (completion rates, confidence scores, task type affinity)
Performance tracking enables continuous improvement by providing the data needed to make informed decisions about agent selection, model choice, and workflow routing.
Why Track Performance
Optimize Routing:
Identify which agents excel at specific task types
Route complex tasks to high-confidence agents
Avoid agents with low success rates for critical work
Identify Failing Agents:
Detect agents with <70% success rate
Alert when agent performance degrades
Replace or retrain underperforming agents
Reduce Costs:
Find cost-effective model alternatives
Identify expensive agents with low success rates
Optimize tier thresholds based on actual performance
Improve Quality:
Track correlation between confidence scores and success
Identify patterns in successful implementations
Learn which models produce best results for task types
Execution Flow:
1. Model completes task
Model: grok
Task: Code review
Latency: 1800ms
Cost: $0.002
Result: Success
2. Update model metrics
models["grok"].totalRuns += 1
models["grok"].successCount += 1
models["grok"].avgLatency = rolling_avg(1800)
models["grok"].totalCost += 0.002
models["grok"].lastUsed = NOW
models["grok"].taskTypePerformance["code-review"].success += 1
3. Compare model performance
Claude Sonnet: avgLatency=2500ms, cost=$0.45 (120 runs)
Grok Fast: avgLatency=1800ms, cost=$0.08 (35 runs)
Analysis:
- Grok is 28% faster (1800ms vs 2500ms)
- Grok is 82% cheaper per run ($0.0023 vs $0.0038)
- Both have similar success rates (86% vs 90%)
Recommendation:
- Use Grok for cost-sensitive tasks
- Use Claude for critical tasks (higher success rate)
Pattern 3: Recording Skill Activation
After Skill Activation:
Activation Flow:
1. Skill triggers
Skill: multi-model-validation
Context: User requested /review with 3 models
2. Update skill metrics
skills["multi-model-validation"].activations += 1
skills["multi-model-validation"].lastActivated = NOW
3. Track co-activation
Active skills: ["multi-model-validation", "quality-gates"]
skills["multi-model-validation"].coActivations["quality-gates"] += 1
4. Calculate success correlation
Tasks with this skill active: 15
Successful tasks: 14
Success correlation: 14/15 = 93.3%
5. Pattern detection
Observation: multi-model-validation + quality-gates = 100% success (12/12)
Recommendation: Always pair these skills for high-quality reviews
Pattern 4: Routing Decision Tracking
After Routing Decision:
Routing Flow:
1. Router selects tier
Task: "Implement user profile page"
Analysis: Medium complexity (multiple components, state management)
Selected tier: 2
Agent: ui-developer
Model: sonnet
2. Record routing decision
routing.tierDistribution["tier2"] += 1
routing.decisions.push({
timestamp: NOW,
taskType: "implement-component",
complexity: "medium",
selectedTier: 2,
agent: "ui-developer",
result: "pending"
})
3. After task completes
Update decision with result:
routing.decisions[last].result = "success"
routing.decisions[last].cost = 0.003
4. Trim decision history if > 100 entries
Comparison:
llm-performance.json (existing):
- Model-specific performance
- Cost tracking per model
- Response time tracking
- Used by multi-model-validation
agent-performance.json (new):
- Agent-level metrics (multi-run aggregation)
- Skill activation tracking
- Routing decision history
- Task type affinity
Integration:
- agent-performance.json imports model data from llm-performance.json
- Both files updated in parallel
- llm-performance.json focuses on single-run details
- agent-performance.json focuses on aggregate trends
Migration Path:
Step 1: Create .claude/agent-performance.json with schema 1.0.0
Step 2: Import historical data from llm-performance.json
Step 3: Update both files going forward
Step 4: Deprecate llm-performance.json after 6 months (optional)
Data Cleanup and Rotation
Automatic Cleanup:
Cleanup Rules:
1. Agent history arrays
Max entries: 100
Strategy: FIFO (oldest removed first)
Trigger: After every agent execution
2. Routing decision arrays
Max entries: 100
Strategy: FIFO
Trigger: After every routing decision
3. Session snapshots
Max files: 100
Strategy: FIFO (delete oldest session files)
Trigger: After every session ends
4. Skill co-activation maps
Max entries per skill: 50
Strategy: Keep top 50 by count
Trigger: Weekly cleanup
Manual Cleanup:
When to manually reset:
1. After major workflow changes
- Agent capabilities changed
- New skills added
- Routing logic updated
โ Reset metrics to start fresh
2. After agent retraining
- Agent prompt updated
- Agent model changed
โ Reset agent-specific metrics
3. After prolonged period (>6 months)
- Metrics may be outdated
โ Archive old data, start fresh
How to reset:
Backup: cp .claude/agent-performance.json .claude/agent-performance-backup-{DATE}.json
Reset: echo '{"schemaVersion":"1.0.0","lastUpdated":"...","agents":{},...}' > .claude/agent-performance.json
For test-architect (60% success):
1. Analyze failure patterns
Review history entries where result="failure"
Common failure reasons:
- "Tests too brittle" (8 occurrences)
- "Missing test coverage" (5 occurrences)
- "Test timeout" (2 occurrences)
2. Identify root cause
Pattern: test-architect struggles with async/timing tests
Evidence: All timeout failures involved async code
3. Take action
Option A: Retrain agent
- Update prompt with async testing best practices
- Add examples of proper async test patterns
- Reset metrics after retraining
Option B: Route differently
- Route async test tasks to backend-developer (90% success on async)
- Keep test-architect for synchronous unit tests
Option C: Replace agent
- Create new specialized-async-test-architect
- Deprecate test-architect for async work
Optimization 2: Find Cost-Effective Model Alternatives
Analysis:
Compare model cost-effectiveness:
Model: sonnet
Total cost: $0.45
Total runs: 120
Success count: 108
Cost per task: $0.0038
Cost per success: $0.0042
Success rate: 90%
Model: grok
Total cost: $0.08
Total runs: 35
Success count: 30
Cost per task: $0.0023
Cost per success: $0.0027
Success rate: 86%
Model: gemini
Total cost: $0.02
Total runs: 20
Success count: 16
Cost per task: $0.0010
Cost per success: $0.0013
Success rate: 80%
Cost Efficiency Ranking:
1. Gemini Flash: $0.0013 per success (80% success rate)
2. Grok Fast: $0.0027 per success (86% success rate)
3. Claude Sonnet: $0.0042 per success (90% success rate)
Quality-Cost Tradeoff:
- Gemini: 69% cheaper than Claude, but 10% lower success rate
- Grok: 36% cheaper than Claude, but 4% lower success rate
Action:
Optimization strategy:
Tier 1 (Simple tasks):
Use: Gemini Flash
Reason: Lowest cost, acceptable success rate for simple work
Example: "Fix typo in comment", "Format code"
Tier 2 (Medium tasks):
Use: Grok Fast
Reason: Good balance of cost and quality
Example: "Implement CRUD endpoint", "Add validation"
Tier 3 (Complex tasks):
Use: Claude Sonnet
Reason: Highest success rate justifies cost
Example: "Design architecture", "Complex refactoring"
Tier 4 (Critical tasks):
Use: Claude Sonnet + Multi-model validation
Reason: Quality > cost for critical work
Example: "Security review", "Production bug fix"
Expected savings:
Current: 90% Claude usage ร $0.0042 = $0.00378 avg per task
Optimized: 20% Claude + 50% Grok + 30% Gemini = $0.00257 avg per task
Savings: 32% cost reduction with minimal quality impact
Performance comparison:
Fastest: Gemini (1500ms) - 50% faster than Claude
Most thorough: Claude (5 issues) - Found 1 extra issue
Best value: Gemini ($0.001, 3 issues) - Lowest cost, good coverage
Cost analysis:
Total: $0.006 for 3-model review
vs Single Claude: $0.003 (double cost, but 2x validation)
ROI: Found 2 CRITICAL issues all models agreed on = high confidence
Consensus validation:
UNANIMOUS issues (100% confidence) โ Fix immediately
MAJORITY issues (67% confidence) โ Fix before merge
DIVERGENT issues (33% confidence) โ Low priority (possible false positive)
Recommendation:
Multi-model validation worth the cost for critical code (auth, payments, security)
Single-model sufficient for non-critical code (UI components, docs)
Example 2: Identifying Model Performance Differences
Scenario: After 100 tasks, compare model performance for optimization
Execution:
Step 1: Load performance data
Read: .claude/agent-performance.json
Step 2: Extract model metrics
Claude Sonnet:
Total runs: 120
Success: 108, Failure: 12
Success rate: 90.0%
Avg latency: 2500ms
Total cost: $0.45
Cost per task: $0.00375
Cost per success: $0.00417
Grok Fast:
Total runs: 35
Success: 30, Failure: 5
Success rate: 85.7%
Avg latency: 1800ms
Total cost: $0.08
Cost per task: $0.00229
Cost per success: $0.00267
Gemini Flash:
Total runs: 20
Success: 16, Failure: 4
Success rate: 80.0%
Avg latency: 1500ms
Total cost: $0.02
Cost per task: $0.00100
Cost per success: $0.00125
Step 3: Analyze task type performance
Code Review:
Claude: 25 success, 2 failure = 92.6%
Grok: 18 success, 2 failure = 90.0%
Gemini: 10 success, 4 failure = 71.4%
Winner: Claude (highest quality)
Best value: Grok (90% at lower cost)
Implementation:
Claude: 40 success, 5 failure = 88.9%
Grok: 12 success, 3 failure = 80.0%
Gemini: 6 success, 1 failure = 85.7%
Winner: Claude (highest quality)
Surprising: Gemini performs well here (86% success)
Testing:
Claude: 20 success, 3 failure = 87.0%
Grok: No data
Gemini: No data
Winner: Claude (only option)
Action: Try Grok/Gemini for testing tasks to gather data
Step 4: Calculate cost-effectiveness by task type
Code Review Cost-Effectiveness:
Claude: $0.00417 per success, 92.6% quality
Grok: $0.00267 per success, 90.0% quality (36% cheaper, -2.6% quality)
Gemini: $0.00125 per success, 71.4% quality (70% cheaper, -21.2% quality)
Recommendation: Use Grok for cost-effective reviews (minimal quality loss)
Implementation Cost-Effectiveness:
Claude: $0.00417 per success, 88.9% quality
Grok: $0.00267 per success, 80.0% quality (36% cheaper, -8.9% quality)
Gemini: $0.00125 per success, 85.7% quality (70% cheaper, -3.2% quality)
Recommendation: Use Gemini for simple implementation (best value)
Step 5: Generate optimization plan
Current usage (120 total tasks):
Claude: 100 tasks (83%)
Grok: 15 tasks (13%)
Gemini: 5 tasks (4%)
Optimized usage (maintain quality >85%):
tier1 (Simple): Gemini (30% of tasks)
tier2 (Medium): Grok (40% of tasks)
tier3 (Complex): Claude (25% of tasks)
tier4 (Critical): Claude + Multi-model (5% of tasks)
Expected impact:
Current avg cost: $0.00375 per task
Optimized avg cost: $0.00240 per task
Savings: 36% cost reduction
Current avg success: 88.5%
Optimized avg success: 86.2% (projected)
Quality impact: -2.3% (acceptable tradeoff)
Step 6: Implement gradual rollout
Week 1: Route 20% of tier1 tasks to Gemini
Monitor: Success rate, cost savings
Target: >80% success rate
Week 2: Route 40% of tier2 tasks to Grok
Monitor: Success rate, cost savings
Target: >85% success rate
Week 3: Evaluate results
If successful: Increase percentages
If unsuccessful: Rollback and investigate
Step 7: Track optimization results
After 2 weeks:
Gemini tier1 success: 82% โ (above 80% target)
Grok tier2 success: 87% โ (above 85% target)
Cost savings: 28% โ (approaching 36% target)
Decision: Continue rollout
Next: Route 50% tier1 to Gemini, 60% tier2 to Grok
Insights from Analysis:
Key findings:
1. Grok is best value for code reviews (90% quality at 36% lower cost)
2. Gemini surprisingly good for implementation (86% vs 89% Claude)
3. Claude still best for critical work (92% code review success)
4. Latency varies significantly (Gemini 40% faster than Claude)
Optimization strategy:
- Use Gemini for simple, latency-sensitive tasks
- Use Grok for medium-complexity, cost-sensitive tasks
- Use Claude for critical, quality-sensitive tasks
- Use multi-model for maximum confidence (despite cost)
Expected ROI:
- 36% cost reduction (from $0.00375 to $0.00240 per task)
- 2.3% quality tradeoff (from 88.5% to 86.2% success)
- Worth it: Save $135 per 100,000 tasks with minimal quality impact
Example 3: Optimizing Routing Based on Accumulated Data
Scenario: After 100 routing decisions, optimize tier thresholds
Execution:
Step 1: Load routing data
Read: .claude/agent-performance.json
Focus: routing.tierDistribution, routing.decisions
Step 2: Analyze tier distribution
Current distribution:
tier1: 45 tasks (45.9%)
tier2: 30 tasks (30.6%)
tier3: 15 tasks (15.3%)
tier4: 8 tasks (8.2%)
Skew analysis:
Heavy on tier1 (46%) - Router prefers simple classification
Light on tier4 (8%) - Router rarely escalates
Step 3: Calculate tier success rates
tier1 (Simple tasks):
Total: 45
Success: 42, Failure: 3
Success rate: 93.3% โ
Avg cost: $0.001
Avg duration: 25s
tier2 (Medium tasks):
Total: 30
Success: 25, Failure: 5
Success rate: 83.3% โ ๏ธ (target: 90%)
Avg cost: $0.002
Avg duration: 45s
tier3 (Complex tasks):
Total: 15
Success: 12, Failure: 3
Success rate: 80.0% โ ๏ธ (target: 90%)
Avg cost: $0.004
Avg duration: 90s
tier4 (Critical tasks):
Total: 8
Success: 8, Failure: 0
Success rate: 100% โ
Avg cost: $0.008
Avg duration: 120s
Step 4: Analyze tier2/tier3 failures
tier2 failures (5 tasks):
1. "Implement complex state management" (complexity: 6)
- Should have been tier3 (underestimated)
2. "Add authentication to API" (complexity: 6)
- Should have been tier3 (security = critical)
3. "Refactor component with hooks" (complexity: 5)
- Should have been tier2 (correctly routed, agent issue)
4. "Implement drag-and-drop" (complexity: 6)
- Should have been tier3 (complex interaction)
5. "Add real-time updates" (complexity: 6)
- Should have been tier3 (WebSocket complexity)
Pattern: 4/5 failures were borderline tier2/tier3 (complexity 6)
Root cause: tier2 upper threshold too high (should be 5, not 6)
tier3 failures (3 tasks):
1. "Design microservices architecture" (complexity: 9)
- Should have been tier4 (architecture = critical)
2. "Implement payment processing" (complexity: 9)
- Should have been tier4 (money = critical)
3. "Refactor authentication system" (complexity: 8)
- Correctly routed, agent struggled with complexity
Pattern: 2/3 failures should have been tier4 (complexity 9)
Root cause: tier3 upper threshold too high (should be 8, not 9)
Step 5: Propose threshold adjustments
Current thresholds:
tier1: complexity 0-3
tier2: complexity 4-6
tier3: complexity 7-9
tier4: complexity 10+
Problem: Borderline tasks (6, 9) cause failures
Optimized thresholds:
tier1: complexity 0-2 (narrower, more confident)
tier2: complexity 3-5 (shift down, avoid borderline 6)
tier3: complexity 6-8 (shift down, avoid borderline 9)
tier4: complexity 9+ (broader, include borderline cases)
Rationale:
- Move borderline complexity 6 from tier2 โ tier3
- Move borderline complexity 9 from tier3 โ tier4
- Accept 15% higher cost for 10% better success rate
Step 6: Simulate new distribution
Reclassify historical tasks with new thresholds:
tier1 (0-2): 35 tasks (35%)
Success rate: 34/35 = 97.1% โ (was 93.3%)
tier2 (3-5): 32 tasks (32%)
Success rate: 30/32 = 93.8% โ (was 83.3%)
tier3 (6-8): 18 tasks (18%)
Success rate: 17/18 = 94.4% โ (was 80.0%)
tier4 (9+): 13 tasks (13%)
Success rate: 13/13 = 100% โ (was 100%)
Overall success rate: 94/98 = 95.9% โ (was 87.8%)
Step 7: Calculate cost impact
Current avg cost: $0.00240 per task
Optimized avg cost: $0.00276 per task (+15%)
Cost breakdown:
tier1 (35%): $0.001 ร 0.35 = $0.00035
tier2 (32%): $0.002 ร 0.32 = $0.00064
tier3 (18%): $0.004 ร 0.18 = $0.00072
tier4 (13%): $0.008 ร 0.13 = $0.00104
Total: $0.00275 (rounded $0.00276)
ROI calculation:
Cost increase: +$0.00036 per task (+15%)
Success increase: +8.1% (from 87.8% to 95.9%)
Failure reduction: 12 โ 4 failures (67% reduction)
Value: Preventing 8 failures per 100 tasks worth the 15% cost increase
Step 8: Implement new thresholds
Update task-complexity-router skill:
OLD:
if (complexity <= 3) return "tier1";
if (complexity <= 6) return "tier2";
if (complexity <= 9) return "tier3";
return "tier4";
NEW:
if (complexity <= 2) return "tier1";
if (complexity <= 5) return "tier2";
if (complexity <= 8) return "tier3";
return "tier4";
Document change:
Reason: Performance data showed borderline tasks caused failures
Expected: 8% success rate improvement, 15% cost increase
Monitoring: Track next 100 tasks to validate improvement
Step 9: Monitor post-optimization
After 50 tasks with new thresholds:
tier1: 18 tasks, 18 success = 100% โ
tier2: 16 tasks, 15 success = 93.8% โ
tier3: 10 tasks, 9 success = 90.0% โ
tier4: 6 tasks, 6 success = 100% โ
Overall: 48/50 = 96.0% success โ (matches projection)
Avg cost: $0.00280 โ (matches projection)
Verdict: Optimization successful, keep new thresholds
Insights from Optimization:
Key findings:
1. Borderline complexity scores (6, 9) caused most failures
2. Router was too aggressive in keeping tasks at lower tiers
3. Small threshold adjustments (6โ5, 9โ8) had big impact
Optimization results:
- Success rate: 87.8% โ 96.0% (+8.2%)
- Failure rate: 12.2% โ 4.0% (-67%)
- Cost per task: $0.00240 โ $0.00280 (+15%)
- ROI: Strong (quality improvement worth cost increase)
Lessons learned:
- Track tier success rates, not just overall success
- Borderline cases benefit from tier escalation
- Performance data reveals routing blind spots
- Continuous monitoring enables iterative improvement
Next steps:
- Continue monitoring for 100 more tasks
- Consider dynamic thresholds (adjust based on live data)
- Explore agent-specific routing (some agents handle complexity better)
Troubleshooting
Problem: Agent performance.json file growing too large
Cause: History arrays not being trimmed
Solution: Implement automatic trimming after each update
functionupdateAgentMetrics(agentId, execution) {
const agent = metrics.agents[agentId];
// Update aggregates
agent.totalRuns += 1;
agent.successCount += execution.result === "success" ? 1 : 0;
// Add to history
agent.history.push(execution);
// Trim to max 100 entries (FIFO)if (agent.history.length > 100) {
agent.history = agent.history.slice(-100);
}
}
Problem: Metrics don't reflect recent changes
Cause: Stale data from old workflows
Solution: Reset metrics after major changes
# Backup current metricscp .claude/agent-performance.json .claude/agent-performance-backup-$(date +%Y%m%d).json
# Reset relevant sections (keep models, reset agents)# Edit .claude/agent-performance.json manually or use script
Problem: Success rate calculations seem wrong
Cause: Inconsistent result values ("success", "SUCCESS", "completed", etc.)
