# Performance Tracking > Track agent, skill, and model performance metrics for optimization. Use when measuring agent success rates, tracking model latency, analyzing routing effectiveness, or optimizing cost-per-task. Trigger keywords - "performance", "metrics", "tracking", "success rate", "agent performance", "model latency", "cost tracking", "optimization", "routing metrics". - Skill: `madappgang/performance-tracking` (Agent Skill) - Install (CLI): `npx skillmds@latest add madappgang/performance-tracking` - Raw SKILL.md: https://api.skillmd.com/api/skills/madappgang/performance-tracking/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: MadAppGang (https://skillmd.com/u/madappgang) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/madappgang/performance-tracking --- # Performance Tracking **Version:** 1.0.0 **Purpose:** Track agent, skill, and model performance metrics for continuous optimization **Status:** Production Ready ## Overview Performance tracking transforms workflows from "fire and forget" to **data-driven optimization systems**. By measuring what actually works, you can route tasks more effectively, identify failing patterns early, and reduce costs. This skill provides battle-tested patterns for: - **Agent success tracking** (completion rates, confidence scores, task type affinity) - **Skill effectiveness** (activation counts, success correlation, usage patterns) - **Model performance** (latency, cost, quality, provider comparison) - **Routing optimization** (tier distribution, routing accuracy, cost efficiency) - **Historical analysis** (trend detection, degradation alerts, pattern discovery) Performance tracking enables **continuous improvement** by providing the data needed to make informed decisions about agent selection, model choice, and workflow routing. ### Why Track Performance **Optimize Routing:** - Identify which agents excel at specific task types - Route complex tasks to high-confidence agents - Avoid agents with low success rates for critical work **Identify Failing Agents:** - Detect agents with <70% success rate - Alert when agent performance degrades - Replace or retrain underperforming agents **Reduce Costs:** - Find cost-effective model alternatives - Identify expensive agents with low success rates - Optimize tier thresholds based on actual performance **Improve Quality:** - Track correlation between confidence scores and success - Identify patterns in successful implementations - Learn which models produce best results for task types ### What We Track **Agent Metrics:** - Total runs, success/failure counts - Average confidence scores - Task type distribution - Last used timestamp - Individual execution history **Skill Metrics:** - Activation counts per skill - Last activation timestamps - Success correlation (when skill active, what's success rate?) - Co-activation patterns **Model Metrics:** - Total runs, success/failure counts - Average latency (response time) - Total cost (cumulative spend) - Cost per successful task - Last used timestamp **Routing Metrics:** - Tier distribution (how often each tier selected) - Routing accuracy (did tier match complexity?) - Cost efficiency (tier1 vs tier4 cost ratio) - Decision history with outcomes ### Integration with task-complexity-router The performance tracker provides critical feedback to the task-complexity-router: ``` Routing Feedback Loop: 1. Router selects tier based on complexity → task-complexity-router analyzes task → Routes to tier2 (medium complexity) 2. Agent executes task → Records: tier=2, agent=ui-developer, result=success 3. Performance tracker updates metrics → tier2 usage +1 → ui-developer success +1 → Confidence in tier2 routing increases 4. Future routing decisions informed by history → Router sees tier2 has 85% success rate → Router sees ui-developer excels at UI tasks → Router confidently routes similar tasks to tier2 ``` --- ## Metrics Schema ### JSON Structure (Version 1.0.0) Store performance metrics in `.claude/agent-performance.json`: ```json { "schemaVersion": "1.0.0", "lastUpdated": "2026-01-28T15:30:00Z", "agents": { "ui-developer": { "totalRuns": 42, "successCount": 38, "failureCount": 4, "avgConfidence": 0.85, "lastUsed": "2026-01-28T15:30:00Z", "taskTypes": { "implement-component": 15, "fix-styling": 12, "refactor-ui": 8, "review-code": 7 }, "history": [ { "timestamp": "2026-01-28T15:30:00Z", "taskType": "implement-component", "result": "success", "confidence": 0.90, "duration": 45000, "tier": 2, "model": "claude-sonnet-4-5-20250929" }, { "timestamp": "2026-01-28T14:20:00Z", "taskType": "fix-styling", "result": "success", "confidence": 0.85, "duration": 30000, "tier": 1, "model": "claude-sonnet-4-5-20250929" } ] }, "backend-developer": { "totalRuns": 28, "successCount": 25, "failureCount": 3, "avgConfidence": 0.88, "lastUsed": "2026-01-28T14:00:00Z", "taskTypes": { "implement-api": 12, "fix-bug": 8, "database-migration": 5, "write-tests": 3 }, "history": [] } }, "skills": { "multi-model-validation": { "activations": 15, "lastActivated": "2026-01-28T15:00:00Z", "successCorrelation": 0.92, "coActivations": { "quality-gates": 12, "error-recovery": 8 } }, "task-complexity-router": { "activations": 68, "lastActivated": "2026-01-28T15:30:00Z", "successCorrelation": 0.85, "coActivations": { "multi-agent-coordination": 45, "hierarchical-coordinator": 30 } } }, "models": { "claude-sonnet-4-5-20250929": { "totalRuns": 120, "successCount": 108, "failureCount": 12, "avgLatency": 2500, "totalCost": 0.45, "lastUsed": "2026-01-28T15:30:00Z", "taskTypePerformance": { "code-review": { "success": 25, "failure": 2 }, "implementation": { "success": 40, "failure": 5 }, "testing": { "success": 20, "failure": 3 } } }, "x-ai/grok-code-fast-1": { "totalRuns": 35, "successCount": 30, "failureCount": 5, "avgLatency": 1800, "totalCost": 0.08, "lastUsed": "2026-01-28T13:00:00Z", "taskTypePerformance": { "code-review": { "success": 18, "failure": 2 }, "implementation": { "success": 12, "failure": 3 } } } }, "routing": { "tierDistribution": { "tier1": 45, "tier2": 30, "tier3": 15, "tier4": 8 }, "decisions": [ { "timestamp": "2026-01-28T15:30:00Z", "taskType": "implement-component", "complexity": "medium", "selectedTier": 2, "agent": "ui-developer", "result": "success", "cost": 0.003 }, { "timestamp": "2026-01-28T14:20:00Z", "taskType": "fix-styling", "complexity": "low", "selectedTier": 1, "agent": "ui-developer", "result": "success", "cost": 0.001 } ] } } ``` ### Schema Field Definitions **Agent Metrics:** - `totalRuns`: Total task executions - `successCount`: Tasks completed successfully - `failureCount`: Tasks that failed or required retry - `avgConfidence`: Rolling average of agent confidence scores (0.0-1.0) - `lastUsed`: ISO-8601 timestamp of last execution - `taskTypes`: Distribution of task types (understand agent specialization) - `history`: Array of recent executions (max 100 entries, FIFO) **Skill Metrics:** - `activations`: Total times skill was triggered - `lastActivated`: ISO-8601 timestamp - `successCorrelation`: Success rate when this skill is active (0.0-1.0) - `coActivations`: Skills frequently activated together (detect patterns) **Model Metrics:** - `totalRuns`: Total executions - `successCount`/`failureCount`: Outcome tracking - `avgLatency`: Average response time in milliseconds - `totalCost`: Cumulative spend in USD - `lastUsed`: ISO-8601 timestamp - `taskTypePerformance`: Success/failure breakdown by task type **Routing Metrics:** - `tierDistribution`: Count of tasks routed to each tier - `decisions`: Array of routing decisions with outcomes (max 100, FIFO) --- ## Tracking Patterns ### Pattern 1: Capturing Agent Performance **After Agent Completes Task:** ``` Execution Flow: 1. Agent executes task Task: ui-developer Input: "Implement login form component" Result: Success Confidence: 0.90 Duration: 45 seconds Tier: 2 Model: claude-sonnet-4-5-20250929 2. Update agent metrics Read: .claude/agent-performance.json Update: agents["ui-developer"].totalRuns += 1 agents["ui-developer"].successCount += 1 agents["ui-developer"].avgConfidence = rolling_avg(0.90) agents["ui-developer"].lastUsed = NOW agents["ui-developer"].taskTypes["implement-component"] += 1 agents["ui-developer"].history.push({ timestamp: NOW, taskType: "implement-component", result: "success", confidence: 0.90, duration: 45000, tier: 2, model: "claude-sonnet-4-5-20250929" }) Trim history if > 100 entries Write: .claude/agent-performance.json 3. Calculate derived metrics Success rate: successCount / totalRuns = 38/42 = 90.5% Avg duration: sum(history.duration) / history.length Task affinity: taskTypes sorted by count ``` **After Agent Fails:** ``` Failure Flow: 1. Agent fails task Task: backend-developer Input: "Implement complex payment flow" Result: Failure (error, timeout, or low quality) Confidence: 0.65 Tier: 3 2. Update failure metrics agents["backend-developer"].totalRuns += 1 agents["backend-developer"].failureCount += 1 agents["backend-developer"].avgConfidence = rolling_avg(0.65) agents["backend-developer"].history.push({ timestamp: NOW, taskType: "implement-api", result: "failure", confidence: 0.65, duration: 120000, tier: 3, error: "Exceeded max iterations" }) 3. Check for degradation If failureCount / totalRuns > 0.30: Alert: "backend-developer has >30% failure rate" Recommendation: "Review recent failures, retrain, or replace" ``` ### Pattern 2: Tracking Model Performance **After Model Execution:** ``` Execution Flow: 1. Model completes task Model: x-ai/grok-code-fast-1 Task: Code review Latency: 1800ms Cost: $0.002 Result: Success 2. Update model metrics models["x-ai/grok-code-fast-1"].totalRuns += 1 models["x-ai/grok-code-fast-1"].successCount += 1 models["x-ai/grok-code-fast-1"].avgLatency = rolling_avg(1800) models["x-ai/grok-code-fast-1"].totalCost += 0.002 models["x-ai/grok-code-fast-1"].lastUsed = NOW models["x-ai/grok-code-fast-1"].taskTypePerformance["code-review"].success += 1 3. Compare model performance Claude Sonnet: avgLatency=2500ms, cost=$0.45 (120 runs) Grok Fast: avgLatency=1800ms, cost=$0.08 (35 runs) Analysis: - Grok is 28% faster (1800ms vs 2500ms) - Grok is 82% cheaper per run ($0.0023 vs $0.0038) - Both have similar success rates (86% vs 90%) Recommendation: - Use Grok for cost-sensitive tasks - Use Claude for critical tasks (higher success rate) ``` ### Pattern 3: Recording Skill Activation **After Skill Activation:** ``` Activation Flow: 1. Skill triggers Skill: multi-model-validation Context: User requested /review with 3 models 2. Update skill metrics skills["multi-model-validation"].activations += 1 skills["multi-model-validation"].lastActivated = NOW 3. Track co-activation Active skills: ["multi-model-validation", "quality-gates"] skills["multi-model-validation"].coActivations["quality-gates"] += 1 4. Calculate success correlation Tasks with this skill active: 15 Successful tasks: 14 Success correlation: 14/15 = 93.3% 5. Pattern detection Observation: multi-model-validation + quality-gates = 100% success (12/12) Recommendation: Always pair these skills for high-quality reviews ``` ### Pattern 4: Routing Decision Tracking **After Routing Decision:** ``` Routing Flow: 1. Router selects tier Task: "Implement user profile page" Analysis: Medium complexity (multiple components, state management) Selected tier: 2 Agent: ui-developer Model: claude-sonnet-4-5-20250929 2. Record routing decision routing.tierDistribution["tier2"] += 1 routing.decisions.push({ timestamp: NOW, taskType: "implement-component", complexity: "medium", selectedTier: 2, agent: "ui-developer", result: "pending" }) 3. After task completes Update decision with result: routing.decisions[last].result = "success" routing.decisions[last].cost = 0.003 4. Trim decision history if > 100 entries ``` ### Pattern 5: Session-Level Aggregation **End of Session Summary:** ``` Session Summary Flow: 1. Aggregate session metrics Session ID: 2026-01-28-session-15 Duration: 2 hours Tasks executed: 15 Success rate: 14/15 = 93.3% Total cost: $0.045 Models used: Claude (12), Grok (3) 2. Create session snapshot File: ai-docs/performance-history/2026-01-28-session-15.json Content: { "sessionId": "2026-01-28-session-15", "startTime": "2026-01-28T13:00:00Z", "endTime": "2026-01-28T15:00:00Z", "duration": 7200000, "tasks": 15, "successRate": 0.933, "totalCost": 0.045, "modelUsage": { "claude": 12, "grok": 3 }, "topAgents": ["ui-developer", "backend-developer"], "activeSkills": ["task-complexity-router", "multi-model-validation"] } 3. Update rolling metrics .claude/agent-performance.json (persistent) ai-docs/performance-history/ (snapshots) 4. Cleanup old snapshots Keep last 100 session snapshots Delete older entries ``` --- ## File Location and Management ### Primary Performance File **Location:** `.claude/agent-performance.json` **Purpose:** Persistent, project-level performance tracking **When to Update:** - After every agent execution - After every model execution - After every skill activation - After every routing decision **Format:** JSON schema version 1.0.0 (see Metrics Schema section) **Rotation:** Keep full history, but trim individual history arrays to 100 entries ### Session Snapshots **Location:** `ai-docs/performance-history/` **Purpose:** Point-in-time session summaries for historical analysis **Naming:** `{YYYY-MM-DD}-session-{N}.json` **Example:** ``` ai-docs/performance-history/ 2026-01-28-session-1.json 2026-01-28-session-2.json 2026-01-27-session-1.json ... ``` **Retention:** Keep last 100 sessions, delete older ### Integration with Existing Files **Relationship with ai-docs/llm-performance.json:** ``` Comparison: llm-performance.json (existing): - Model-specific performance - Cost tracking per model - Response time tracking - Used by multi-model-validation agent-performance.json (new): - Agent-level metrics (multi-run aggregation) - Skill activation tracking - Routing decision history - Task type affinity Integration: - agent-performance.json imports model data from llm-performance.json - Both files updated in parallel - llm-performance.json focuses on single-run details - agent-performance.json focuses on aggregate trends ``` **Migration Path:** ``` Step 1: Create .claude/agent-performance.json with schema 1.0.0 Step 2: Import historical data from llm-performance.json Step 3: Update both files going forward Step 4: Deprecate llm-performance.json after 6 months (optional) ``` ### Data Cleanup and Rotation **Automatic Cleanup:** ``` Cleanup Rules: 1. Agent history arrays Max entries: 100 Strategy: FIFO (oldest removed first) Trigger: After every agent execution 2. Routing decision arrays Max entries: 100 Strategy: FIFO Trigger: After every routing decision 3. Session snapshots Max files: 100 Strategy: FIFO (delete oldest session files) Trigger: After every session ends 4. Skill co-activation maps Max entries per skill: 50 Strategy: Keep top 50 by count Trigger: Weekly cleanup ``` **Manual Cleanup:** ``` When to manually reset: 1. After major workflow changes - Agent capabilities changed - New skills added - Routing logic updated → Reset metrics to start fresh 2. After agent retraining - Agent prompt updated - Agent model changed → Reset agent-specific metrics 3. After prolonged period (>6 months) - Metrics may be outdated → Archive old data, start fresh How to reset: Backup: cp .claude/agent-performance.json .claude/agent-performance-backup-{DATE}.json Reset: echo '{"schemaVersion":"1.0.0","lastUpdated":"...","agents":{},...}' > .claude/agent-performance.json ``` --- ## Using Metrics for Optimization ### Optimization 1: Identify Underperforming Agents **Detection:** ``` Analyze agent success rates: agents["ui-developer"]: successCount: 38 totalRuns: 42 success rate: 38/42 = 90.5% ✅ GOOD agents["test-architect"]: successCount: 15 totalRuns: 25 success rate: 15/25 = 60% ❌ UNDERPERFORMING Threshold: <70% success rate = underperforming ``` **Action:** ``` For test-architect (60% success): 1. Analyze failure patterns Review history entries where result="failure" Common failure reasons: - "Tests too brittle" (8 occurrences) - "Missing test coverage" (5 occurrences) - "Test timeout" (2 occurrences) 2. Identify root cause Pattern: test-architect struggles with async/timing tests Evidence: All timeout failures involved async code 3. Take action Option A: Retrain agent - Update prompt with async testing best practices - Add examples of proper async test patterns - Reset metrics after retraining Option B: Route differently - Route async test tasks to backend-developer (90% success on async) - Keep test-architect for synchronous unit tests Option C: Replace agent - Create new specialized-async-test-architect - Deprecate test-architect for async work ``` ### Optimization 2: Find Cost-Effective Model Alternatives **Analysis:** ``` Compare model cost-effectiveness: Model: claude-sonnet-4-5-20250929 Total cost: $0.45 Total runs: 120 Success count: 108 Cost per task: $0.0038 Cost per success: $0.0042 Success rate: 90% Model: x-ai/grok-code-fast-1 Total cost: $0.08 Total runs: 35 Success count: 30 Cost per task: $0.0023 Cost per success: $0.0027 Success rate: 86% Model: google/gemini-2.5-flash Total cost: $0.02 Total runs: 20 Success count: 16 Cost per task: $0.0010 Cost per success: $0.0013 Success rate: 80% Cost Efficiency Ranking: 1. Gemini Flash: $0.0013 per success (80% success rate) 2. Grok Fast: $0.0027 per success (86% success rate) 3. Claude Sonnet: $0.0042 per success (90% success rate) Quality-Cost Tradeoff: - Gemini: 69% cheaper than Claude, but 10% lower success rate - Grok: 36% cheaper than Claude, but 4% lower success rate ``` **Action:** ``` Optimization strategy: Tier 1 (Simple tasks): Use: Gemini Flash Reason: Lowest cost, acceptable success rate for simple work Example: "Fix typo in comment", "Format code" Tier 2 (Medium tasks): Use: Grok Fast Reason: Good balance of cost and quality Example: "Implement CRUD endpoint", "Add validation" Tier 3 (Complex tasks): Use: Claude Sonnet Reason: Highest success rate justifies cost Example: "Design architecture", "Complex refactoring" Tier 4 (Critical tasks): Use: Claude Sonnet + Multi-model validation Reason: Quality > cost for critical work Example: "Security review", "Production bug fix" Expected savings: Current: 90% Claude usage × $0.0042 = $0.00378 avg per task Optimized: 20% Claude + 50% Grok + 30% Gemini = $0.00257 avg per task Savings: 32% cost reduction with minimal quality impact ``` ### Optimization 3: Optimize Routing Tier Thresholds **Analysis:** ``` Review tier distribution: routing.tierDistribution: tier1: 45 tasks (45.9%) tier2: 30 tasks (30.6%) tier3: 15 tasks (15.3%) tier4: 8 tasks (8.2%) Analyze tier accuracy: Tier 1 (Simple): Tasks: 45 Success: 42 Failures: 3 Success rate: 93.3% ✅ Verdict: Well-calibrated Tier 2 (Medium): Tasks: 30 Success: 25 Failures: 5 Success rate: 83.3% ⚠️ Verdict: Slightly low (target 90%) Tier 3 (Complex): Tasks: 15 Success: 12 Failures: 3 Success rate: 80.0% ⚠️ Verdict: Too low (target 90%) Tier 4 (Critical): Tasks: 8 Success: 8 Failures: 0 Success rate: 100% ✅ Verdict: Well-calibrated ``` **Action:** ``` Adjust tier thresholds: Current thresholds (task-complexity-router): tier1: complexity score 0-3 tier2: complexity score 4-6 tier3: complexity score 7-9 tier4: complexity score 10+ Problem: tier2 and tier3 have lower success rates Root cause: Tasks slightly too complex for assigned tier Optimized thresholds: tier1: complexity score 0-2 (narrower range) tier2: complexity score 3-5 (shift down) tier3: complexity score 6-8 (shift down) tier4: complexity score 9+ (broader range) Rationale: - Shift more borderline tasks to higher tiers - Accept slightly higher cost for better success rates - tier2/tier3 success should improve to 90%+ Expected impact: - tier1 usage: 45 → 35 tasks (fewer simple tasks) - tier2 usage: 30 → 32 tasks (more medium tasks) - tier3 usage: 15 → 18 tasks (more complex tasks) - tier4 usage: 8 → 13 tasks (more critical tasks) - Overall success rate: 88% → 92% - Overall cost: +15% (acceptable tradeoff for quality) ``` ### Optimization 4: Detect Model-Task Affinity Patterns **Analysis:** ``` Analyze task type performance by model: Task type: code-review Claude Sonnet: Success: 25, Failure: 2 Success rate: 92.6% ✅ Grok Fast: Success: 18, Failure: 2 Success rate: 90.0% ✅ Gemini Flash: Success: 10, Failure: 4 Success rate: 71.4% ⚠️ → Pattern: Claude and Grok excel at code review, Gemini struggles Task type: implementation Claude Sonnet: Success: 40, Failure: 5 Success rate: 88.9% ✅ Grok Fast: Success: 12, Failure: 3 Success rate: 80.0% ⚠️ Gemini Flash: Success: 6, Failure: 1 Success rate: 85.7% ✅ → Pattern: Claude best for implementation, Grok/Gemini acceptable Task type: testing Claude Sonnet: Success: 20, Failure: 3 Success rate: 87.0% ✅ Grok Fast: Success: 0, Failure: 0 Success rate: N/A Gemini Flash: Success: 0, Failure: 0 Success rate: N/A → Pattern: Only Claude has testing data (others not used for this) ``` **Action:** ``` Task-specific model routing: code-review tasks: tier1: Grok Fast (90% success, low cost) tier2: Grok Fast (90% success, low cost) tier3: Claude Sonnet (93% success, high quality) tier4: Multi-model (Claude + Grok consensus) implementation tasks: tier1: Gemini Flash (86% success, lowest cost) tier2: Grok Fast (80% success, medium cost) tier3: Claude Sonnet (89% success, highest quality) tier4: Claude Sonnet (89% success, proven) testing tasks: tier1-4: Claude Sonnet (only model with proven testing capability) Expected impact: - 25% cost savings on code reviews (use Grok instead of Claude) - 10% cost savings on implementation (use Gemini for simple) - Maintain quality (route by proven success rates) ``` ### Optimization 5: Alert on Performance Degradation **Detection:** ``` Monitor for degradation: Week 1 (baseline): ui-developer success rate: 90.5% Average task duration: 45s Week 2: ui-developer success rate: 88.2% (↓2.3%) Average task duration: 48s (↑3s) Week 3: ui-developer success rate: 85.1% (↓5.4% from baseline) Average task duration: 52s (↑7s from baseline) Week 4: ui-developer success rate: 78.3% (↓12.2% from baseline) 🚨 Average task duration: 58s (↑13s from baseline) 🚨 Threshold exceeded: ❌ Success rate dropped >10% (78.3% vs 90.5%) ❌ Duration increased >20% (58s vs 45s) → ALERT: ui-developer performance degraded significantly ``` **Action:** ``` Degradation response: 1. Investigate root cause Review recent history: - Task complexity increased? (Check taskTypes distribution) - Model changed? (Check model field in history) - Failures clustered around specific task type? Finding: All recent failures on "complex-state-management" tasks Root cause: New task type introduced, agent not trained for it 2. Take corrective action Option A: Retrain agent - Update prompt with state management patterns - Add examples of successful state management - Reset metrics after retraining Option B: Route differently - Route state management tasks to specialized agent - Keep ui-developer for simpler UI tasks Option C: Escalate to human - Alert: "ui-developer performance degraded" - Request: "Manual review of recent failures needed" 3. Monitor recovery Week 5 (after retraining): Success rate: 85.0% (recovering) Week 6: Success rate: 89.2% (near baseline) Week 7: Success rate: 91.0% (recovered ✅) ``` --- ## Integration with Orchestration Plugin ### Integration 1: multi-model-validation **How multi-model-validation records model performance:** ``` Multi-Model Review Flow: 1. Execute parallel review Models: [claude-sonnet, grok-fast, gemini-flash] Task: Code review of auth.ts 2. Collect model responses Each model returns: - Review findings - Confidence score - Latency - Cost 3. Record individual model performance For each model: models[modelId].totalRuns += 1 models[modelId].avgLatency = rolling_avg(latency) models[modelId].totalCost += cost 4. Determine success/failure If review found critical issues → success (doing its job) If review crashed/errored → failure 5. Update success counts models[modelId].successCount += 1 (or failureCount) models[modelId].taskTypePerformance["code-review"].success += 1 6. Consolidate findings Generate consensus report Track which models agreed (co-occurrence patterns) 7. User feedback (optional) User rates review quality: "Helpful" | "Not helpful" Update successCorrelation for multi-model-validation skill ``` ### Integration 2: task-complexity-router **How task-complexity-router reads performance data:** ``` Routing Decision Flow: 1. Analyze task complexity Input: "Implement user authentication with OAuth" Analysis: Complex (multiple components, external API, security) Base tier: 3 2. Read performance history Load: .claude/agent-performance.json Check: routing.tierDistribution 3. Adjust tier based on history tier3 historical success rate: 80% (below 90% target) tier4 historical success rate: 100% Decision: Bump to tier4 for higher success probability 4. Select agent based on task type affinity Task type: "implement-api" Candidates: backend-developer, full-stack-developer Check affinity: backend-developer.taskTypes["implement-api"]: 12 (high affinity) full-stack-developer.taskTypes["implement-api"]: 3 (low affinity) Decision: Select backend-developer (proven track record) 5. Select model based on tier + task type tier4 + implement-api: models[claude].taskTypePerformance["implementation"]: 89% success models[grok].taskTypePerformance["implementation"]: 80% success Decision: Select Claude (higher success rate for tier4) 6. Record routing decision routing.decisions.push({ timestamp: NOW, taskType: "implement-api", complexity: "complex", selectedTier: 4, agent: "backend-developer", model: "claude-sonnet-4-5-20250929", result: "pending" }) 7. After execution, update result routing.decisions[last].result = "success" routing.decisions[last].cost = 0.005 ``` ### Integration 3: hierarchical-coordinator **How hierarchical-coordinator tracks phase success:** ``` Phase Execution Tracking: 1. Execute workflow phases Phase 1: Planning (architect agent) Phase 2: Implementation (developer agent) Phase 3: Testing (tester agent) Phase 4: Review (reviewer agent) 2. Track phase-level metrics Create phase-specific tracking: agents["architect"].phasePerformance = { "planning": { success: 15, failure: 2 }, "architecture": { success: 8, failure: 1 } } agents["developer"].phasePerformance = { "implementation": { success: 25, failure: 5 }, "refactoring": { success: 10, failure: 2 } } 3. Detect phase-specific issues Analysis: developer has 20% failure rate on implementation phase But: developer has 83% success rate overall Insight: Failures concentrated in specific phase 4. Optimize phase assignment Current: developer handles all implementation Optimized: Split by complexity - Simple implementation → junior-developer (cheaper) - Complex implementation → senior-developer (higher success) 5. Track coordinator effectiveness skills["hierarchical-coordinator"].activations += 1 skills["hierarchical-coordinator"].successCorrelation = 0.92 Insight: Workflows using coordinator have 92% success (vs 80% without) ``` ### Integration 4: quality-gates **How quality-gates uses performance thresholds:** ``` Quality Gate Decision: 1. Agent completes task Agent: ui-developer Task: "Implement dashboard component" Confidence: 0.75 2. Check agent performance history Load: agents["ui-developer"] Historical avg confidence: 0.85 Current confidence: 0.75 (below average 🚨) 3. Apply quality gate Threshold: If confidence < avg - 0.10, trigger validation Decision: 0.75 < 0.75 (borderline) Action: Trigger designer validation (extra quality check) 4. Designer validates Result: Found 3 minor issues Verdict: Quality gate prevented low-quality work from proceeding 5. Update metrics Without gate: ui-developer would have 1 more failure With gate: Issues caught early, fixed before user sees skills["quality-gates"].successCorrelation += 1 (Success correlation increases when gate prevents failures) 6. Continuous improvement Pattern: Low-confidence tasks benefit from extra validation Threshold: Automatically adjust based on correlation data Future: If confidence < 0.80, always trigger validation ``` --- ## Best Practices ### Do - ✅ **Track all agent executions** (success and failure provide learning signal) - ✅ **Record model latency and cost** (optimize for cost-effectiveness) - ✅ **Maintain execution history** (detect patterns and trends) - ✅ **Set success rate thresholds** (<70% = investigate, <50% = replace) - ✅ **Alert on performance degradation** (>10% drop from baseline) - ✅ **Use task type affinity** (route tasks to agents with proven success) - ✅ **Compare model cost-effectiveness** (cost per success, not just cost per task) - ✅ **Track skill co-activation** (identify successful skill combinations) - ✅ **Rotate history data** (keep last 100 entries, prevent unbounded growth) - ✅ **Create session snapshots** (point-in-time analysis) - ✅ **Integrate with routing** (feed performance data back to router) ### Don't - ❌ **Track only successes** (failures provide valuable learning signal) - ❌ **Ignore degradation** (small drops compound into big problems) - ❌ **Use stale data** (>6 months old metrics may not reflect current state) - ❌ **Over-optimize on cost alone** (balance cost and quality) - ❌ **Forget to update metrics** (incomplete data leads to poor decisions) - ❌ **Store unbounded history** (trim arrays to prevent file bloat) - ❌ **Mix session metrics** (isolate session data for cleaner analysis) - ❌ **Ignore task type affinity** (agents specialize, use it) - ❌ **Skip validation after major changes** (reset metrics when workflows change) ### Privacy Considerations **What to Track:** - Aggregate metrics (counts, averages, distributions) - Task types (generic categories like "implement-component") - Success/failure outcomes - Model performance data - Timing and cost data **What NOT to Track:** - User-specific data (usernames, emails) - Sensitive code snippets - API keys or credentials - Personal information - Business logic details **Data Retention:** - Keep aggregate metrics indefinitely (no PII) - Rotate detailed history after 100 entries - Delete session snapshots after 100 sessions - Archive old metrics before major resets ### When to Reset Metrics **Situations Requiring Reset:** 1. **Agent capabilities changed** - Prompt updated significantly - Agent model changed - Agent skills added/removed → Reset agent-specific metrics 2. **Workflow architecture changed** - New routing logic - New tier definitions - New skill combinations → Reset routing and skill metrics 3. **Model pricing changed** - Cost per token updated - New pricing tier → Reset cost calculations (keep counts) 4. **After prolonged period (>6 months)** - Metrics may be outdated - Workflow patterns changed → Archive and reset all metrics **How to Reset:** ```bash # Backup current metrics cp .claude/agent-performance.json .claude/agent-performance-backup-$(date +%Y%m%d).json # Reset to empty state cat > .claude/agent-performance.json << 'EOF' { "schemaVersion": "1.0.0", "lastUpdated": "2026-01-28T16:00:00Z", "agents": {}, "skills": {}, "models": {}, "routing": { "tierDistribution": {}, "decisions": [] } } EOF # Archive old session snapshots mkdir -p ai-docs/performance-history/archive-$(date +%Y%m%d) mv ai-docs/performance-history/*.json ai-docs/performance-history/archive-$(date +%Y%m%d)/ ``` ### Metric Hygiene **Regular Maintenance:** ``` Weekly: - Review top agents (ensure success rates >70%) - Check model cost trends (identify cost spikes) - Trim co-activation maps (keep top 50 per skill) Monthly: - Analyze task type affinity changes - Compare model cost-effectiveness - Review tier distribution accuracy - Archive old session snapshots (keep last 100) Quarterly: - Deep analysis of performance trends - Optimize routing thresholds - Identify underperforming patterns - Consider agent retraining or replacement Annually: - Full metrics review and reset (if needed) - Archive all historical data - Update baseline success rates - Document lessons learned ``` --- ## Examples ### Example 1: Tracking a Multi-Model Review Session **Scenario:** User requests `/review` with 3 models (Claude, Grok, Gemini) **Execution:** ``` Step 1: Initialize session Session ID: 2026-01-28-session-15 Start time: 15:00:00Z Step 2: Execute multi-model review Models: [claude-sonnet, grok-fast, gemini-flash] Task: Code review of auth/login.ts (450 lines) Step 3: Track individual model executions Model: claude-sonnet-4-5-20250929 Start: 15:00:05Z End: 15:00:08Z Latency: 3000ms Cost: $0.003 Result: Found 5 issues (2 CRITICAL, 3 HIGH) Outcome: Success Update metrics: models["claude-sonnet-4-5-20250929"].totalRuns = 121 models["claude-sonnet-4-5-20250929"].successCount = 109 models["claude-sonnet-4-5-20250929"].avgLatency = 2520ms models["claude-sonnet-4-5-20250929"].totalCost = $0.453 models["claude-sonnet-4-5-20250929"].taskTypePerformance["code-review"].success = 26 Model: x-ai/grok-code-fast-1 Start: 15:00:05Z End: 15:00:07Z Latency: 2000ms Cost: $0.002 Result: Found 4 issues (2 CRITICAL, 2 HIGH) Outcome: Success Update metrics: models["x-ai/grok-code-fast-1"].totalRuns = 36 models["x-ai/grok-code-fast-1"].successCount = 31 models["x-ai/grok-code-fast-1"].avgLatency = 1820ms models["x-ai/grok-code-fast-1"].totalCost = $0.082 models["x-ai/grok-code-fast-1"].taskTypePerformance["code-review"].success = 19 Model: google/gemini-2.5-flash Start: 15:00:05Z End: 15:00:06Z Latency: 1500ms Cost: $0.001 Result: Found 3 issues (1 CRITICAL, 2 MEDIUM) Outcome: Success Update metrics: models["google/gemini-2.5-flash"].totalRuns = 21 models["google/gemini-2.5-flash"].successCount = 17 models["google/gemini-2.5-flash"].avgLatency = 1480ms models["google/gemini-2.5-flash"].totalCost = $0.021 models["google/gemini-2.5-flash"].taskTypePerformance["code-review"].success = 11 Step 4: Track skill activation skills["multi-model-validation"].activations = 16 skills["multi-model-validation"].lastActivated = 15:00:08Z skills["multi-model-validation"].coActivations["quality-gates"] = 13 Step 5: Consolidate findings Consensus issues (all 3 models agreed): - CRITICAL: SQL injection vulnerability (UNANIMOUS) - CRITICAL: Missing authentication check (UNANIMOUS) Majority issues (2/3 models agreed): - HIGH: Insufficient input validation (Claude, Grok) - HIGH: Missing error handling (Claude, Grok) Divergent issues (1/3 models): - MEDIUM: Code duplication (Gemini only) Step 6: Record session summary Session complete: Duration: 8 seconds Models used: 3 Total cost: $0.006 Issues found: 5 (2 unanimous, 2 majority, 1 divergent) Result: Success Create snapshot: ai-docs/performance-history/2026-01-28-session-15.json Step 7: Update aggregate metrics Overall session success rate: 3/3 models successful = 100% Cost efficiency: $0.002 per model = good value ``` **Insights from Tracking:** ``` Performance comparison: Fastest: Gemini (1500ms) - 50% faster than Claude Most thorough: Claude (5 issues) - Found 1 extra issue Best value: Gemini ($0.001, 3 issues) - Lowest cost, good coverage Cost analysis: Total: $0.006 for 3-model review vs Single Claude: $0.003 (double cost, but 2x validation) ROI: Found 2 CRITICAL issues all models agreed on = high confidence Consensus validation: UNANIMOUS issues (100% confidence) → Fix immediately MAJORITY issues (67% confidence) → Fix before merge DIVERGENT issues (33% confidence) → Low priority (possible false positive) Recommendation: Multi-model validation worth the cost for critical code (auth, payments, security) Single-model sufficient for non-critical code (UI components, docs) ``` --- ### Example 2: Identifying Model Performance Differences **Scenario:** After 100 tasks, compare model performance for optimization **Execution:** ``` Step 1: Load performance data Read: .claude/agent-performance.json Step 2: Extract model metrics Claude Sonnet: Total runs: 120 Success: 108, Failure: 12 Success rate: 90.0% Avg latency: 2500ms Total cost: $0.45 Cost per task: $0.00375 Cost per success: $0.00417 Grok Fast: Total runs: 35 Success: 30, Failure: 5 Success rate: 85.7% Avg latency: 1800ms Total cost: $0.08 Cost per task: $0.00229 Cost per success: $0.00267 Gemini Flash: Total runs: 20 Success: 16, Failure: 4 Success rate: 80.0% Avg latency: 1500ms Total cost: $0.02 Cost per task: $0.00100 Cost per success: $0.00125 Step 3: Analyze task type performance Code Review: Claude: 25 success, 2 failure = 92.6% Grok: 18 success, 2 failure = 90.0% Gemini: 10 success, 4 failure = 71.4% Winner: Claude (highest quality) Best value: Grok (90% at lower cost) Implementation: Claude: 40 success, 5 failure = 88.9% Grok: 12 success, 3 failure = 80.0% Gemini: 6 success, 1 failure = 85.7% Winner: Claude (highest quality) Surprising: Gemini performs well here (86% success) Testing: Claude: 20 success, 3 failure = 87.0% Grok: No data Gemini: No data Winner: Claude (only option) Action: Try Grok/Gemini for testing tasks to gather data Step 4: Calculate cost-effectiveness by task type Code Review Cost-Effectiveness: Claude: $0.00417 per success, 92.6% quality Grok: $0.00267 per success, 90.0% quality (36% cheaper, -2.6% quality) Gemini: $0.00125 per success, 71.4% quality (70% cheaper, -21.2% quality) Recommendation: Use Grok for cost-effective reviews (minimal quality loss) Implementation Cost-Effectiveness: Claude: $0.00417 per success, 88.9% quality Grok: $0.00267 per success, 80.0% quality (36% cheaper, -8.9% quality) Gemini: $0.00125 per success, 85.7% quality (70% cheaper, -3.2% quality) Recommendation: Use Gemini for simple implementation (best value) Step 5: Generate optimization plan Current usage (120 total tasks): Claude: 100 tasks (83%) Grok: 15 tasks (13%) Gemini: 5 tasks (4%) Optimized usage (maint …(truncated)