Review Metrics

Review Board Metrics Analyzer

type: slash-command triggers: /review-metrics, review metrics, review board stats output: HTML report + actionable recommendations

Context

The Review Board Review Metrics skill synthesizes review data to detect patterns, measure specialist effectiveness, and recommend process improvements. It runs on demand (slash command) and analyses the metrics log generated by the Chief Reviewer.

Inputs:

• .claude/review-board/metrics.jsonl — one JSON object per line, one per review completion
• Date range: defaults to last 30 days; can be customized

Outputs:

• Weekly summary table
• Specialist effectiveness ranking
• Recurring issue patterns (same finding type in 3+ consecutive PRs)
• Quality trends (blocking rate, duration, complexity over time)
• Preventive action recommendations

Instructions

1. Read and Parse Metrics Log

Open .claude/review-board/metrics.jsonl and parse each line as JSON:

{
  "timestamp": "2026-03-28T14:30:00Z",
  "pr_number": 1234,
  "verdict": "REQUEST_CHANGES",
  "specialists": [
    {
      "name": "dx-review",
      "findings": 3,
      "critical": 0,
      "high": 1,
      "medium": 2,
      "low": 0,
      "confidence": 87,
      "duration_ms": 3200
    }
  ],
  "total_findings": 5,
  "total_duration_ms": 8500,
  "human_override": false,
  "notes": "User fixed 2 of 5 findings before merge"
}

If the file doesn't exist, initialise it as empty and note that metrics are not yet available.

2. Calculate Aggregate Metrics

For the entire dataset (or date range specified):

• Total reviews: count of entries
• Verdict distribution: count APPROVE, REQUEST_CHANGES, COMMENT verdicts
• Blocking rate: (REQUEST_CHANGES + COMMENT) / total_reviews * 100
• Pass rate: APPROVE / total_reviews * 100
• Avg review duration: mean of total_duration_ms
• Std dev review duration: standard deviation (detect bottlenecks)
• Median findings per review: median of total_findings
• Human override rate: count(human_override: true) / total_reviews * 100

3. Specialist Effectiveness Analysis

For each specialist that appeared in reviews, calculate:

• Total reviews participated: count of entries where specialist appeared
• Avg findings per review: mean of specialist's findings field
• Finding distribution: sum of CRITICAL, HIGH, MEDIUM, LOW across all reviews
• Avg confidence: mean of specialist's confidence scores
• Avg duration: mean of specialist's duration_ms
• Hit rate: reviews where specialist found at least one finding / total reviews
• False positive rate: reviews where specialist flagged but human override was required (if tracked)

Ranking: Sort by (1) avg confidence, (2) finding hit rate, (3) avg duration (fastest is best if finding count is same).

4. Identify Recurring Issues

A recurring issue is a finding type that appears in 3+ consecutive PRs.

Algorithm:

• For each specialist, extract finding types (issue field from specialist output)
• For each finding type, check if it appeared in the last N PRs
• If same finding type appears in 3+ consecutive PRs, flag as recurring

Example: If dx-review found "cognitive complexity >20" in PRs #1210, #1211, #1212, #1214 (not #1213), then we have two occurrences of a recurring pattern.

Report format:

Recurring: [Specialist] — [Finding Type]
  Last 3+ PRs: #1210 (high), #1211 (high), #1212 (high)
  Recommendation: Add ESLint rule or pre-commit hook

5. Detect Quality Trends

Blocking rate trend: Calculate blocking rate for weeks 1-4 (if 30 days of data); detect if rate is increasing, decreasing, or stable.

Finding complexity trend: Average severity of findings over time; detect if team is improving (fewer CRITICAL/HIGH) or regressing.

Review duration trend: Avg review time per week; detect bottlenecks (duration increasing = complexity or specialist slowness).

Specialist confidence trend: Is avg confidence increasing (learning) or decreasing (fatigue)?

6. Generate Recommendations

For each recurring issue pattern:

• If same issue in 3+ PRs: Recommend adding ESLint/pre-commit hook to catch earlier
• If specialist avg duration >8 minutes: Review specialist rules for false positives or over-analysis
• If human override rate >30% for a specialist: Recalibrate confidence threshold; rules may be too strict
• If blocking rate >50%: Recommend team training on common issues
• If blocking rate <20%: Review specialist coverage; may be too lenient
• If specialist appears in <10% of reviews: Consider removing from panel (low signal)

7. Output Report

Generate an HTML report with:

1. Executive Summary

   • Total reviews analysed
   • Overall pass rate
   • Avg review duration
   • Blocking rate

2. Weekly Breakdown (table)

   • Week starting date

   • PRs reviewed

   • Pass rate
   • Avg findings
   • Avg duration
   • Blocking rate trend (🔴 increasing, 🟢 decreasing, ⚪ stable)

3. Specialist Rankings (table)

   • Specialist name

   • Reviews

   • Avg findings per review
   • Confidence (avg)
   • Duration (avg)
   • Hit rate
   • Recommendation (Keep / Review / Remove)

4. Recurring Issues (list)

   • Issue type
   • Specialist
   • Count of consecutive PRs
   • Suggested prevention (lint rule, hook, docs)

5. Quality Trends (text + chart)

   • Blocking rate trend (week-by-week)
   • Finding severity trend
   • Review duration trend
   • Confidence trend

6. Actionable Recommendations (prioritized list)

   • High-impact actions first
   • Include Synthex-specific context (team size, review frequency)

Example Recurring Issue

Recurring Issue: Cognitive Complexity >20
Specialist: dx-review
PRs: #1208, #1209, #1210, #1211 (4 consecutive)
Severity: HIGH
Recommendation: Add `eslint-plugin-complexity` with max: 15 to catch in local development
Priority: HIGH (1 in 4 reviews blocked on this)
Action: Update .eslintrc.js, run npm run lint, document in DEVELOPMENT.md

Learning Signals

Monitor these signals for process improvement:

Output Schema

{
  "report_generated_at": "2026-03-28T15:00:00Z",
  "period": {
    "start": "2026-02-27",
    "end": "2026-03-28",
    "days": 30
  },
  "summary": {
    "total_reviews": 24,
    "pass_rate_percent": 62.5,
    "block_rate_percent": 37.5,
    "avg_duration_ms": 6200,
    "median_findings": 2,
    "human_override_rate_percent": 8.3
  },
  "weekly_breakdown": [
    {
      "week_starting": "2026-02-27",
      "reviews": 6,
      "pass_rate": 66.7,
      "avg_findings": 2.3,
      "avg_duration_ms": 5900,
      "trend": "stable"
    }
  ],
  "specialist_rankings": [
    {
      "specialist": "supabase-patterns",
      "reviews": 24,
      "avg_findings": 2.1,
      "avg_confidence": 91,
      "avg_duration_ms": 4200,
      "hit_rate": 79.2,
      "recommendation": "Keep — highest confidence and hit rate"
    }
  ],
  "recurring_issues": [
    {
      "finding_type": "Cognitive complexity >20",
      "specialist": "dx-review",
      "consecutive_prs": ["#1208", "#1209", "#1210", "#1211"],
      "count": 4,
      "severity": "HIGH",
      "recommendation": "Add ESLint rule: max-complexity: 15"
    }
  ],
  "quality_trends": {
    "blocking_rate_trend": "decreasing",
    "avg_severity_trend": "improving",
    "duration_trend": "stable",
    "confidence_trend": "increasing"
  },
  "recommendations": [
    {
      "priority": "HIGH",
      "action": "Add ESLint rule for cognitive complexity",
      "reason": "Cognitive complexity >20 found in 4 consecutive reviews",
      "impact": "Catch ~1-2 issues per sprint before review"
    }
  ]
}

Synthex-Specific Calibration

Team size: 1 developer (Phill) = slow PR cadence; expect 1-2 PRs/week

Specialists panel: 3 specialists (dx-review, supabase-patterns, review-metrics)

Quality gate: Blocking rate target 30-40% (enough rigor without being punitive)

False positive tolerance: Human override rate <15% (specialist rules are well-calibrated)

Confidence bar: All findings must be >=80% confidence

When to Run

• Manually: /review-metrics on demand
• Scheduled: Can be configured to run weekly (Monday morning) via scheduled task
• Triggered: Optionally triggered after every 10th PR review to detect emerging patterns

Error Handling

If metrics log is missing or empty:

No reviews recorded yet. Metrics will be available after 5+ reviews are completed.
Current status: 0 reviews, 0 findings.

If date range has <5 reviews:

Insufficient data for trend analysis (3 reviews in range).
Recommendation: Run again after 10+ reviews accumulated.
Current specialist rankings:
  [limited data, trust scores less]

If a specialist appears in only 1 review:

Specialist 'new-specialist' has insufficient data (1 review).
Min threshold for ranking: 5 reviews. Wait for more data.