Retrospective Validation

Retrospective Validation

Validate methodologies with historical data, not live deployment.

When you have 1,000 past errors, you don't need to wait for 1,000 future errors to prove your methodology works.

When to Use This Skill

Use this skill when:

• 📊 Rich historical data: 100+ instances (errors, test failures, performance issues)
• 🎯 Observable patterns: Methodology targets detectable issues
• 🔍 Pattern matching feasible: Clear detection heuristics, measurable false positive rate
• ⚡ High deployment friction: CI/CD integration costly, user studies time-consuming
• 📈 Statistical rigor needed: Want confidence intervals, not just hunches
• ⏰ Time constrained: Need validation in hours, not weeks

Don't use when:

• ❌ Insufficient data (<50 instances)
• ❌ Emergent effects (human behavior change, UX improvements)
• ❌ Pattern matching unreliable (>20% false positive rate)
• ❌ Low deployment friction (1-2 hour CI/CD integration)

Quick Start (30 minutes)

Step 1: Check Historical Data (5 min)

# Example: Error data for meta-cc
meta-cc query-tools --status error | jq '. | length'
# Output: 1336 errors ✅ (>100 threshold)

# Example: Test failures from CI logs
grep "FAILED" ci-logs/*.txt | wc -l
# Output: 427 failures ✅

Threshold: ≥100 instances for statistical confidence

Step 2: Define Detection Rule (10 min)

Tool: validate-path.sh
Prevents: "File not found" errors
Detection:
  - Error message matches: "no such file or directory"
  - OR "cannot read file"
  - OR "file does not exist"
Confidence: High (90%+) - deterministic check

Step 3: Apply Rule to Historical Data (10 min)

# Count matches
grep -E "(no such file|cannot read|does not exist)" errors.log | wc -l
# Output: 163 errors (12.2% of total)

# Sample manual validation (30 errors)
# True positives: 28/30 (93.3%)
# Adjusted: 163 * 0.933 = 152 preventable ✅

Step 4: Calculate Confidence (5 min)

Confidence = Data Quality × Accuracy × Logical Correctness
           = 0.85 × 0.933 × 1.0
           = 0.79 (High confidence)

Result: Tool would have prevented 152 errors with 79% confidence.

Four-Phase Process

Phase 1: Data Collection

1. Identify Data Sources

For Claude Code / meta-cc:

# Error history
meta-cc query-tools --status error

# User pain points
meta-cc query-user-messages --pattern "error|fail|broken"

# Error context
meta-cc query-context --error-signature "..."

For other projects:

• Git history (commits, diffs, blame)
• CI/CD logs (test failures, build errors)
• Application logs (runtime errors)
• Issue trackers (bug reports)

2. Quantify Baseline

Metrics needed:

• Volume: Total instances (e.g., 1,336 errors)
• Rate: Frequency (e.g., 5.78% error rate)
• Distribution: Category breakdown (e.g., file-not-found: 12.2%)
• Impact: Cost (e.g., MTTD: 15 min, MTTR: 30 min)

Phase 2: Pattern Definition

1. Create Detection Rules

For each tool/methodology:

what_it_prevents: Error type or failure mode
detection_rule: Pattern matching heuristic
confidence: Estimated accuracy (high/medium/low)

2. Define Success Criteria

prevention: Message matches AND tool would catch it
speedup: Tool faster than manual debugging
reliability: No false positives/negatives in sample

Phase 3: Validation Execution

1. Apply Rules to Historical Data

# Pseudo-code
for instance in historical_data:
  category = classify(instance)
  tool = find_applicable_tool(category)
  if would_have_prevented(tool, instance):
    count_prevented++

prevention_rate = count_prevented / total * 100

2. Sample Manual Validation

Sample size: 30 instances (95% confidence)
For each: "Would tool have prevented this?"
Calculate: True positive rate, False positive rate
Adjust: prevention_claim * true_positive_rate

Example (Bootstrap-003):

Sample: 30/317 claimed prevented
True positives: 28 (93.3%)
Adjusted: 317 * 0.933 = 296 errors
Confidence: High (93%+)

3. Measure Performance

# Tool time
time tool.sh < test_input
# Output: 0.05s

# Manual time (estimate from historical data)
# Average debug time: 15 min = 900s

# Speedup: 900 / 0.05 = 18,000x

Phase 4: Confidence Assessment

Confidence Formula:

Confidence = D × A × L

Where:
D = Data Quality (0.5-1.0)
A = Accuracy (True Positive Rate, 0.5-1.0)
L = Logical Correctness (0.5-1.0)

Data Quality (D):

• 1.0: Complete, accurate, representative
• 0.8-0.9: Minor gaps or biases
• 0.6-0.7: Significant gaps
• <0.6: Unreliable data

Accuracy (A):

• 1.0: 100% true positive rate (verified)
• 0.8-0.95: High (sample validation 80-95%)
• 0.6-0.8: Medium (60-80%)
• <0.6: Low (unreliable pattern matching)

Logical Correctness (L):

• 1.0: Deterministic (tool directly addresses root cause)
• 0.8-0.9: High correlation (strong evidence)
• 0.6-0.7: Moderate correlation
• <0.6: Weak or speculative

Example (Bootstrap-003):

D = 0.85 (Complete error logs, minor gaps in context)
A = 0.933 (93.3% true positive rate from sample)
L = 1.0 (File validation is deterministic)

Confidence = 0.85 × 0.933 × 1.0 = 0.79 (High)

Interpretation:

• ≥0.75: High confidence (publishable)
• 0.60-0.74: Medium confidence (needs caveats)
• 0.45-0.59: Low confidence (suggestive, not conclusive)
• <0.45: Insufficient confidence (need prospective validation)

Comparison: Retrospective vs Prospective

When to use each:

• Retrospective: Fast validation, high data volume, observable patterns
• Prospective: Behavioral effects, UX, emergent properties
• Hybrid: Retrospective first, limited prospective for edge cases

Success Criteria

Retrospective validation succeeded when:

1. Sufficient data: ≥100 instances analyzed
2. High confidence: ≥0.75 overall confidence score
3. Sample validated: ≥80% true positive rate
4. Impact quantified: Prevention % or speedup measured
5. Time savings: 40-60% faster than prospective validation

Bootstrap-003 Validation:

• ✅ Data: 1,336 errors analyzed
• ✅ Confidence: 0.79 (high)
• ✅ Sample: 93.3% true positive rate
• ✅ Impact: 23.7% error prevention
• ✅ Time: 3 hours vs 2+ weeks (prospective)

Related Skills

Parent framework:

• methodology-bootstrapping - Core OCA cycle

Complementary acceleration:

• rapid-convergence - Fast iteration (uses retrospective)
• baseline-quality-assessment - Strong iteration 0

References

Core guide:

• Four-Phase Process - Detailed methodology
• Confidence Calculation - Statistical rigor
• Detection Rules - Pattern matching guide

Examples:

• Error Recovery Validation - Bootstrap-003

Status: ✅ Validated | Bootstrap-003 | 0.79 confidence | 40-60% time reduction