Exp Lens Error Budget

Error Budget Experimental Design Lens

Philosophical Mode: Statistical Primary Question: "Are error risks sized and controlled?" Focus: Type I/II Errors, Power, Minimum Detectable Effect, Multiplicity, Sequential Monitoring

When to Use

• Need to verify statistical power before running an experiment
• Multiple comparisons are performed without a stated correction strategy
• Sequential testing or interim analysis is in use without defined stopping rules
• User invokes /exp-lens-error-budget or /make-experiment-diag error

Critical Constraints

NEVER:

• Modify any source code files
• Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
• Accept default alpha=0.05 without checking whether it is appropriate for the decision context

ALWAYS:

• Enumerate every statistical test and account for its error contribution
• Distinguish per-test error rates from family-wise error rates
• Flag any sequential peeking without a formal stopping rule as a critical defect
• Evaluate whether the minimum detectable effect is practically meaningful, not just statistically chosen
• BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Sample Size & Power

• Find sample size calculations and power analyses
• Identify assumed effect sizes and the alpha used for sizing
• Look for: sample_size, power, n_samples, effect_size, alpha, significance

Multiple Comparisons

• Find how many statistical tests are performed in total
• Identify any multiplicity correction strategy
• Look for: test, compare, pvalue, p_value, bonferroni, fdr, holm, adjust, correct

Sequential Analysis

• Find whether data is analyzed as it arrives or at scheduled interim looks
• Identify formal stopping boundaries if any exist
• Look for: sequential, interim, early_stop, monitor, peek, accumulate, streaming

Decision Thresholds

• Find significance levels, confidence intervals, and decision rules
• Identify how raw test results are translated into accept/reject decisions
• Look for: threshold, alpha, significance, confidence, critical, reject, accept

Effect Size Context

• Find what effect sizes are practically meaningful for the decision
• Identify whether the minimum detectable effect is set by business context or convention
• Look for: effect, difference, improvement, minimum, meaningful, practical, clinical

Step 2: Build the Error Budget

Construct a complete accounting of error risks:

1. What is the Type I error rate per test and family-wise across all tests?
2. What is the power for the minimum practically meaningful effect?
3. How many tests are performed and how is multiplicity handled (or not)?
4. Is there sequential monitoring, and what are the stopping boundaries?
5. Is each error tolerance justified by decision context, or is it a convention default?

Step 3: Analyze Error Allocation

CRITICAL — Analyze Error Allocation: For every statistical test:

• Is the error rate justified by the decision context (cost of false positive vs. false negative), or is it a default (alpha=0.05)?
• Are different tests allocated different error tolerances based on their importance?
• Does the power calculation assume an effect size that is realistic and practically significant?
• Is the family-wise error rate acceptable given the number of simultaneous decisions?

Rate each test's error budget alignment:

• ALIGNED: Error tolerances match the stakes of the decision
• CONVENTIONAL: Standard values used without explicit justification
• MISALIGNED: Error tolerances are inconsistent with decision consequences

Step 4: Create Optional Decision-Flow Diagram

This lens is primarily argumentative and produces tables. An optional decision-flow diagram may be added when the test structure is non-trivial.

Direction: TB (decision flows from data through tests to conclusions)

Minimal diagram structure: Data sources → Statistical tests → Decision thresholds → Conclusions, with error rates labeled on edges

Node Styling:

• cli class: Data sources
• handler class: Statistical tests
• detector class: Significance thresholds
• output class: Decisions / conclusions
• gap class: Uncorrected multiplicity or missing stopping rules

Step 5: Write Output

Write the analysis to: temp/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md

Output Template

# Error Budget Analysis: {System / Experiment Name}

**Lens:** Error Budget (Statistical)
**Question:** Are error risks sized and controlled?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Error Budget

| Test | Alpha | Power | MDE | Multiplicity Correction |
|------|-------|-------|-----|------------------------|
| {test name} | {alpha} | {power %} | {minimum detectable effect} | {bonferroni / fdr / none} |

## Operating Characteristics Summary

| Metric | Value | Status |
|--------|-------|--------|
| Total tests | {N} | |
| Family-wise Type I error (uncorrected) | {value} | {OK / WARNING / CRITICAL} |
| Family-wise Type I error (corrected) | {value} | {OK / WARNING / CRITICAL} |
| Minimum power across tests | {value} | {OK / WARNING / CRITICAL} |
| Sample size adequacy | {adequate / underpowered / unknown} | |

## Sequential Monitoring Plan

{If sequential monitoring is used:}

| Look | Timing | Stopping Boundary | Alpha Spent |
|------|--------|------------------|-------------|
| {interim N} | {calendar / information fraction} | {boundary value} | {alpha} |

{If no sequential monitoring: "No interim analyses planned — full sequential peeking risk applies if data is inspected before planned end."}

## Decision Alignment Assessment

| Test | Decision Stakes | Error Tolerance Justification | Alignment |
|------|----------------|------------------------------|-----------|
| {test} | {what decision it drives} | {why this alpha/power} | {ALIGNED / CONVENTIONAL / MISALIGNED} |

## Optional Decision-Flow Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 40, 'rankSpacing': 50, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Data ["DATA SOURCES"]
        DS1["Data Source 1<br/>━━━━━━━━━━<br/>N observations"]
        DS2["Data Source 2<br/>━━━━━━━━━━<br/>N observations"]
    end

    subgraph Tests ["STATISTICAL TESTS"]
        T1["Primary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
        T2["Secondary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
        MULTI["Multiplicity<br/>━━━━━━━━━━<br/>correction applied?"]
    end

    subgraph Thresholds ["DECISION THRESHOLDS"]
        THR["Significance Gate<br/>━━━━━━━━━━<br/>p < alpha"]
    end

    subgraph Decisions ["CONCLUSIONS"]
        DEC1["Accept Variant<br/>━━━━━━━━━━<br/>ship / launch"]
        DEC2["Reject Variant<br/>━━━━━━━━━━<br/>do not ship"]
        UNCORR["Uncorrected Tests<br/>━━━━━━━━━━<br/>inflated Type I error"]
    end

    DS1 -->|"alpha={a}"| T1
    DS2 -->|"alpha={a}"| T2
    T1 --> MULTI
    T2 --> MULTI
    MULTI -->|"corrected"| THR
    MULTI -.->|"no correction"| UNCORR
    THR -->|"p < alpha"| DEC1
    THR -->|"p >= alpha"| DEC2

    class DS1,DS2 cli;
    class T1,T2 handler;
    class MULTI,THR detector;
    class DEC1,DEC2 output;
    class UNCORR gap;

Color Legend:

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for experimental lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-severity-testing` - For assessing whether tests are capable of detecting the errors they claim to test for
- `/exp-lens-variance-stability` - For diagnosing variance inflation that undermines power calculations