# Exp Lens Error Budget

> Analyze statistical error budget showing Type I/II errors, power, minimum detectable effect, multiplicity corrections, and sequential monitoring. Statistical lens answering "Are error risks sized and controlled?"

- Skill: `trecek/exp-lens-error-budget` (Agent Skill)
- Install (CLI): `npx skillmds add trecek/exp-lens-error-budget`
- Raw SKILL.md: https://api.skillmd.com/api/skills/trecek/exp-lens-error-budget/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Finance & Business
- Author: Trecek (https://skillmd.com/u/trecek)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/trecek/exp-lens-error-budget

---


# Error Budget Experimental Design Lens

**Philosophical Mode:** Statistical
**Primary Question:** "Are error risks sized and controlled?"
**Focus:** Type I/II Errors, Power, Minimum Detectable Effect, Multiplicity, Sequential Monitoring

## When to Use

- Need to verify statistical power before running an experiment
- Multiple comparisons are performed without a stated correction strategy
- Sequential testing or interim analysis is in use without defined stopping rules
- User invokes `/exp-lens-error-budget` or `/make-experiment-diag error`

## Critical Constraints

**NEVER:**
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Accept default alpha=0.05 without checking whether it is appropriate for the decision context

**ALWAYS:**
- Enumerate every statistical test and account for its error contribution
- Distinguish per-test error rates from family-wise error rates
- Flag any sequential peeking without a formal stopping rule as a critical defect
- Evaluate whether the minimum detectable effect is practically meaningful, not just statistically chosen
- BEFORE creating any diagram, LOAD the `/mermaid` skill using the Skill tool - this is MANDATORY

---

## Analysis Workflow

### Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

**Sample Size & Power**
- Find sample size calculations and power analyses
- Identify assumed effect sizes and the alpha used for sizing
- Look for: sample_size, power, n_samples, effect_size, alpha, significance

**Multiple Comparisons**
- Find how many statistical tests are performed in total
- Identify any multiplicity correction strategy
- Look for: test, compare, pvalue, p_value, bonferroni, fdr, holm, adjust, correct

**Sequential Analysis**
- Find whether data is analyzed as it arrives or at scheduled interim looks
- Identify formal stopping boundaries if any exist
- Look for: sequential, interim, early_stop, monitor, peek, accumulate, streaming

**Decision Thresholds**
- Find significance levels, confidence intervals, and decision rules
- Identify how raw test results are translated into accept/reject decisions
- Look for: threshold, alpha, significance, confidence, critical, reject, accept

**Effect Size Context**
- Find what effect sizes are practically meaningful for the decision
- Identify whether the minimum detectable effect is set by business context or convention
- Look for: effect, difference, improvement, minimum, meaningful, practical, clinical

### Step 2: Build the Error Budget

Construct a complete accounting of error risks:
1. What is the Type I error rate per test and family-wise across all tests?
2. What is the power for the minimum practically meaningful effect?
3. How many tests are performed and how is multiplicity handled (or not)?
4. Is there sequential monitoring, and what are the stopping boundaries?
5. Is each error tolerance justified by decision context, or is it a convention default?

### Step 3: Analyze Error Allocation

**CRITICAL — Analyze Error Allocation:**
For every statistical test:
- Is the error rate justified by the decision context (cost of false positive vs. false negative), or is it a default (alpha=0.05)?
- Are different tests allocated different error tolerances based on their importance?
- Does the power calculation assume an effect size that is realistic and practically significant?
- Is the family-wise error rate acceptable given the number of simultaneous decisions?

Rate each test's error budget alignment:
- **ALIGNED**: Error tolerances match the stakes of the decision
- **CONVENTIONAL**: Standard values used without explicit justification
- **MISALIGNED**: Error tolerances are inconsistent with decision consequences

### Step 4: Create Optional Decision-Flow Diagram

This lens is primarily argumentative and produces tables. An optional decision-flow diagram may be added when the test structure is non-trivial.

**Direction:** `TB` (decision flows from data through tests to conclusions)

**Minimal diagram structure:** Data sources → Statistical tests → Decision thresholds → Conclusions, with error rates labeled on edges

**Node Styling:**
- `cli` class: Data sources
- `handler` class: Statistical tests
- `detector` class: Significance thresholds
- `output` class: Decisions / conclusions
- `gap` class: Uncorrected multiplicity or missing stopping rules

### Step 5: Write Output

Write the analysis to: `temp/exp-lens-error-budget/exp_diag_error_budget_{YYYY-MM-DD_HHMMSS}.md`

---

## Output Template

```markdown
# Error Budget Analysis: {System / Experiment Name}

**Lens:** Error Budget (Statistical)
**Question:** Are error risks sized and controlled?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Error Budget

| Test | Alpha | Power | MDE | Multiplicity Correction |
|------|-------|-------|-----|------------------------|
| {test name} | {alpha} | {power %} | {minimum detectable effect} | {bonferroni / fdr / none} |

## Operating Characteristics Summary

| Metric | Value | Status |
|--------|-------|--------|
| Total tests | {N} | |
| Family-wise Type I error (uncorrected) | {value} | {OK / WARNING / CRITICAL} |
| Family-wise Type I error (corrected) | {value} | {OK / WARNING / CRITICAL} |
| Minimum power across tests | {value} | {OK / WARNING / CRITICAL} |
| Sample size adequacy | {adequate / underpowered / unknown} | |

## Sequential Monitoring Plan

{If sequential monitoring is used:}

| Look | Timing | Stopping Boundary | Alpha Spent |
|------|--------|------------------|-------------|
| {interim N} | {calendar / information fraction} | {boundary value} | {alpha} |

{If no sequential monitoring: "No interim analyses planned — full sequential peeking risk applies if data is inspected before planned end."}

## Decision Alignment Assessment

| Test | Decision Stakes | Error Tolerance Justification | Alignment |
|------|----------------|------------------------------|-----------|
| {test} | {what decision it drives} | {why this alpha/power} | {ALIGNED / CONVENTIONAL / MISALIGNED} |

## Optional Decision-Flow Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 40, 'rankSpacing': 50, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Data ["DATA SOURCES"]
        DS1["Data Source 1<br/>━━━━━━━━━━<br/>N observations"]
        DS2["Data Source 2<br/>━━━━━━━━━━<br/>N observations"]
    end

    subgraph Tests ["STATISTICAL TESTS"]
        T1["Primary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
        T2["Secondary Test<br/>━━━━━━━━━━<br/>alpha={a} power={p}"]
        MULTI["Multiplicity<br/>━━━━━━━━━━<br/>correction applied?"]
    end

    subgraph Thresholds ["DECISION THRESHOLDS"]
        THR["Significance Gate<br/>━━━━━━━━━━<br/>p < alpha"]
    end

    subgraph Decisions ["CONCLUSIONS"]
        DEC1["Accept Variant<br/>━━━━━━━━━━<br/>ship / launch"]
        DEC2["Reject Variant<br/>━━━━━━━━━━<br/>do not ship"]
        UNCORR["Uncorrected Tests<br/>━━━━━━━━━━<br/>inflated Type I error"]
    end

    DS1 -->|"alpha={a}"| T1
    DS2 -->|"alpha={a}"| T2
    T1 --> MULTI
    T2 --> MULTI
    MULTI -->|"corrected"| THR
    MULTI -.->|"no correction"| UNCORR
    THR -->|"p < alpha"| DEC1
    THR -->|"p >= alpha"| DEC2

    class DS1,DS2 cli;
    class T1,T2 handler;
    class MULTI,THR detector;
    class DEC1,DEC2 output;
    class UNCORR gap;
```

**Color Legend:**
| Color | Category | Description |
|-------|----------|-------------|
| Dark Blue | Data | Data sources feeding statistical tests |
| Orange | Tests | Statistical tests with error parameters |
| Red | Thresholds | Significance gates and decision boundaries |
| Dark Teal | Decisions | Conclusions reached from tests |
| Yellow | Gaps | Uncorrected multiplicity or missing stopping rules |
```

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for experimental lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-severity-testing` - For assessing whether tests are capable of detecting the errors they claim to test for
- `/exp-lens-variance-stability` - For diagnosing variance inflation that undermines power calculations