Exp Lens Validity Threats

Validity Threats Experimental Design Lens

Philosophical Mode: Adversarial Primary Question: "What alternative explanations survive?" Focus: History Effects, Instrumentation Changes, Selection Effects, Co-interventions, Treatment Diffusion

When to Use

• Quasi-experimental designs without full randomization
• System experiments with environmental changes
• Longitudinal comparisons
• User invokes /exp-lens-validity-threats or /make-experiment-diag validity

Critical Constraints

NEVER:

• Modify any source code files
• Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

ALWAYS:

• Apply Campbell & Stanley's full threat taxonomy — do not skip threats just because they seem unlikely
• For every observed difference, enumerate at least 3 alternative explanations
• Assess mitigation strength honestly — "partially mitigated" is better than false confidence
• Distinguish threats that are ruled out by design from those that remain plausible
• BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Temporal Changes (History)

• Find whether the environment changed between conditions
• Look for: version, update, release, change, drift, before, after, time, date

Instrumentation Changes

• Find whether measurement tools or methods changed
• Look for: log, metric, instrument, measurement, format, schema, API, breaking

Selection & Filtering

• Find whether groups are compared after differential filtering
• Look for: filter, exclude, select, subsample, eligible, criteria, dropout, attrition

Co-interventions

• Find whether multiple changes occurred simultaneously
• Look for: also, simultaneously, together, combined, alongside, additionally, migration

Treatment Diffusion

• Find whether control units could access treatment
• Look for: access, share, copy, learn, adopt, contaminate, crossover, spillover

Step 2: Apply Threat Taxonomy

Apply Campbell & Stanley's threat taxonomy to the specific experiment. For each threat category:

1. Is it plausible in this context?
2. What design feature mitigates it?
3. How strong is the mitigation?

Build the threat matrix.

Step 3: Analyze Alternative Explanations

CRITICAL — Analyze Alternative Explanations: For every observed difference between conditions:

• List at least 3 alternative explanations besides the intended causal claim
• For each: What evidence rules it out? What evidence remains consistent with it?

Step 4: Create the Diagram

Use the mermaid skill conventions to create a threat-flow diagram with:

Direction: TB (threats flow from sources through mitigations to residual risk)

Subgraphs:

• THREAT SOURCES
• DESIGN MITIGATIONS
• RESIDUAL THREATS

Node Styling:

• gap class: Active unmitigated threats
• detector class: Design mitigations
• stateNode class: Threat sources
• handler class: Partially mitigated threats
• output class: Successfully mitigated threats

Step 5: Write Output

Write the diagram to: temp/exp-lens-validity-threats/exp_diag_validity_threats_{YYYY-MM-DD_HHMMSS}.md

Output Template

# Validity Threat Analysis: {Experiment Name}

**Lens:** Validity Threats (Adversarial)
**Question:** What alternative explanations survive?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Threat Matrix

| Threat | Category | Plausibility | Mitigation | Residual Risk |
|--------|----------|-------------|------------|---------------|
| {threat} | {Campbell-Stanley category} | {Low/Medium/High} | {design feature} | {Low/Medium/High} |

## Alternative Explanations

| Observed Difference | Alternative Explanation | Ruling-Out Evidence | Consistent Evidence |
|--------------------|------------------------|---------------------|---------------------|
| {difference} | {explanation} | {evidence} | {evidence} |

## Threat-Flow Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Sources ["THREAT SOURCES"]
        direction TB
        HIST["History Effect<br/>━━━━━━━━━━<br/>Environmental change<br/>between conditions"]
        INST["Instrumentation<br/>━━━━━━━━━━<br/>Measurement tool<br/>change"]
        SEL["Selection Effect<br/>━━━━━━━━━━<br/>Differential filtering<br/>of units"]
    end

    subgraph Mitigations ["DESIGN MITIGATIONS"]
        direction TB
        RAND["Randomization<br/>━━━━━━━━━━<br/>Controls selection<br/>and history"]
        CTRL["Control Group<br/>━━━━━━━━━━<br/>Absorbs environmental<br/>changes"]
    end

    subgraph Residual ["RESIDUAL THREATS"]
        direction TB
        DIFF["Treatment Diffusion<br/>━━━━━━━━━━<br/>Control access<br/>to treatment"]
        COINT["Co-interventions<br/>━━━━━━━━━━<br/>Simultaneous<br/>changes"]
    end

    HIST --> CTRL
    INST --> CTRL
    SEL --> RAND
    RAND --> CTRL
    CTRL --> DIFF
    CTRL --> COINT

    %% CLASS ASSIGNMENTS %%
    class HIST,INST,SEL stateNode;
    class RAND,CTRL detector;
    class DIFF,COINT gap;

Campbell-Stanley Checklist

Key Findings

• {Description of most critical surviving alternative explanations}

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For DAG-based causal structure analysis
- `/exp-lens-severity-testing` - For statistical severity and power analysis