# Exp Lens Validity Threats

> Create a validity threat matrix identifying alternative explanations and design mitigations. Adversarial lens answering "What alternative explanations survive?"

- Skill: `trecek/exp-lens-validity-threats` (Agent Skill)
- Install (CLI): `npx skillmds@latest add trecek/exp-lens-validity-threats`
- Raw SKILL.md: https://api.skillmd.com/api/skills/trecek/exp-lens-validity-threats/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: Trecek (https://skillmd.com/u/trecek)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/trecek/exp-lens-validity-threats

---


# Validity Threats Experimental Design Lens

**Philosophical Mode:** Adversarial
**Primary Question:** "What alternative explanations survive?"
**Focus:** History Effects, Instrumentation Changes, Selection Effects, Co-interventions, Treatment Diffusion

## When to Use

- Quasi-experimental designs without full randomization
- System experiments with environmental changes
- Longitudinal comparisons
- User invokes `/exp-lens-validity-threats` or `/make-experiment-diag validity`

## Critical Constraints

**NEVER:**
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

**ALWAYS:**
- Apply Campbell & Stanley's full threat taxonomy — do not skip threats just because they seem unlikely
- For every observed difference, enumerate at least 3 alternative explanations
- Assess mitigation strength honestly — "partially mitigated" is better than false confidence
- Distinguish threats that are ruled out by design from those that remain plausible
- BEFORE creating any diagram, LOAD the `/mermaid` skill using the Skill tool - this is MANDATORY

---

## Analysis Workflow

### Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

**Temporal Changes (History)**
- Find whether the environment changed between conditions
- Look for: version, update, release, change, drift, before, after, time, date

**Instrumentation Changes**
- Find whether measurement tools or methods changed
- Look for: log, metric, instrument, measurement, format, schema, API, breaking

**Selection & Filtering**
- Find whether groups are compared after differential filtering
- Look for: filter, exclude, select, subsample, eligible, criteria, dropout, attrition

**Co-interventions**
- Find whether multiple changes occurred simultaneously
- Look for: also, simultaneously, together, combined, alongside, additionally, migration

**Treatment Diffusion**
- Find whether control units could access treatment
- Look for: access, share, copy, learn, adopt, contaminate, crossover, spillover

### Step 2: Apply Threat Taxonomy

Apply Campbell & Stanley's threat taxonomy to the specific experiment. For each threat category:
1. Is it plausible in this context?
2. What design feature mitigates it?
3. How strong is the mitigation?

Build the threat matrix.

### Step 3: Analyze Alternative Explanations

**CRITICAL — Analyze Alternative Explanations:**
For every observed difference between conditions:
- List at least 3 alternative explanations besides the intended causal claim
- For each: What evidence rules it out? What evidence remains consistent with it?

### Step 4: Create the Diagram

Use the mermaid skill conventions to create a threat-flow diagram with:

**Direction:** `TB` (threats flow from sources through mitigations to residual risk)

**Subgraphs:**
- THREAT SOURCES
- DESIGN MITIGATIONS
- RESIDUAL THREATS

**Node Styling:**
- `gap` class: Active unmitigated threats
- `detector` class: Design mitigations
- `stateNode` class: Threat sources
- `handler` class: Partially mitigated threats
- `output` class: Successfully mitigated threats

### Step 5: Write Output

Write the diagram to: `temp/exp-lens-validity-threats/exp_diag_validity_threats_{YYYY-MM-DD_HHMMSS}.md`

---

## Output Template

```markdown
# Validity Threat Analysis: {Experiment Name}

**Lens:** Validity Threats (Adversarial)
**Question:** What alternative explanations survive?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Threat Matrix

| Threat | Category | Plausibility | Mitigation | Residual Risk |
|--------|----------|-------------|------------|---------------|
| {threat} | {Campbell-Stanley category} | {Low/Medium/High} | {design feature} | {Low/Medium/High} |

## Alternative Explanations

| Observed Difference | Alternative Explanation | Ruling-Out Evidence | Consistent Evidence |
|--------------------|------------------------|---------------------|---------------------|
| {difference} | {explanation} | {evidence} | {evidence} |

## Threat-Flow Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Sources ["THREAT SOURCES"]
        direction TB
        HIST["History Effect<br/>━━━━━━━━━━<br/>Environmental change<br/>between conditions"]
        INST["Instrumentation<br/>━━━━━━━━━━<br/>Measurement tool<br/>change"]
        SEL["Selection Effect<br/>━━━━━━━━━━<br/>Differential filtering<br/>of units"]
    end

    subgraph Mitigations ["DESIGN MITIGATIONS"]
        direction TB
        RAND["Randomization<br/>━━━━━━━━━━<br/>Controls selection<br/>and history"]
        CTRL["Control Group<br/>━━━━━━━━━━<br/>Absorbs environmental<br/>changes"]
    end

    subgraph Residual ["RESIDUAL THREATS"]
        direction TB
        DIFF["Treatment Diffusion<br/>━━━━━━━━━━<br/>Control access<br/>to treatment"]
        COINT["Co-interventions<br/>━━━━━━━━━━<br/>Simultaneous<br/>changes"]
    end

    HIST --> CTRL
    INST --> CTRL
    SEL --> RAND
    RAND --> CTRL
    CTRL --> DIFF
    CTRL --> COINT

    %% CLASS ASSIGNMENTS %%
    class HIST,INST,SEL stateNode;
    class RAND,CTRL detector;
    class DIFF,COINT gap;
```

## Campbell-Stanley Checklist

| Threat | Applicable? | Assessment |
|--------|-------------|------------|
| History | {Yes/No/Partial} | {assessment} |
| Maturation | {Yes/No/Partial} | {assessment} |
| Testing | {Yes/No/Partial} | {assessment} |
| Instrumentation | {Yes/No/Partial} | {assessment} |
| Statistical Regression | {Yes/No/Partial} | {assessment} |
| Selection | {Yes/No/Partial} | {assessment} |
| Experimental Mortality | {Yes/No/Partial} | {assessment} |
| Selection-Maturation Interaction | {Yes/No/Partial} | {assessment} |
| Diffusion of Treatments | {Yes/No/Partial} | {assessment} |
| Co-interventions | {Yes/No/Partial} | {assessment} |

## Key Findings

- {Description of most critical surviving alternative explanations}
```

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For DAG-based causal structure analysis
- `/exp-lens-severity-testing` - For statistical severity and power analysis