Exp Lens Estimand Clarity

Estimand Clarity Experimental Design Lens

Philosophical Mode: Evidential Primary Question: "What exactly is the claim?" Focus: Effect Definition, Target Population, Outcome Specification, Comparator, Aggregation Level, Complication Handling

When to Use

• Experiment has unclear or shifting hypotheses
• Multiple stakeholders interpret results differently
• Claims mix causal and predictive language
• User invokes /exp-lens-estimand-clarity or /make-experiment-diag estimand

Critical Constraints

NEVER:

• Modify any source code or experiment files
• Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

ALWAYS:

• Decompose every stated claim into formal contrast notation (Treatment A vs Treatment B on Outcome Y in Population Z)
• Flag every mismatch between prose claims and code implementation
• Identify the aggregation level (unit, group, time) explicitly
• Document how complications (missing data, failures, exclusions) are handled
• BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Stated Claims & Hypotheses

• Find hypothesis statements, research questions, README claims
• Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms

Treatment Definition

• Find what intervention or manipulation is applied
• Look for: treatment, intervention, method, approach, condition, configuration, ablation

Outcome Definition

• Find what is measured as the result
• Look for: outcome, metric, measure, endpoint, target, response, dependent

Population & Scope

• Find what units, datasets, or contexts the claim covers
• Look for: dataset, population, sample, domain, task, benchmark, scenario, setting

Complication Handling

• Find how missing data, failures, timeouts, or exclusions are handled
• Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete

Step 2: Extract the Implicit Estimand

Answer each question from the code (not the docs):

1. What is the treatment?
2. What is the comparator?
3. What is the outcome?
4. What is the population?
5. What is the time horizon?
6. How are complications handled?

Step 3: Compare Claims to Implementation

Compare the explicit claims (from docs/papers) to the implicit estimand (from code). Flag mismatches between what the prose asserts and what the implementation actually measures.

CRITICAL — Analyze Claim Precision: For every stated claim:

• Can you write it as a formal contrast (Treatment A vs Treatment B on Outcome Y in Population Z)? If not, what is ambiguous?
• Does the code measure what the prose claims?

Step 4: Create the Optional Claim-Flow Diagram

If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.

Direction: TB (claim flows from intervention through measurement to conclusion)

Small diagram: 4-6 nodes showing Treatment → Mechanism → Outcome → Claim

Node Styling:

• cli class: treatment/intervention nodes
• handler class: mechanism/pipeline nodes
• output class: measured outcome nodes
• phase class: stated claim nodes
• gap class: ambiguity or mismatch between claim and measurement

Step 5: Write Output

Write the analysis to: temp/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md

Output Template

# Estimand Clarity Analysis: {Experiment Name}

**Lens:** Estimand Clarity (Evidential)
**Question:** What exactly is the claim?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Estimand Decomposition

| Component | Stated | Implemented | Match? |
|-----------|--------|-------------|--------|
| Treatment | {from prose} | {from code} | Yes / No / Partial |
| Comparator | {from prose} | {from code} | Yes / No / Partial |
| Outcome | {from prose} | {from code} | Yes / No / Partial |
| Population | {from prose} | {from code} | Yes / No / Partial |
| Time Horizon | {from prose} | {from code} | Yes / No / Partial |
| Complication Handling | {from prose} | {from code} | Yes / No / Partial |

## Claim Precision Assessment

| Claim | Formal Contrast | Ambiguities |
|-------|----------------|-------------|
| "{stated claim}" | Treatment A vs B on Y in Z | {list ambiguities} |

## Claim-Flow Diagram (Optional)

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    TREATMENT["Treatment<br/>━━━━━━━━━━<br/>{intervention name}"]
    MECHANISM["Mechanism<br/>━━━━━━━━━━<br/>{pipeline step}"]
    OUTCOME["Measured Outcome<br/>━━━━━━━━━━<br/>{metric}"]
    CLAIM["Stated Claim<br/>━━━━━━━━━━<br/>{claim text}"]
    MISMATCH["Mismatch<br/>━━━━━━━━━━<br/>{ambiguity description}"]

    TREATMENT --> MECHANISM
    MECHANISM --> OUTCOME
    OUTCOME --> CLAIM
    OUTCOME -.->|"diverges"| MISMATCH

    class TREATMENT cli;
    class MECHANISM handler;
    class OUTCOME output;
    class CLAIM phase;
    class MISMATCH gap;

Color Legend:

Ambiguity Register

Recommendations

1. {Specific action to resolve most critical ambiguity}
2. {Rewrite suggestion for vague claim}
3. {Code change to align implementation with stated estimand}

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For causal structure of the stated claim
- `/exp-lens-measurement-validity` - For whether the outcome metric is valid