Causal Design

Causal Design

Design and audit identification strategies for observational causal inference.

Output Path

Per rules/review-artefact-routing.md (auto-loads in research projects (path-scoped to paper-*/ and paper/)):

• Source slug: causal-design
• Write reports to: reviews/<scope>/causal-design/<YYYY-MM-DD-HHMM>.md inside the project, where <scope> is the paper slug (e.g. paper-philtech) for paper-level audits or _project for project-level reviews. Path is relative to the research project root, not the Task-Management repo.
• Never at project root (./CRITIC-REPORT.md-style filenames are forbidden — pre-rule layout).
• Idempotency: if today's timestamp exists, append a same-day descriptor to the path base ({date}-HHMM-revision.md, {date}-HHMM-r2.md, {date}-HHMM-pre-submission.md) — never overwrite.
• Index update: if reviews/INDEX.md exists, write a one-line entry under "Latest per source" pointing at the new file. Otherwise review-recap will rebuild the index next time it runs.
• Infrastructure repos (Task-Management, atlas-workspace, etc.): this section does not apply — the path-scoped rule won't load there.

Modes

Default: Design. If the user points to an existing paper or estimation script, auto-select Audit mode.

When to Use

• Choosing an identification strategy for an observational study
• Stress-testing whether an existing strategy is credible
• Verifying that code implements the claimed identification design
• Mapping causal claims to their identifying assumptions

When NOT to Use

• Experimental design (RCTs, surveys, factorial) -- use experiment-design
• Running the analysis or generating results -- use data-analysis
• Literature search or citation gathering -- use an installed literature workflow or scholarly search
• Proofreading or compiling the paper -- use proofread, latex

Shared References

• Method probing questions: shared/method-probing-questions.md — ask before running any analysis (DiD, IV, RDD sections)
• Validation tiers: shared/validation-tiers.md — declare tier before designing strategy
• Escalation protocol: shared/escalation-protocol.md — escalate when identification is vague or unsound

Mode: Design

Phase 1: Interview

Before opening the interview, confirm the project's validation tier per shared/validation-tiers.md — Exploratory designs warrant lighter identification stress-testing than Publication-ready ones. Use shared/method-probing-questions.md as the interview backbone; the prompts below adapt those probes to causal identification specifically.

Conduct a structured interview to understand the research setting. Ask these questions (adapt to what the user has already shared):

1. Causal question: What causal effect are you trying to estimate? What is the treatment? What is the outcome?
2. Variation: What source of variation in treatment do you exploit? Is it natural, policy-driven, institutional?
3. Confounders: What are the main threats to identification? What unobservables worry you?
4. Data structure: Panel, cross-section, or repeated cross-section? What units and time periods?
5. Institutional context: Any thresholds, cutoffs, rollout dates, or instruments available?
6. Prior literature: What identification strategies have others used for similar questions?

Do not proceed until the causal question and data structure are clear.

Phase 2: Strategy Selection

Read references/design-decision-tree.md and walk through the decision tree with the user's answers:

• Match the research setting to the strongest available strategy
• If multiple strategies are viable, rank them by credibility and discuss trade-offs
• If the setting does not support any strong strategy, say so explicitly -- do not force a weak design

Phase 3: Strategy Memo

Write a strategy memo using references/strategy-memo-template.md. Save to docs/causal-strategy.md (or project-appropriate location).

The memo must specify:

1. Estimand -- the exact causal parameter being estimated, in formal notation
2. Identification strategy -- how variation is generated and why it is exogenous
3. Key assumptions -- each one stated, with a defence or test plan
4. Threats and mitigations -- what could go wrong and how to address it
5. Diagnostics plan -- which tests to run before trusting the estimates
6. Robustness checks -- pre-committed alternative specifications
7. Alternative strategies considered -- why they were rejected

This memo is what data-analysis Phase 3 checks for before allowing estimation. It locks the research design per the design-before-results rule.

Phase 4: Adversarial Review

The reviewer follows shared/escalation-protocol.md — when identification is vague or assumptions are hand-waved, the reviewer escalates rather than accommodating.

Delegate an adversarial review to the domain-reviewer agent. Read references/causal-audit-prompt.md and pass it as the prompt to the fresh-context sub-agent mechanism:

Launch the domain-reviewer agent with this prompt:
"You are reviewing a causal identification strategy memo. [Insert contents of causal-audit-prompt.md, customised with the specific strategy chosen]. The memo is at [path]. Focus exclusively on identification credibility."

The agent will produce a report at reviews/<scope>/domain-reviewer/<YYYY-MM-DD-HHMM>.md in the project, where <scope> is the paper slug or _project.

Phase 5: Iterate

Present the domain-reviewer's findings to the user. For each issue flagged:

• Discuss whether it is a genuine threat or can be addressed
• Update the strategy memo if the design changes
• If the strategy is fundamentally flawed, return to Phase 2

Mode: Audit

Phase 1: Extract Claims

Read the paper (.tex files) and/or estimation scripts to extract every causal claim:

• What effects does the paper claim to estimate?
• What language is used? ("causal", "effect of", "impact of", "leads to")
• Are claims hedged appropriately or overstated?

Produce a numbered list of claims with their locations (file:line).

Phase 2: Map Estimands to Identification

For each causal claim, determine:

Flag any claim where:

• The estimand is undefined or vague
• The identification strategy is not stated
• Key assumptions are not listed or defended
• The strategy does not match the claim (e.g., claiming ATE but estimating LATE)

Phase 3: Assumption Diagnostics

For each identification strategy found, check whether the required diagnostics are present and passing:

DiD / Event Study:

• Pre-treatment parallel trends test (visual + formal)
• Staggered treatment handling (TWFE bias check, Callaway-Sant'Anna or Sun-Abraham if staggered)
• Anticipation effects check
• Treatment effect heterogeneity assessment

IV:

• First-stage F-statistic reported (> 10 for Stock-Yogo, > 104.7 for modern thresholds)
• Exclusion restriction argument (quality of narrative)
• Monotonicity discussion
• Over-identification test (if multiple instruments)
• Reduced form reported

RDD:

• McCrary density test (no bunching at cutoff)
• Bandwidth sensitivity (MSE-optimal + alternatives)
• Covariate balance at the cutoff
• Donut hole specification
• Placebo cutoffs

Synthetic Control:

• Pre-treatment fit quality (RMSPE)
• Donor pool selection justification
• Placebo tests (in-space, in-time)
• Leave-one-out robustness

Event Study:

• Pre-event coefficients jointly zero
• Dynamic treatment effects plotted
• Clean control group definition
• Anticipation effects addressed

Phase 4: Code-Design Alignment

If estimation code exists, verify it implements the claimed design:

• Does the regression specification match the paper's equations?
• Are standard errors computed correctly for the design? (clustered at the right level, heteroskedasticity-robust)
• Are the treatment and control groups defined as claimed?
• Are the diagnostics actually run, not just mentioned?
• Do robustness checks exist in code, or only in the text?

Audit Report

Produce an audit report at reviews/<scope>/causal-design/<YYYY-MM-DD-HHMM>.md (where <scope> is the paper slug or _project) with:

# Causal Audit Report

**Document:** [filename]
**Date:** YYYY-MM-DD
**Mode:** Audit

## Claims Inventory

[Numbered list of causal claims with locations]

## Estimand-Identification Map

[Table from Phase 2]

## Diagnostics Assessment

| Strategy | Diagnostic | Present? | Passing? | Notes |
|----------|-----------|----------|----------|-------|
| ... | ... | ... | ... | ... |

## Code-Design Alignment

[Phase 4 findings, or "N/A -- no code found"]

## Critical Issues

[List of issues that threaten identification credibility]

## Recommendations

[Ordered list of fixes, from most to least important]

Cross-References