gabrielmoreira Designing Experiments
Helps choose and specify a research design before data analysis starts. This skill owns study-design decisions: what is treated, what is compared, what outcome is measured, which assumptions are required, which validation or recovery experiment should follow a failed scientific experiment, and which design is defensible.
It does not fit causal models, estimate treatment effects, interpret fitted model output from existing data, or debug software/build failures.
Decision Framework
Control Group?
- Yes: Go to Step 2.
- No: Consider Interrupted Time Series (ITS).
Unit Structure?
- Single Treated Unit:
- With multiple controls: Synthetic Control (SC).
- No controls: ITS.
- Multiple Treated Units:
- With control group: Difference-in-Differences (DiD).
- Single Treated Unit:
Time Structure?
- Panel Data (Multiple units over time): Required for DiD and SC.
- Time Series (Single unit over time): Required for ITS.
Method Quick Reference
- Difference-in-Differences (DiD): Compares trend changes between treated and control groups. Assumes Parallel Trends.
- Interrupted Time Series (ITS): Analyzes trend/level change for a single unit after intervention. Assumes Trend Continuity.
- Synthetic Control (SC): Constructs a synthetic counterfactual from weighted control units. Assumes Convex Hull (treated unit within range of controls).
Failed Experiment Recovery
When a scientific experiment or optimization plan produces weak or contradictory results, use the same design surface to:
- Separate implementation or measurement errors from design-assumption failures.
- Identify which assumption should be tested next.
- Define a minimal validation experiment before abandoning the approach.
- State the decision rule for continuing, revising, or stopping the line of work.