Robustness Test Checklist
Systematic checklist of robustness tests for empirical research. Use this to ensure comprehensive testing before submission.
Core Robustness Categories
1. Alternative Samples
| Test |
Description |
When to Use |
| Exclude outliers |
Winsorize/trim at different levels (0.5%, 2%, 5%) |
Always |
| Drop financial firms |
Exclude SIC 6000-6999 |
If not already excluded |
| Drop regulated industries |
Exclude utilities, telecoms |
Industry-specific effects |
| Different time periods |
Split sample pre/post crisis, early/late |
Results may be period-specific |
| Geographic subsamples |
By region, state, country |
External validity |
| Size subsamples |
Small vs. large firms |
Heterogeneous effects |
| Balanced panel |
Require continuous observations |
Survivorship concerns |
2. Alternative Specifications
| Test |
Description |
When to Use |
| Different fixed effects |
Firm, industry×year, state×year |
Control for unobservables |
| Additional controls |
Add variables referees might suggest |
Omitted variable concerns |
| Drop controls |
Verify not over-controlling |
Mediator concerns |
| Different clustering |
Firm, industry, state, two-way |
Inference robustness |
| Different standard errors |
Bootstrap, Newey-West, Driscoll-Kraay |
Serial/cross-sectional correlation |
| Nonlinear specifications |
Quadratic terms, splines |
Linearity assumption |
| Log vs. level |
Transform dependent variable |
Skewed distributions |
3. Alternative Measures
| Test |
Description |
When to Use |
| Alternative dependent variable |
Different proxy for same concept |
Measurement concerns |
| Alternative treatment measure |
Continuous vs. binary, different threshold |
Treatment definition |
| Alternative control measures |
Different proxies for size, leverage, etc. |
Standard practice |
| Scaled differently |
By assets, sales, employees |
Scaling choice matters |
4. Identification Tests
| Test |
Description |
When to Use |
| Placebo/Falsification |
|
|
| Placebo timing |
Fake treatment 1-3 years before actual |
DiD parallel trends |
| Placebo outcome |
Effect on outcome that shouldn't be affected |
Specificity of mechanism |
| Placebo treatment |
Random assignment of treatment |
Rule out spurious correlation |
| Pre-trends |
|
|
| Event study plot |
Coefficient for each pre/post period |
Visual parallel trends |
| Joint F-test |
Test pre-period coefficients = 0 |
Statistical parallel trends |
| Endogeneity |
|
|
| Instrumental variables |
Find exogenous variation |
Selection concerns |
| Heckman selection |
Model selection explicitly |
Sample selection |
| Propensity score matching |
Match treated/control |
Observable selection |
| Entropy balancing |
Reweight to balance covariates |
Covariate imbalance |
| Regression discontinuity |
If threshold exists |
Sharp identification |
5. Inference Robustness
| Test |
Description |
When to Use |
| Wild cluster bootstrap |
Small number of clusters |
<50 clusters |
| Randomization inference |
Permutation-based p-values |
Few treated units |
| Conley standard errors |
Spatial correlation |
Geographic data |
| Multiple hypothesis correction |
Bonferroni, FDR |
Many outcomes tested |
DiD-Specific Tests
For difference-in-differences designs:
IV-Specific Tests
For instrumental variables:
Quick Commands
- "robustness checklist" - Full checklist for current paper
- "DiD robustness" - DiD-specific tests only
- "what tests for [method]" - Tests for specific identification strategy
- "referee-proof" - Most commonly requested tests
- "prioritize tests" - Rank by importance for your setting
Referee-Proof Minimum
At minimum, most papers should include:
- Sample robustness: Exclude outliers, alternative time periods
- Specification robustness: Alternative fixed effects, controls
- Measurement robustness: Alternative variable definitions
- Inference robustness: Alternative clustering
- Identification test: At least one placebo or pre-trend test
Reporting Template
For robustness tables:
Table X: Robustness Tests
Panel A: Alternative Samples
(1) Baseline
(2) Exclude financial firms
(3) Exclude 2008-2009
(4) Winsorize at 5%
Panel B: Alternative Specifications
(5) Add industry×year FE
(6) Control for firm age
(7) Cluster by industry
Panel C: Alternative Measures
(8) Alternative dependent variable
(9) Continuous treatment measure
1---2name: robustness3description: Checklist of empirical robustness tests for finance/economics papers4---56# Robustness Test Checklist78Systematic checklist of robustness tests for empirical research. Use this to ensure comprehensive testing before submission.910---1112## Core Robustness Categories1314### 1. Alternative Samples1516| Test | Description | When to Use |17|------|-------------|-------------|18| Exclude outliers | Winsorize/trim at different levels (0.5%, 2%, 5%) | Always |19| Drop financial firms | Exclude SIC 6000-6999 | If not already excluded |20| Drop regulated industries | Exclude utilities, telecoms | Industry-specific effects |21| Different time periods | Split sample pre/post crisis, early/late | Results may be period-specific |22| Geographic subsamples | By region, state, country | External validity |23| Size subsamples | Small vs. large firms | Heterogeneous effects |24| Balanced panel | Require continuous observations | Survivorship concerns |2526### 2. Alternative Specifications2728| Test | Description | When to Use |29|------|-------------|-------------|30| Different fixed effects | Firm, industry×year, state×year | Control for unobservables |31| Additional controls | Add variables referees might suggest | Omitted variable concerns |32| Drop controls | Verify not over-controlling | Mediator concerns |33| Different clustering | Firm, industry, state, two-way | Inference robustness |34| Different standard errors | Bootstrap, Newey-West, Driscoll-Kraay | Serial/cross-sectional correlation |35| Nonlinear specifications | Quadratic terms, splines | Linearity assumption |36| Log vs. level | Transform dependent variable | Skewed distributions |3738### 3. Alternative Measures3940| Test | Description | When to Use |41|------|-------------|-------------|42| Alternative dependent variable | Different proxy for same concept | Measurement concerns |43| Alternative treatment measure | Continuous vs. binary, different threshold | Treatment definition |44| Alternative control measures | Different proxies for size, leverage, etc. | Standard practice |45| Scaled differently | By assets, sales, employees | Scaling choice matters |4647### 4. Identification Tests4849| Test | Description | When to Use |50|------|-------------|-------------|51| **Placebo/Falsification** | | |52| Placebo timing | Fake treatment 1-3 years before actual | DiD parallel trends |53| Placebo outcome | Effect on outcome that shouldn't be affected | Specificity of mechanism |54| Placebo treatment | Random assignment of treatment | Rule out spurious correlation |55| **Pre-trends** | | |56| Event study plot | Coefficient for each pre/post period | Visual parallel trends |57| Joint F-test | Test pre-period coefficients = 0 | Statistical parallel trends |58| **Endogeneity** | | |59| Instrumental variables | Find exogenous variation | Selection concerns |60| Heckman selection | Model selection explicitly | Sample selection |61| Propensity score matching | Match treated/control | Observable selection |62| Entropy balancing | Reweight to balance covariates | Covariate imbalance |63| Regression discontinuity | If threshold exists | Sharp identification |6465### 5. Inference Robustness6667| Test | Description | When to Use |68|------|-------------|-------------|69| Wild cluster bootstrap | Small number of clusters | <50 clusters |70| Randomization inference | Permutation-based p-values | Few treated units |71| Conley standard errors | Spatial correlation | Geographic data |72| Multiple hypothesis correction | Bonferroni, FDR | Many outcomes tested |7374---7576## DiD-Specific Tests7778For difference-in-differences designs:7980- [ ] Event study with pre-treatment coefficients81- [ ] Parallel trends test (formal)82- [ ] Bacon decomposition (staggered treatment)83- [ ] Callaway-Sant'Anna or Sun-Abraham estimator (heterogeneous treatment effects)84- [ ] Placebo treatment timing85- [ ] Vary treatment window86- [ ] Triple-difference if possible87- [ ] Exclude always-treated or never-treated8889---9091## IV-Specific Tests9293For instrumental variables:9495- [ ] First-stage F-statistic (>10, prefer >100)96- [ ] Weak instrument robust inference (Anderson-Rubin)97- [ ] Overidentification test (if multiple IVs)98- [ ] Exclusion restriction discussion99- [ ] Reduced form results100- [ ] Compare OLS vs. IV magnitudes101102---103104## Quick Commands105106- **"robustness checklist"** - Full checklist for current paper107- **"DiD robustness"** - DiD-specific tests only108- **"what tests for [method]"** - Tests for specific identification strategy109- **"referee-proof"** - Most commonly requested tests110- **"prioritize tests"** - Rank by importance for your setting111112---113114## Referee-Proof Minimum115116At minimum, most papers should include:1171181. **Sample robustness:** Exclude outliers, alternative time periods1192. **Specification robustness:** Alternative fixed effects, controls1203. **Measurement robustness:** Alternative variable definitions1214. **Inference robustness:** Alternative clustering1225. **Identification test:** At least one placebo or pre-trend test123124---125126## Reporting Template127128For robustness tables:129130```131Table X: Robustness Tests132Panel A: Alternative Samples133 (1) Baseline134 (2) Exclude financial firms135 (3) Exclude 2008-2009136 (4) Winsorize at 5%137138Panel B: Alternative Specifications139 (5) Add industry×year FE140 (6) Control for firm age141 (7) Cluster by industry142143Panel C: Alternative Measures144 (8) Alternative dependent variable145 (9) Continuous treatment measure146```
brycewang-stanford