Data Analysis: Estimation, Cost-Benefit & Distribution (jpam-data-analysis)
JPAM analysis has two layers most field-journal papers skip: beyond the causal estimate, reviewers
expect attention to cost-benefit and distributional consequences — who gains, who pays, and is
it worth it? The estimate answers "does the policy work"; the cost-benefit and distributional work
answers "should we do it, and for whom." Both must be reported honestly, with uncertainty carried
through.
When to trigger
- Producing the main estimates and the robustness/heterogeneity suite
- Adding (or being asked to add) cost-benefit or distributional analysis
- A reviewer questioned standard errors, robustness, or the policy-relevance of the magnitude
- Translating an effect size into a decision-relevant quantity (per-dollar, per-recipient, MVPF)
Estimation norms
- Report effects in policy-relevant units — percentage points, dollars, per-recipient, per-dollar-
spent — not just standardized coefficients.
- Robustness as a coherent suite, not a coefficient dump: alternative specifications, samples,
bandwidths/estimators, and a placebo where the design allows. Show the result is not knife-edge.
- Theory-driven heterogeneity (from
jpam-theory-building), pre-specified where possible; report
which subgroup tests are primary and adjust for multiplicity.
- Honest uncertainty — confidence intervals, not just stars; discuss precision when a null is
policy-relevant ("we can rule out effects larger than X").
Cost-benefit & distributional analysis (JPAM premium)
- Cost-benefit: monetize benefits and costs on a common basis, state the discount rate and the
perspective (government budget vs. society), and run sensitivity to key assumptions. Where suitable,
report the Marginal Value of Public Funds (MVPF) or benefit-cost ratio.
- Distributional: show who gains and who bears the cost (by income, race, region, recipient vs.
taxpayer). A positive average effect with regressive incidence is a different policy story — say so.
- Fiscal externalities: account for downstream budget effects (e.g., reduced transfers, increased
tax revenue) where the literature does.
- Carry uncertainty through to the cost-benefit conclusion — do not present a point ratio as if the
estimate were certain.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. JPAM is policy analysis — program evaluation is the core; DiD/IV/RDD and the policy-relevant magnitude are decisive.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or benjamini_hochberg.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley.
- Re-fit off one handle:
audit_result(result_id) lists missing checks + the exact
suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Decisive checks in the body, exhaustive battery in the appendix.
JF execution walkthrough.
Checklist
Anti-patterns
- Reporting only standardized effects a policymaker cannot act on
- A robustness "kitchen sink" that never states which checks address which threat
- Cost-benefit with a hidden discount rate or perspective, and no sensitivity analysis
- A flattering average effect that hides regressive distribution
- Presenting a benefit-cost ratio as certain when the underlying estimate has a wide CI
- Post hoc subgroup hunting presented as confirmatory
Calibration anchors (hedged)
- The cost-benefit and distributional layers are what most distinguish a JPAM analysis from a field-
economics paper — budget time for them, do not bolt them on at the end.
- An MVPF or benefit-cost ratio is only as credible as the estimate it rests on; report its sensitivity
to the effect-size CI and to the discount rate, not a single point.
- A precisely estimated null can be a publishable JPAM result if it rules out a policy-relevant effect
— frame it as "we can rule out effects larger than X," not "no effect."
Worked micro-example (illustrative)
An evaluation finds a job-training program raises quarterly earnings by $420 (95% CI $120–$720). The
JPAM analysis does not stop there: it converts this to a benefit-cost ratio (lifetime earnings gain
vs. per-participant cost) under a stated discount rate, runs sensitivity across the CI and discount
rate, shows the gain is concentrated among longer-tenured entrants (theory-driven heterogeneity), and
notes the program is net-positive to the government budget only above a take-up threshold. The policy
story is the package, not the $420. (Numbers illustrative.)
Output format
【Main estimate】effect in policy-relevant units (+ CI)
【Robustness】checks mapped to specific threats
【Heterogeneity】theory-driven subgroups + multiplicity handling
【Cost-benefit】perspective, discount rate, MVPF/BCR, sensitivity
【Distribution】who gains / who pays
【Next】jpam-tables-figures
Supplementary resources
1---2name: jpam-data-analysis3description: Use when running and reporting the estimation for a Journal of Policy Analysis and Management (JPAM) manuscript — program-evaluation estimates plus the cost-benefit and distributional analysis JPAM expects, with robustness, heterogeneity, and honest uncertainty. Guides analysis norms; it does not replace the identification design.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/jpam-data-analysis6---78# Data Analysis: Estimation, Cost-Benefit & Distribution (jpam-data-analysis)910JPAM analysis has two layers most field-journal papers skip: beyond the **causal estimate**, reviewers11expect attention to **cost-benefit** and **distributional** consequences — *who gains, who pays, and is12it worth it?* The estimate answers "does the policy work"; the cost-benefit and distributional work13answers "should we do it, and for whom." Both must be reported honestly, with uncertainty carried14through.1516## When to trigger1718- Producing the main estimates and the robustness/heterogeneity suite19- Adding (or being asked to add) cost-benefit or distributional analysis20- A reviewer questioned standard errors, robustness, or the policy-relevance of the magnitude21- Translating an effect size into a decision-relevant quantity (per-dollar, per-recipient, MVPF)2223## Estimation norms2425- **Report effects in policy-relevant units** — percentage points, dollars, per-recipient, per-dollar-26 spent — not just standardized coefficients.27- **Robustness as a coherent suite**, not a coefficient dump: alternative specifications, samples,28 bandwidths/estimators, and a placebo where the design allows. Show the result is not knife-edge.29- **Theory-driven heterogeneity** (from `jpam-theory-building`), pre-specified where possible; report30 which subgroup tests are primary and adjust for multiplicity.31- **Honest uncertainty** — confidence intervals, not just stars; discuss precision when a null is32 policy-relevant ("we can rule out effects larger than X").3334## Cost-benefit & distributional analysis (JPAM premium)3536- **Cost-benefit:** monetize benefits and costs on a common basis, state the discount rate and the37 perspective (government budget vs. society), and run sensitivity to key assumptions. Where suitable,38 report the **Marginal Value of Public Funds (MVPF)** or benefit-cost ratio.39- **Distributional:** show *who* gains and *who* bears the cost (by income, race, region, recipient vs.40 taxpayer). A positive average effect with regressive incidence is a different policy story — say so.41- **Fiscal externalities:** account for downstream budget effects (e.g., reduced transfers, increased42 tax revenue) where the literature does.43- **Carry uncertainty through** to the cost-benefit conclusion — do not present a point ratio as if the44 estimate were certain.4546## Execution bridge (StatsPAI / Stata MCP)4748Run the battery, don't just enumerate it. Full map:49[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). JPAM is policy analysis — program evaluation is the core; DiD/IV/RDD and the policy-relevant magnitude are decisive.5051- **Many outcomes / specifications:** `romano_wolf` (step-down FWER) or `benjamini_hochberg`.52- **OVB sensitivity:** `oster_delta` / `sensemakr`.53- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`.54- **Re-fit off one handle:** `audit_result(result_id)` lists missing checks + the exact55 `suggest_function` for each.56- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.5758Decisive checks in the body, exhaustive battery in the appendix.59[JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).60## Checklist6162- [ ] Effects reported in policy-relevant units, not only standardized63- [ ] Robustness suite addresses the design's specific vulnerabilities64- [ ] Heterogeneity tied to the theory of change; multiple-testing handled65- [ ] Confidence intervals reported; informative nulls discussed66- [ ] Cost-benefit with stated perspective, discount rate, and sensitivity (MVPF / BCR where apt)67- [ ] Distributional incidence shown — who gains, who pays68- [ ] Fiscal externalities considered where relevant69- [ ] Every number in the text matches the deposited replication output7071## Anti-patterns7273- Reporting only standardized effects a policymaker cannot act on74- A robustness "kitchen sink" that never states which checks address which threat75- Cost-benefit with a hidden discount rate or perspective, and no sensitivity analysis76- A flattering average effect that hides regressive distribution77- Presenting a benefit-cost ratio as certain when the underlying estimate has a wide CI78- Post hoc subgroup hunting presented as confirmatory7980## Calibration anchors (hedged)8182- The cost-benefit and distributional layers are what most distinguish a JPAM analysis from a field-83 economics paper — budget time for them, do not bolt them on at the end.84- An MVPF or benefit-cost ratio is only as credible as the estimate it rests on; report its sensitivity85 to the effect-size CI and to the discount rate, not a single point.86- A precisely estimated null can be a publishable JPAM result if it rules out a policy-relevant effect87 — frame it as "we can rule out effects larger than X," not "no effect."8889## Worked micro-example (illustrative)9091An evaluation finds a job-training program raises quarterly earnings by \$420 (95% CI \$120–\$720). The92JPAM analysis does not stop there: it converts this to a **benefit-cost ratio** (lifetime earnings gain93vs. per-participant cost) under a stated discount rate, runs **sensitivity** across the CI and discount94rate, shows the gain is **concentrated among longer-tenured entrants** (theory-driven heterogeneity), and95notes the program is **net-positive to the government budget** only above a take-up threshold. The policy96story is the package, not the \$420. (Numbers illustrative.)9798## Output format99100```101【Main estimate】effect in policy-relevant units (+ CI)102【Robustness】checks mapped to specific threats103【Heterogeneity】theory-driven subgroups + multiplicity handling104【Cost-benefit】perspective, discount rate, MVPF/BCR, sensitivity105【Distribution】who gains / who pays106【Next】jpam-tables-figures107```108109## Supplementary resources110111- [`../../resources/code/`](../../resources/code/) — estimation + robustness skeletons (Stata + Python)112- [`../../../shared-resources/empirical-methods/reporting-standards.md`](../../../shared-resources/empirical-methods/reporting-standards.md) — inference + reporting table stakes113- [`../../resources/external_tools.md`](../../resources/external_tools.md) — cost-benefit / MVPF tooling and policy data sources