Arecon Evidence Standards

Evidence Appraisal & Balance (arecon-evidence-standards)

When to trigger

• The framework is set and you are filling cells with conflicting empirical results
• You must decide whether "the literature finds X" is actually supported, or only loosely
• The field has rival schools, a live controversy, or estimates that disagree
• You are a contributor to this literature and worry the review tilts toward your own work

Appraising evidence you did not produce

An ARE review runs no identification of its own — there is no design to defend, no replication package of your data. Instead you act as the field's referee-of-record for adjacent readers: you judge how much weight each primary study can bear so the review weighs the evidence correctly. Make the appraisal explicit, not implicit:

• DID / event study: does the cited paper use TWFE on staggered timing without addressing heterogeneity bias? A study that predates Callaway–Sant'Anna / Sun–Abraham corrections may not support the magnitude you attribute to it — flag it.
• IV: is the first stage strong and the exclusion restriction defended, or is it a convenience instrument? Weak-IV results carry less weight in your synthesis.
• RDD: density/manipulation checks, bandwidth robustness — a fragile RDD is a fragile data point.
• Structural / calibration: are the parameters identified and the counterfactual policy-invariant, or is it calibration presented as estimation?
• Experiments: pre-registration, balance, attrition, multiple-hypothesis adjustment, external validity.

You are not re-running these — you are rating their credibility in the evidence matrix so the review's conclusions track the best evidence, not the loudest paper.

Execution bridge — when rating is not enough

A few moments in an ARE review justify running something instead of only rating it: a survey table pools estimates and wants a formal meta-analytic average or publication-bias check; a pivotal magnitude predates the staggered-DiD corrections and its replication data are public, so you can report what callaway_santanna or bacon_decomposition actually does to it rather than speculate; or a weak-instrument worry could be settled by an effective_f_test on the archived first stage. For these, hand off to execution-with-mcp, which maps each design and reviewer objection to the callable StatsPAI / Stata MCP chain (detect_design → fit with as_handle=true → audit_result). Label any such figure as your re-analysis, and never report a number you did not compute.

Weighing, not vote-counting

Conflicting results are reconciled by credibility and by what each study estimates, never by tallying "7 studies positive, 4 negative." Two estimates that disagree often measure different objects (different populations, estimands, time horizons); say so, and let the framework's cells carry the distinction. A pooled "consensus" across non-comparable designs manufactures false agreement that ARE's methodologically literate readers will catch.

Comprehensiveness vs. selectivity: the ARE contract

A review must be comprehensive in coverage yet selective in emphasis — and stay accessible in ~25–40 pages. Tier the corpus:

Comprehensiveness is proven by the citation set + saturation log (arecon-literature-synthesis); selectivity is exercised in the prose.

Fairness and the self-citation trap

ARE referees are frequently the surveyed authors themselves, so balance is strategic as well as ethical:

• Steelman every camp. State each school's strongest case in terms its proponents would accept before noting weaknesses.
• Attribute ideas to originators, not popularizers (a recurring referee complaint).
• Handle live controversies without resolving by fiat. Lay out the disagreement, what evidence would settle it, and where your own read sits — labelled as your read, not as consensus.
• Audit self-citation. Your own work appears at the tier its importance to the field warrants — no more; rivals get their strongest statement; a reader who does not know the author cannot tell from the emphasis.

Checklist

• Each pivotal primary study carries a credibility appraisal (design, identification, robustness)
• Outdated or fragile designs flagged where the review leans on their magnitudes
• Conflicting findings reconciled by credibility + estimand, not vote-counting
• Corpus tiered; prose emphasis matches tier; coverage provable from the saturation log
• Every rival school stated at its strongest before critique (steelman)
• Idea attribution traces to originators
• Live controversies presented with what evidence would settle them; author's read labelled
• Self-citation audited: own work at warranted tier; emphasis is identity-blind

Anti-patterns

• Citing a study's headline number without noting its identification is now known to be biased
• Vote-counting conflicting results instead of weighing credibility and estimand
• Pooling non-comparable estimates into one "the literature shows…" magnitude
• Comprehensiveness theatre: equal-length summaries of every paper (no editorial judgment)
• Strawmanning the camp the author disagrees with
• A review that doubles as the author's CV (the most-punished ARE balance failure)
• Declaring a live controversy "resolved" by assertion rather than the evidentiary state

Output format

【Credibility appraisal】pivotal studies rated (design/identification/robustness)? Y/N
【Conflict handling】reconciled by credibility + estimand (not vote-count)? Y/N
【Tiering】corpus split foundational/important/confirmatory/tangential? Y/N
【Comprehensiveness】saturation log supports "nothing important missing"? Y/N
【Steelman】each rival school stated at its strongest? Y/N
【Controversy】evidence-to-settle stated; author's read labelled? Y/N
【Self-citation audit】own work at warranted tier; emphasis identity-blind? Y/N
【Next step】→ arecon-tables-figures (who-found-what tables) → arecon-writing-style

Source: brycewang-stanford/Awesome-Journal-Skills → Annual-Review-of-Economics-Skills/skills/arecon-evidence-standards/SKILL.md