Policyengine Calibration Diagnostics

PolicyEngine calibration diagnostics

Converts tribal "I'd check the takeup rate first" knowledge into a structured sensitivity registry, and pairs it with the live per-target calibration API so hypotheses are ranked against real relative_error numbers rather than assumptions. When an /analyze-policy comparison returns INVESTIGATE, this skill supplies the ranked candidate causes.

The calibrated microdata is now a Microcosm build (see the policyengine-data skill for how targets, weights, and L0 sparsity work). Calibration targets live in the Microcosm build's target set — not in a hand-maintained loss file — and their fit is queryable per release from the dashboard API below.

When to use

• Stage 5.6 / Stage 6 of /analyze-policy — invoked by the calibration-diagnostics agent.
• Code review of microsim PRs where the headline number differs from priors.
• Designing or auditing a Microcosm calibration target.
• Debugging why a state-level run looks volatile.

Top-level architecture

PolicyEngine microsim results depend on three layers:

1. Country model logic (policyengine-us, policyengine-uk, policyengine-canada) — formulas, parameters.
2. Calibrated microdata (Microcosm) — survey weights + imputations matched to administrative targets.
3. Behavioral assumptions (takeup rates, labor-supply elasticities) — usually parameters but easy to overlook.

A magnitude mismatch is almost always rooted in layer 2 or 3, not layer 1 (layer-1 mismatches show up as outright simulation errors, not magnitude drift).

Live calibration API (check this first)

Per-target fit for the current Microcosm release, no auth, reads the release from Hugging Face:

BASE = https://calibration-diagnostics.vercel.app/calibration/dashboard/api/populace
GET {BASE}/target-diagnostics?source=<source>     # every target for a source, with relative_error
GET {BASE}/target-investigation?target=<id>        # full investigation packet for one target
GET {BASE}/releases                                # release ids for pinning

Sources map to reform domains: Social Security → ssa; Medicaid/ACA/Medicare → cms_medicaid / cms_aca / cms_medicare; TANF → hhs_acf_tanf; income tax / credits → irs_soi, jct, cbo; state income tax → state_income_*. A target already >10% off in the release diagnostics is a stronger hypothesis than any prior. (Dashboard UI: calibration-diagnostics.vercel.app.)

The three-ring reading method (align with /analyze-policy Stage 5.6)

Checking only the reform's primary marginal is the classic miss — a reform's cost usually depends on the joint distribution of its variable with other income, and the primary marginal can calibrate perfectly while the interaction is off. Check three rings outward:

1. Ring 1 — the primary variable's marginals. Map the reform's domain to a calibration source and fetch its targets.
2. Ring 2 — the mechanism's other inputs. Every variable that enters the formula the reform changes. SS-benefit taxation depends on combined income, so dividends, taxable interest, and pensions (irs_soi) are load-bearing; a CTC phase-in depends on earnings; a SNAP change on rents and deductions.
3. Ring 3 — the interacted quantity itself (best single check). Search for the downstream quantity the reform directly reprices — taxable_social_security_amount for SS-taxation, eitc_amount for EITC, itemizer counts for deduction caps. When such a target exists it validates the joint distribution end-to-end through the actual formula, and its relative_error bounds the data-side bias of the score. Weight it above every marginal. When no ring-3 target exists, say so — "marginals within tolerance; joint distribution untargeted" is materially weaker evidence.

Worked example (HR 904, SS-taxation reform): ssa total benefits calibrate to −0.2% (ring 1 fine) while irs_soi ... taxable_social_security_amount runs −10.9% (ring 3) — the score's operative base is ~11% low in the data even though the headline marginal is nearly exact.

Interpretation is informative, not a gate: a poorly-calibrated but reform-relevant target does not flip the comparator verdict by itself, but it belongs in the report's uncertainty discussion. Escalate to INVESTIGATE when a directly-load-bearing target (the primary variable OR the ring-3 interacted quantity) is off by |relative_error| > 25%, even if headline numbers match anchors.

Sensitivity table — top programs

The rows below are the paradigm-independent knowledge: which calibration input moves which result, and in which direction. They hold regardless of which build produced the weights — use them to decide which targets to pull from the API above.

EITC

EITC coverage note: thinner than the CTC and SALT rows. The diagnostics agent should emit coverage_note: "EITC row is partial; hypotheses cover takeup and age-cohort but NOT joint-filer marriage-penalty mechanics, investment-income-limit interactions, or self-employment-income imputation. If your deviation signature touches any of these, widen the confidence band."

CTC

SALT cap

State income tax (any state)

Refundable credits broadly

Deviation signature playbook

Sources

• calibration-diagnostics.vercel.app — live per-target diagnostics API and dashboard (first stop; reads the current Microcosm release). Source: github.com/PolicyEngine/calibration-diagnostics.
• PolicyEngine/microcosm — where calibration targets, weights, and the L0 sparsity live; the build that produces the certified dataset. See the policyengine-data skill.
• github.com/PolicyEngine/policyengine-us/issues — known program-level model issues.

Related skills

• policyengine-data — how Microcosm builds targets, weights, and the sparse-vs-dense tradeoff.
• policyengine-prior-scores — what a run is being compared against.
• policyengine — running the perturbation tests this skill recommends.