Uai Reproducibility

UAI Reproducibility

UAI's 2026 CFP did not impose a formal reproducibility checklist (one may appear later — 待核实 each cycle), but it strongly encouraged code and data availability and instructed reviewers to judge whether claims are backed up convincingly. At this venue "convincing" has a technical meaning: stochastic-inference results carry diagnostics, not just point estimates. This skill turns that norm into an audit.

The UAI-specific bar

Reproducibility questions at a probabilistic-inference venue go one level deeper than "can I rerun the script":

• A sampler that reproduces the same posterior mean but different tail quantiles has not reproduced the paper — report and check the diagnostics that detect this (R-hat, ESS, divergent transitions where relevant).
• A variational result is reproducible when the ELBO trajectory and the selected restart are recoverable, not merely the final metric; multi-restart selection rules must be stated.
• Calibration claims reproduce only if the binning scheme, the split used for recalibration, and the α grid are all pinned down; empirical coverage moves with all three.
• Causal-discovery results depend on graph generation as much as on the algorithm: publish the SCM sampler, noise families, and intervention protocol.

Determinism ledger

Record every randomness source once, in code, and cite it from the paper:

# repro/ledger.py — imported by every experiment entry point
import json, platform, random, numpy as np

def fix_and_log(seed: int, path: str = "run_manifest.json"):
    random.seed(seed)
    np.random.seed(seed)
    manifest = {
        "seed": seed,
        "python": platform.python_version(),
        "numpy": np.__version__,
        "chains": 4, "warmup": 1000, "draws": 2000,   # sampler config lives here
        "elbo_restarts": 10, "restart_rule": "best final ELBO",
    }
    json.dump(manifest, open(path, "w"), indent=2)
    return manifest

The manifest style matters more than the specific fields: one machine-readable file per run, checked into the artifact, lets a reviewer reconcile the paper's Table 3 with an actual execution.

Disclosure map

Diagnostic quick reference

What "reported convincingly" tends to mean per inference family — as conventions of the field, not venue mandates:

If a row's "minimum" column is missing for your method family, expect the backing criterion to absorb the damage.

Honesty over completeness

• If code cannot be released (industrial constraints, licensed data), say so in the paper and compensate: fuller pseudocode, exact hyperparameters, synthetic surrogates for private datasets. The encouraged-not-mandatory wording gives room for honesty, not for vagueness.
• Report failure modes you observed — initializations that collapse, chains that need longer warmup on one dataset. Probabilistic-ML reviewers trust papers that know where their method breaks.
• Never let variance disappear in the retelling: if three of ten seeds underperform, the aggregate table must reflect it (mean ± sd over all ten, or a stated, principled selection rule).

Where reproducibility evidence lives

Split by tier deliberately: the availability statement and diagnostic summaries in the reviewed PDF (body or appendix), where they count toward backing; manifests, loaders, and per-run logs in the ZIP, where they support spot-checks. Never leave the only mention of seeds or repeat counts inside the optional archive — reviewers grade what the PDF says.

Pre-submission reproducibility drill

1. Clone your own artifact onto a machine that never ran the project; follow only the README.
2. Reproduce the smallest headline number end to end, including the diagnostic that validates it.
3. Diff the regenerated figure against the paper's; investigate any visible deviation before a reviewer does.
4. Grep the paper for every "we observe/we find/consistently" and confirm each maps to a logged run.
5. Write the availability statement last, describing what is actually in the ZIP — not what was planned in January.

Output format

[Repro grade] turnkey / recoverable with effort / under-specified
[Diagnostics reported] <R-hat/ESS/ELBO/coverage/SHD... as applicable>
[Determinism ledger] present / partial / absent
[Disclosure gaps] <items from the map still missing>
[Availability statement] drafted / needs honesty pass / missing

Source: brycewang-stanford/Awesome-Journal-Skills → UAI-Skills/skills/uai-reproducibility/SKILL.md