brycewang-stanford CVPR Reproducibility
At CVPR, reproducibility failures rarely look like fraud; they look like a table nobody can match because one augmentation flag, one crop size, or one pretraining corpus went unstated. This skill hardens the paper against that fate, anchored in the 2026-cycle machinery (checked 2026-07-08): the Compute Reporting Form, the anonymous supplement, and reviewers trained on a decade of un-reproducible state-of-the-art claims.
The CRF as a reproducibility floor
The 2026 cycle attached a Compute Reporting Form to every submission — Section 1 (hardware specification) and Section 5 (verification) mandatory, deeper sections optional, with an explicit opt-out route for proprietary constraints. Treat the mandatory floor as the start, not the ceiling:
| CRF layer | What it pins down | Why reviewers care |
|---|---|---|
| Hardware (mandatory) | GPU model, count, primary configuration | Grounds every "real-time" and "efficient" claim |
| Verification (mandatory) | Author attestation | Somebody owns the numbers |
| Task/compute (optional) | GPU-hours or FLOPs per result | Separates a 4-GPU method from a 512-GPU method |
| Full logs (optional) | Run-level records | The strongest possible "we actually ran this" |
If your contribution is efficiency, filling only the mandatory sections undercuts your own claim — report the compute and let the numbers argue.
The recipe ledger
Vision results are recipe-sensitive. Maintain one machine-readable ledger from the first experiment, and generate the paper's implementation-details paragraph from it instead of reconstructing details in deadline week:
# recipe-ledger.yaml — one block per reported table row
table3_row2:
backbone: vit-b16, pretrain: <corpus + checkpoint hash>
data: <dataset version + split file sha256>
aug: [rrc-224, hflip, randaug-m9]
optim: adamw, lr: 1.0e-4, sched: cosine, epochs: 90, batch: 1024
seed: 3407 # and whether cudnn deterministic was set
hardware: 8xA100-80G # must agree with CRF Section 1
command: scripts/train.sh configs/table3_row2.yaml
The ledger's second job is internal: when a reviewer asks in January which schedule produced Figure 5, you answer from the file in minutes.
Benchmark hygiene that survives scrutiny
- Split discipline. Name the exact split files; never tune on anything downstream of the test set — including "just checking" runs that leak into architecture choices.
- Test-set frequency. State how many times the test set was evaluated. Val-set ablations + one final test run is the defensible pattern.
- Metric provenance. Cite the metric implementation (which mAP variant? whose FID code, which feature extractor?). Same-name metrics differ across repos by whole points.
- Pretraining disclosure. "Initialized from X" changes the comparison class; a method beating from-scratch baselines while riding a web-scale pretrain is a different claim.
- Baseline re-runs. Say which baseline numbers are quoted from papers versus re-run under your protocol, and match backbones and schedules when you re-run.
Variance at vision scale
Full multi-seed grids are often unaffordable at modern training budgets, and reviewers know it. The credible middle ground: multi-seed the cheap decisive experiments (small backbone, headline ablation) and report mean ± std; run the flagship once but state so explicitly; never present a 0.2-point gain as a finding when the same table shows seed-level noise of 0.4. If evaluation itself is stochastic (generation, sampling-based detection), repeat evaluation, not just training.
Determinism knobs, documented once
Bit-exact reproduction is often impossible on GPU stacks, but stating your determinism posture is always possible and costs three lines in the supplement:
- Which seeds were fixed (Python/NumPy/framework/dataloader workers) and whether deterministic kernels were enabled — note the speed cost if they were not.
- Sources of accepted nondeterminism: atomics in scatter ops, multi-GPU reduction order, augmentation pipelines keyed to worker scheduling.
- The reproduction tolerance you actually observed: "re-running row 2 varies by ±0.15 mAP across identical-seed runs on different node counts" tells a reproducer whether their 0.1-point discrepancy is a bug or physics.
Teams that measure this once, early, stop having the "is 78.4 vs 78.6 a failure to reproduce?" argument — with reviewers and with themselves.
Statement-level honesty
Reproducibility text is a claims surface. "Code will be released" is a promise the community tracks; "results reproducible from the supplement" is checkable in January. Write the availability paragraph to match what is genuinely packaged: what ships in the supplement now, what is released at camera-ready (datasets claimed as contributions must be public by then — verified 2026 policy), and what cannot be released and why.
Held-out evaluation servers
Several vision benchmarks score on withheld test sets via submission servers with rate limits. This changes reproducibility mechanics: your reported test number is a server receipt, not a rerunnable command. Record the submission ID and date in the recipe ledger, respect per-week submission caps as an ethics matter (burning entries to tune on test is the community's canonical sin), and give reproducers the exact validation-set protocol that predicts the server number.
Reverify each cycle
- CRF structure, which sections are mandatory, and award tie-ins.
- Any reproducibility checklist added to the OpenReview form.
- Supplement caps that constrain how much recipe/code detail ships at review time (2026 caps 待核实).
Output format
[Repro grade] recipe-complete / gaps found
[CRF] hardware row consistent with paper claims: yes/no; optional sections: <filled?>
[Ledger] rows covering all reported tables: <n/m>
[Benchmark hygiene] splits · test-set count · metric provenance · pretrain disclosure
[Variance] multi-seeded: <experiments>; single-run flagged: <experiments>
[Fix list] <ordered, highest reviewer-visibility first>