brycewang-stanford ICSE Supplementary Material
ICSE's page model is stricter than most authors coming from ML venues expect, and it inverts their packing instincts. The 2027 research-track format (read 2026-07-08): main text ≤ 10 pages inclusive of all figures, tables, and appendices, plus up to 2 pages of references only. There is no unlimited in-PDF appendix. Everything else travels as supplementary material — an anonymized link in the paper or an upload via HotCRP — and must be assumed optional reading.
The placement rule
One question decides placement: must a reviewer see it to score the paper?
| Content | Placement | Why |
|---|---|---|
| Anything a criterion score depends on: RQ answers, key tables, protocol essentials, the threat analysis | The 10 pages | Optional material cannot carry mandatory weight |
| Full prompt templates, extended proofs, extra result tables, per-subject breakdowns | Package, summarized in-body | Body states the finding + pointer; package holds the mass |
| Tool source, benchmark, scripts, raw outputs | Replication package | This is the verifiability substrate (icse-reproducibility) |
| Interview guides, codebooks, instruments | Package, named in the availability statement | Reviewers audit method through them when they choose to |
| Demo video, screenshots gallery | Package, only if genuinely clarifying | Never as a substitute for a precise in-body description |
The failure mode is load-bearing supplement: "due to space, the evaluation methodology is described in the supplementary material" invites the reviewer to score what the body shows — an unevidenced claim. Compress into the body instead: a protocol table instead of two prose pages, a summary row instead of the full matrix.
Summarize-and-point discipline
Every relocation leaves a residue in the body — the conclusion plus a precise pointer:
Patches for all 214 bugs, with per-bug generation logs, are in the package (
results/patches/); Table 5 reports the aggregate. The full 41-item codebook is incodebook.pdf; Table 2 shows the 6 categories with ≥10% prevalence.
Precise pointers (directory, file, section of the package README) also pay forward: they become the artifact-evaluation roadmap after acceptance.
Package layout reviewers can navigate in five minutes
replication-package/
├── README.md # map: claim/table/figure -> command -> output
├── REQUIREMENTS.md # env: versions, container digest, hardware, runtime
├── LICENSE # open license even at review time (anonymized holder)
├── code/ # tool source, pinned dependencies
├── benchmark/ # subjects or fetch-scripts with SHAs + mining date
├── scripts/ # one entry point per paper table/figure:
│ ├── run_rq1.sh # names match the RQs, not internal jargon
│ └── make_table3.py
├── results/ # raw outputs the paper's numbers derive from
└── appendix.pdf # extended tables/proofs that did not fit the body
The README's first section is a claims map: each headline claim, the command that regenerates its evidence, expected runtime, and expected output. A reviewer who checks one row and succeeds extends trust to the rest; one who hits a broken path on the first try generalizes that, too.
Anonymity of the archive
The supplement must hold the same double-anonymous line as the paper:
git archive exports (no history), scrubbed notebook metadata, no
/home/username/ paths in configs or logs, no institutional hostnames, no
author names in LICENSE/setup.py/CITATION files, hosting through an
anonymizing service rather than a personal account. Scrub, then re-run the
entry-point scripts — anonymization that breaks paths converts your evidence
into a liability.
What must stay out of the package
Screening for exclusions is as important as packing inclusions: third-party code or datasets whose licenses forbid redistribution (link with pinned fetch-scripts instead); human-subjects raw data beyond what consent covers — transcripts, recordings, un-aggregated survey rows; credentials, API keys, and internal hostnames hiding in config files and CI logs; and proprietary industrial code shown to you under NDA, even in fragments quoted in comments. A replication package is a publication: it gets the same legal and ethical review as the paper, and a leak in the supplement is harder to retract than a sentence.
Size and dependency realism
No supplement size cap was verifiable for 2027 (待核实 on the live HotCRP form), so engineer for reviewer patience instead: keep the download small (fetch-scripts for large corpora, with checksums), keep the smoke test under minutes on a laptop, and never require credentials, cluster schedulers, or paid API keys for at least a reduced-scale verification path. Provide cached outputs beside every expensive step so the analysis chain can be verified without the compute.
Reverify each cycle
The 10+2 model, the HotCRP supplementary option, and the availability- statement placement are 2027-cycle facts; page models have changed across ICSE editions before. Confirm the current call before deciding any placement that would be expensive to reverse in deadline week.
Output format
[Body/package split] each relocated item -> residue sentence + pointer present?
[Load-bearing check] any criterion-relevant content living only in the package? (must be none)
[Package audit] layout roles present; claims map covers every headline number
[Anonymity + runnability] scrub done; entry points re-run post-scrub
[Reviewer path] smoke test minutes, laptop-feasible, credential-free