Colm Supplementary

COLM Supplementary Material

The 2026 format gives you a strict 9-page main text, unlimited citation pages, and appendix space whose guaranteed audience is nobody — reviewers may read it, the rebuttal may point into it, but no rule obliges anyone to open it. That asymmetry is the whole design principle: the main text carries every load-bearing element, and the supplement carries the audit trail that makes the main text checkable.

The placement rule

Ask of each artifact: is this needed to believe the claim, or to re-run it? Believe → main text. Re-run → appendix/supplement. Both → summarize in the main text, deposit fully in the appendix, and cross-reference in both directions.

LM-specific supplement content reviewers actually use

Three appendix sections repeatedly decide borderline LM papers:

1. The prompt appendix. Exact strings, exemplar counts, formatting whitespace included. Prompt paraphrase is a known variance source, so approximation here undermines every number.
2. The "what we tried" ledger. Prompt variants and hyperparameters attempted per system — the fairness evidence for baseline comparisons.
3. The contamination appendix. Overlap statistics per evaluation set, or the documented reason none could be computed.

Anonymization of the code/data package

COLM's instructions ban identity-revealing links, and supplements are the usual leak. Before upload, scrub: repository remotes and commit author fields (git log travels inside .git/), Hugging Face org names in from_pretrained(...) calls, W&B entity names in configs, API org IDs in cached headers, absolute paths (/home/<username>/...), and internal cluster hostnames.

# Build the review package from a clean export — never zip a working directory
git archive --format=tar HEAD | (mkdir -p /tmp/colm-pkg && tar -x -C /tmp/colm-pkg)
grep -rniE 'wandb|hf_[A-Za-z0-9]{20,}|api[_-]?key|/home/[a-z]|<lab-name>' /tmp/colm-pkg | head
grep -rlE 'from_pretrained\("[^"]*/' /tmp/colm-pkg --include='*.py' | head   # org-scoped model IDs
(cd /tmp && zip -rq colm-supplement.zip colm-pkg -x '*.git*')

git archive (not cp -r) is the load-bearing choice: it cannot smuggle .git/, untracked scratch files, or environment files into the package.

Referencing the supplement from the main text

• Point at specific anchors ("App. C.2, Table 9"), never "see appendix".
• In the rebuttal window (May 22 - June 8 in 2026), precise anchors let you answer a reviewer in one sentence plus a pointer — the highest-leverage use of everything you deposited.
• Do not promise supplement content that is not there; reviewers who follow a dangling pointer stop trusting the intact ones.

Ordering the appendix for the reader you hope exists

Appendices get opened with a question in hand, not read linearly. Order sections by the probability of the question:

1. A. Experimental details — models, revisions, decoding configs, harness commit: the reproduction question.
2. B. Prompts — verbatim, grouped by experiment: the "how did you ask" question.
3. C. Contamination analysis — the trust question.
4. D. Full results — per-task, per-model grids: the "does it hold on X" question.
5. E. Baseline tuning ledger — the fairness question.
6. F. Human/judge evaluation protocol — instructions, agreement, biases.
7. G. Failure analysis — curated transcripts.
8. H. Compute — the cost question.

Within each, lead with a two-line summary of what the section shows — the reviewer deciding whether to trust an unread appendix trusts one with visible structure. And keep appendix numbering frozen after submission: the rebuttal will cite "App. C.2" and a renumbered revision breaks every pointer in the discussion thread.

What not to include

• Anything load-bearing that appears only in the supplement — if the 9 pages cannot make the case alone, restructure (colm-writing-style).
• Un-curated dumps (every transcript, every checkpoint) — volume without curation reads as evasion.
• Data whose license or ToS forbids redistribution — describe it and state the restriction instead (colm-artifact-evaluation).
• 待核实: no supplementary size cap was verifiable for 2026; check the OpenReview form's actual limits rather than assuming.

Timing note for the 2026-style calendar

In 2026 the supplement traveled with the paper at the March 31 deadline — no separate later upload was posted (待核实 whether future cycles add one). Plan accordingly: the anonymization sweep and package build belong in deadline week's T-3 slot (colm-submission), not in an imagined grace period. And because the May 22 - June 8 rebuttal leans on appendix anchors, whatever you deposit in March is the ammunition you will fight with in June — deposit for your future self, who will be tired and have eighteen days.

Output format

[Placement audit] load-bearing content in supplement? none / move up: <items>
[Prompt appendix] verbatim + system prompts ▢   [Tried-ledger] ▢   [Contamination appendix] ▢
[Anonymity scan] clean / hits: <files>
[Pointer integrity] all anchors resolve / dangling: <refs>
[Package] built via clean export ▢  size: <n> MB (cap 待核实)

Source: brycewang-stanford/Awesome-Journal-Skills → COLM-Skills/skills/colm-supplementary/SKILL.md