Colm Review Process

COLM Review Process

COLM runs a compact, single-cycle-per-year pipeline on OpenReview, and — because the venue is three editions old — its reviewing culture is still visibly under construction. Understanding both the posted mechanics and the young-venue dynamics is what lets authors time their effort correctly.

The 2026 pipeline (verified 2026-07-08)

On this pack's access date — July 8, 2026 — the pipeline is at its decision moment: rebuttals are closed and notifications land today. If you are triaging an outcome, go directly to colm-camera-ready (accept) or the resubmission logic below (reject).

Who reviews you, and why it feels different

Three structural facts shape COLM reviews:

• A self-selected LM reviewer pool. Everyone opted into a language-modeling venue, so you will rarely justify why LMs matter — but you will face specialists on contamination, harness quirks, and prompt sensitivity (colm-experiments lists exactly what they check).
• Reciprocal reviewing feeds the pool. COLM has per-submission and per-reviewer requirements, and any author with 4+ submissions is drafted automatically. Your reviewers are substantially this year's authors — practically minded, current, and reading many similar papers in a short window.
• Young-venue variance. With only two prior editions of precedent, calibration across reviewers is wider than at older venues: score meanings, appendix-reading habits, and rebuttal responsiveness vary. The rational response is to make the paper robust to the strictest plausible reviewer rather than tuning to an imagined average.

Rules that govern the conversation

• Double-blind, both directions — reviewers and ACs cannot see author names; authors cannot see reviewer names. Anything that de-anonymizes you mid-process is a self-inflicted wound.
• Code of Ethics scope — it explicitly covers reviewing and paper discussion, and reviewers are encouraged to raise potential violations they see in papers.
• LLM use in reviewing must be disclosed, whatever its size — the 2026 policy draws the reviewer line stricter than the author line. As an author, this cuts two ways: a review may lawfully have LLM assistance behind it, and a review that looks LLM-generated without engagement is a legitimate thing to flag to the AC in the rebuttal, factually and without accusation theater.

Reading a COLM review packet

Sort reviewer claims into four bins before writing anything:

bin A  misreading           -> point to the page/table; cheapest wins first
bin B  evidence gap         -> runnable inside the 18-day window? do it
                               (cache-based analyses are fastest: colm-artifact-evaluation)
bin C  scope disagreement   -> reviewer wants a different paper; answer once,
                               crisply, and let the AC arbitrate
bin D  correct hit          -> concede, state the fix, thank them briefly

The AC synthesizing the discussion is your real audience (colm-author-response handles execution). Between June 8 and July 8 the process belongs to reviewers and ACs; there is no author-visible discussion phase to work.

What the AC actually reads

Area chairs at a compact venue synthesize under time pressure, and their reading order shapes what survives:

1. The reviews' summary and score fields — first impressions form here.
2. The rebuttal's opening lines per reviewer — which is why conclusion-first responses matter (colm-author-response).
3. Whether reviewers reacted to the rebuttal — an unanswered strong response still counts, but an acknowledged one counts double; polite nudge-questions to silent reviewers are legitimate inside the window.
4. The paper itself, selectively, where reviewers disagree.

Score arithmetic is a poor predictor at a young venue: with wide reviewer calibration variance and no published acceptance rate (待核实), a 5-5-8 with an engaged champion routinely beats a flat 6-6-6. Optimize for giving one reviewer the material to argue your case in the discussion you cannot see, rather than for moving every score by one point.

If the decision is a reject

• Mine the reviews for the bin-D items; they transfer to any venue.
• The calendar is forgiving: a July 8 COLM decision precedes the autumn ICLR deadline and the following spring's NeurIPS cycle, and COLM 2027's expected late-March deadline (待核实) is a natural 8-month runway for a genuine revision.
• Re-run colm-topic-selection honestly — a common COLM reject pattern is a paper whose contribution was a task result wearing LM-analysis framing.

Volatile every edition

• All dates above; the phase structure itself (COLM has already adjusted process details in each of its three editions).
• Reviewer-form contents, score scales, and any acceptance-rate figures (none officially published; 待核实).
• Reciprocal-reviewing thresholds and enforcement.

Output format

[Phase today] <where the current cycle actually is — verify against colmweb.org/dates.html>
[Packet triage] A: <n> misreadings / B: <n> evidence gaps / C: <n> scope fights / D: <n> hits
[AC-facing issue] <the one point the meta-review will turn on>
[Deadline math] <days left in the relevant window>
[Plan] <respond / experiment / concede / escalate-to-AC items, in order>

Source: brycewang-stanford/Awesome-Journal-Skills → COLM-Skills/skills/colm-review-process/SKILL.md