# Colm Supplementary > Use when deciding what goes into a COLM paper's appendices and supplementary material versus the strict 9-page main text — verbatim prompts, full evaluation configurations, per-task result tables, contamination analyses, human-evaluation protocols, and anonymized code/data packages that survive double-blind review. - Skill: `thedixitjain/colm-supplementary` (Agent Skill) - Install (CLI): `npx skillmds add thedixitjain/colm-supplementary` - Raw SKILL.md: https://api.skillmd.com/api/skills/thedixitjain/colm-supplementary/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: thedixitjain (https://skillmd.com/u/thedixitjain) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/thedixitjain/colm-supplementary --- # COLM Supplementary Material The 2026 format gives you a strict 9-page main text, unlimited citation pages, and appendix space whose guaranteed audience is *nobody* — reviewers may read it, the rebuttal may point into it, but no rule obliges anyone to open it. That asymmetry is the whole design principle: the main text carries every load-bearing element, and the supplement carries the *audit trail* that makes the main text checkable. ## The placement rule Ask of each artifact: *is this needed to believe the claim, or to re-run it?* Believe → main text. Re-run → appendix/supplement. Both → summarize in the main text, deposit fully in the appendix, and cross-reference in both directions. | Artifact | Placement | Note | |---|---|---| | Representative prompt (or skeleton) | Main text | Readers cannot judge an LM evaluation without seeing how the model was asked | | Full prompt set, verbatim, incl. system prompts | Appendix | Paraphrased prompts are not reproducible; include exact strings | | Decoding configs per experiment | Appendix table | One row per reported table cell group (`colm-experiments`) | | Contamination analysis method + results | Summary in main text; full in appendix | The *conclusion* is load-bearing; the n-gram machinery is audit trail | | Per-task / per-model full result grids | Appendix | Main text shows the aggregate + the interesting slice | | Human-eval protocol: instructions, pay, demographics, agreement | Appendix | Screenshots of annotator UI earn trust | | Failure cases and transcripts | Appendix, curated | A dozen *representative* transcripts beat 400 raw ones | | Code + data package | Supplementary upload / anonymized repo | Must be anonymous at submission (see below) | | Raw API responses archive | Release artifact (acceptance) | Often too large for review upload; describe it and commit to release | ## LM-specific supplement content reviewers actually use Three appendix sections repeatedly decide borderline LM papers: 1. **The prompt appendix.** Exact strings, exemplar counts, formatting whitespace included. Prompt paraphrase is a known variance source, so approximation here undermines every number. 2. **The "what we tried" ledger.** Prompt variants and hyperparameters attempted per system — the fairness evidence for baseline comparisons. 3. **The contamination appendix.** Overlap statistics per evaluation set, or the documented reason none could be computed. ## Anonymization of the code/data package COLM's instructions ban identity-revealing links, and supplements are the usual leak. Before upload, scrub: repository remotes and commit author fields (`git log` travels inside `.git/`), Hugging Face org names in `from_pretrained(...)` calls, W&B entity names in configs, API org IDs in cached headers, absolute paths (`/home//...`), and internal cluster hostnames. ```bash # Build the review package from a clean export — never zip a working directory git archive --format=tar HEAD | (mkdir -p /tmp/colm-pkg && tar -x -C /tmp/colm-pkg) grep -rniE 'wandb|hf_[A-Za-z0-9]{20,}|api[_-]?key|/home/[a-z]|' /tmp/colm-pkg | head grep -rlE 'from_pretrained\("[^"]*/' /tmp/colm-pkg --include='*.py' | head # org-scoped model IDs (cd /tmp && zip -rq colm-supplement.zip colm-pkg -x '*.git*') ``` `git archive` (not `cp -r`) is the load-bearing choice: it cannot smuggle `.git/`, untracked scratch files, or environment files into the package. ## Referencing the supplement from the main text - Point at specific anchors ("App. C.2, Table 9"), never "see appendix". - In the rebuttal window (May 22 - June 8 in 2026), precise anchors let you answer a reviewer in one sentence plus a pointer — the highest-leverage use of everything you deposited. - Do not promise supplement content that is not there; reviewers who follow a dangling pointer stop trusting the intact ones. ## Ordering the appendix for the reader you hope exists Appendices get opened with a question in hand, not read linearly. Order sections by the probability of the question: 1. **A. Experimental details** — models, revisions, decoding configs, harness commit: the reproduction question. 2. **B. Prompts** — verbatim, grouped by experiment: the "how did you ask" question. 3. **C. Contamination analysis** — the trust question. 4. **D. Full results** — per-task, per-model grids: the "does it hold on X" question. 5. **E. Baseline tuning ledger** — the fairness question. 6. **F. Human/judge evaluation protocol** — instructions, agreement, biases. 7. **G. Failure analysis** — curated transcripts. 8. **H. Compute** — the cost question. Within each, lead with a two-line summary of what the section shows — the reviewer deciding whether to trust an unread appendix trusts one with visible structure. And keep appendix numbering frozen after submission: the rebuttal will cite "App. C.2" and a renumbered revision breaks every pointer in the discussion thread. ## What not to include - Anything load-bearing that appears *only* in the supplement — if the 9 pages cannot make the case alone, restructure (`colm-writing-style`). - Un-curated dumps (every transcript, every checkpoint) — volume without curation reads as evasion. - Data whose license or ToS forbids redistribution — describe it and state the restriction instead (`colm-artifact-evaluation`). - 待核实: no supplementary size cap was verifiable for 2026; check the OpenReview form's actual limits rather than assuming. ## Timing note for the 2026-style calendar In 2026 the supplement traveled with the paper at the March 31 deadline — no separate later upload was posted (待核实 whether future cycles add one). Plan accordingly: the anonymization sweep and package build belong in deadline week's T-3 slot (`colm-submission`), not in an imagined grace period. And because the May 22 - June 8 rebuttal leans on appendix anchors, whatever you deposit in March is the ammunition you will fight with in June — deposit for your future self, who will be tired and have eighteen days. ## Output format ```text [Placement audit] load-bearing content in supplement? none / move up: [Prompt appendix] verbatim + system prompts ▢ [Tried-ledger] ▢ [Contamination appendix] ▢ [Anonymity scan] clean / hits: [Pointer integrity] all anchors resolve / dangling: [Package] built via clean export ▢ size: MB (cap 待核实) ``` --- **Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `COLM-Skills/skills/colm-supplementary/SKILL.md`