COLT Experiments
Start from the venue truth: COLT imposes no experiments requirement, and most COLT
papers contain no experiments. The 2026 CFP (checked 2026-07-08) asks for
theoretical machine-learning contributions and mentions no empirical-evaluation
expectation; papers are accepted on theorems. The decision this skill supports is
therefore whether to include numerics, and only then how.
Should this paper contain numerics at all?
| Situation |
Include numerics? |
Rationale |
| Clean upper/lower bound pair, standard model |
No |
Plots add length, not belief |
| New algorithm whose practicality is part of the pitch |
Small illustration |
Shows the constants are not absurd |
| Theory explaining an empirical phenomenon (in-scope per the CFP's inclusive view) |
Yes, essential |
The phenomenon must be exhibited, then explained |
| Conjectured tightness you cannot prove |
Careful, labeled |
A scaling plot can support a conjecture — never upgrade it |
| Phase transition / separation between models |
Often worthwhile |
A picture of the transition is the most readable evidence |
| Purely structural result (equivalences, characterizations) |
No |
Nothing to simulate |
If the answer is no, spend the pages on proof overviews instead; a decorative
benchmark table in a COLT submission signals venue confusion and can lower reviewer
confidence.
Design rules when numerics earn their place
- Simulate the theorem's exact regime: the assumptions, the adversary, the parameter
ranges. An illustration outside the proved regime must be labeled exploratory.
- Prefer synthetic constructions where ground truth is computable; COLT illustrations
are about visualizing mathematics, not about datasets.
- Overlay the proved bound on the empirical curve so the reader sees slope agreement,
constant gap, and where finite-size effects fade.
- Report the estimator of variability (bars = standard error over R runs, stated in
the caption), fixed seeds, and replication counts — theorists distrust single
trajectories on principle.
- Keep total numerical content to roughly a figure or two; the appendix holds the
procedure description (no code-upload channel existed in the 2026 cycle).
A rate-illustration recipe
The canonical COLT figure is a log-log rate check — empirical error or regret against
the driving parameter, with the theoretical slope for reference:
import numpy as np
rng = np.random.default_rng(seed=2026) # fixed, reported seed
ns, R = np.logspace(2, 5, 8).astype(int), 50 # sample sizes, replications
emp = np.array([[run_once(n, rng) for _ in range(R)] for n in ns])
mean, se = emp.mean(axis=1), emp.std(axis=1, ddof=1) / np.sqrt(R)
slope = np.polyfit(np.log(ns), np.log(mean), 1)[0]
print(f"fitted slope {slope:.3f} vs. theoretical -1/2")
# plot log-log with se bars; overlay C * n**(-0.5) reference line
The caption must state: the model matches Assumptions 1-2, R = 50 replications, bars
are ±1 standard error, and the reference line is the Theorem 1 rate with fitted
constant. A fitted slope of −0.48 against a proved −1/2 is a persuasive picture; a
slope of −0.7 is a finding you must discuss (constants regime? bound loose? bug?)
rather than hide.
Separation and phase-transition pictures
- For a separation between two models or algorithm classes, plot both behaviors on
the same instance family so the divergence is visual, and print the instance-family
parameters in the caption.
- For a threshold phenomenon (learnable iff α > α*), sweep across the threshold with
enough resolution to show the transition sharpening as n grows — a single n is not
a phase-transition picture.
- Adversarial constructions from lower-bound proofs often make the best instances:
simulating your own hard instance shows it is concretely hard, not just
asymptotically.
Placement and captioning inside the paper
- Body placement: at most one illustration figure in the 12-page body, positioned
next to the theorem it visualizes, never in the introduction as decoration; the
remaining panels and the full procedure live in a terminal appendix section
(
colt-supplementary keeps numerics out of the proof flow).
- A COLT-grade caption is self-sufficient: instance family and parameters, the
assumptions regime, replication count, what the bars are, what the reference line
is, and the one-clause takeaway ("empirical slope matches the Theorem 1 rate").
- In-text discussion of the figure belongs in a Remark or a short subsection titled
for what it shows ("Illustration of the separation"), keeping the paper's logical
skeleton purely deductive.
- If space runs out, the figure is the first cut — state the numerical finding in one
sentence with a pointer to the appendix, and spend the recovered half page on a
proof overview.
Honesty rules that theorist reviewers enforce
- Never claim empirical support for regimes the theorem does not cover without the
word "conjecture" or "exploratory" attached.
- Never tune the illustration (seeds, instances) to flatter the bound; if fluctuations
are visible, show them.
- If the observed constants are large, say so — "the constant in Theorem 1 is
pessimistic; empirically C ≈ 3" is a respected sentence at this venue.
- A numerical section may not compensate for a proof gap, and reviewers will not
trade one for the other.
Cycle-volatility warnings
- The no-experiments-expected norm is structural to COLT, but scope language and any
code policy are re-stated each cycle in the CFP (待核实 annually).
- Reviewer appetite for numerics varies by subfield: optimization and RL-theory
reviewers often welcome a good rate plot; pure statistical-learning reviewers
frequently prefer the pages spent on proofs. Calibrate to your subject area.
- If your empirical component outgrows illustration into contribution, re-run venue
selection (
colt-topic-selection) — the paper may be drifting toward NeurIPS,
ICML, or AISTATS.
Output format
[Numerics verdict] none needed / illustration justified / empirical core (re-route?)
[Regime match] simulation inside proved regime / exploratory (labeled?)
[Figure plan] <rate plot / separation / phase transition; caption contents>
[Statistical floor] seeds, replications, bars defined
[Honesty check] <any claim exceeding the theorems>
1---2name: colt-experiments3description: Use when deciding whether a COLT (Conference on Learning Theory) paper needs numerical content at all — COLT has no experiments requirement — and, when numerics genuinely help, designing small illustrative simulations that visualize a proved bound, a separation, or a phase transition without diluting the theory.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/colt-experiments6---78# COLT Experiments910Start from the venue truth: **COLT imposes no experiments requirement, and most COLT11papers contain no experiments.** The 2026 CFP (checked 2026-07-08) asks for12theoretical machine-learning contributions and mentions no empirical-evaluation13expectation; papers are accepted on theorems. The decision this skill supports is14therefore *whether* to include numerics, and only then *how*.1516## Should this paper contain numerics at all?1718| Situation | Include numerics? | Rationale |19|---|---|---|20| Clean upper/lower bound pair, standard model | No | Plots add length, not belief |21| New algorithm whose practicality is part of the pitch | Small illustration | Shows the constants are not absurd |22| Theory explaining an empirical phenomenon (in-scope per the CFP's inclusive view) | Yes, essential | The phenomenon must be exhibited, then explained |23| Conjectured tightness you cannot prove | Careful, labeled | A scaling plot can support a conjecture — never upgrade it |24| Phase transition / separation between models | Often worthwhile | A picture of the transition is the most readable evidence |25| Purely structural result (equivalences, characterizations) | No | Nothing to simulate |2627If the answer is no, spend the pages on proof overviews instead; a decorative28benchmark table in a COLT submission signals venue confusion and can *lower* reviewer29confidence.3031## Design rules when numerics earn their place3233- Simulate the theorem's exact regime: the assumptions, the adversary, the parameter34 ranges. An illustration outside the proved regime must be labeled exploratory.35- Prefer synthetic constructions where ground truth is computable; COLT illustrations36 are about visualizing mathematics, not about datasets.37- Overlay the proved bound on the empirical curve so the reader sees slope agreement,38 constant gap, and where finite-size effects fade.39- Report the estimator of variability (bars = standard error over R runs, stated in40 the caption), fixed seeds, and replication counts — theorists distrust single41 trajectories on principle.42- Keep total numerical content to roughly a figure or two; the appendix holds the43 procedure description (no code-upload channel existed in the 2026 cycle).4445## A rate-illustration recipe4647The canonical COLT figure is a log-log rate check — empirical error or regret against48the driving parameter, with the theoretical slope for reference:4950```python51import numpy as np5253rng = np.random.default_rng(seed=2026) # fixed, reported seed54ns, R = np.logspace(2, 5, 8).astype(int), 50 # sample sizes, replications5556emp = np.array([[run_once(n, rng) for _ in range(R)] for n in ns])57mean, se = emp.mean(axis=1), emp.std(axis=1, ddof=1) / np.sqrt(R)5859slope = np.polyfit(np.log(ns), np.log(mean), 1)[0]60print(f"fitted slope {slope:.3f} vs. theoretical -1/2")61# plot log-log with se bars; overlay C * n**(-0.5) reference line62```6364The caption must state: the model matches Assumptions 1-2, R = 50 replications, bars65are ±1 standard error, and the reference line is the Theorem 1 rate with fitted66constant. A fitted slope of −0.48 against a proved −1/2 is a persuasive picture; a67slope of −0.7 is a finding you must discuss (constants regime? bound loose? bug?)68rather than hide.6970## Separation and phase-transition pictures7172- For a separation between two models or algorithm classes, plot both behaviors on73 the same instance family so the divergence is visual, and print the instance-family74 parameters in the caption.75- For a threshold phenomenon (learnable iff α > α*), sweep across the threshold with76 enough resolution to show the transition sharpening as n grows — a single n is not77 a phase-transition picture.78- Adversarial constructions from lower-bound proofs often make the best instances:79 simulating your own hard instance shows it is concretely hard, not just80 asymptotically.8182## Placement and captioning inside the paper8384- Body placement: at most one illustration figure in the 12-page body, positioned85 next to the theorem it visualizes, never in the introduction as decoration; the86 remaining panels and the full procedure live in a terminal appendix section87 (`colt-supplementary` keeps numerics out of the proof flow).88- A COLT-grade caption is self-sufficient: instance family and parameters, the89 assumptions regime, replication count, what the bars are, what the reference line90 is, and the one-clause takeaway ("empirical slope matches the Theorem 1 rate").91- In-text discussion of the figure belongs in a Remark or a short subsection titled92 for what it shows ("Illustration of the separation"), keeping the paper's logical93 skeleton purely deductive.94- If space runs out, the figure is the first cut — state the numerical finding in one95 sentence with a pointer to the appendix, and spend the recovered half page on a96 proof overview.9798## Honesty rules that theorist reviewers enforce99100- Never claim empirical support for regimes the theorem does not cover without the101 word "conjecture" or "exploratory" attached.102- Never tune the illustration (seeds, instances) to flatter the bound; if fluctuations103 are visible, show them.104- If the observed constants are large, say so — "the constant in Theorem 1 is105 pessimistic; empirically C ≈ 3" is a respected sentence at this venue.106- A numerical section may not compensate for a proof gap, and reviewers will not107 trade one for the other.108109## Cycle-volatility warnings110111- The no-experiments-expected norm is structural to COLT, but scope language and any112 code policy are re-stated each cycle in the CFP (待核实 annually).113- Reviewer appetite for numerics varies by subfield: optimization and RL-theory114 reviewers often welcome a good rate plot; pure statistical-learning reviewers115 frequently prefer the pages spent on proofs. Calibrate to your subject area.116- If your empirical component outgrows illustration into contribution, re-run venue117 selection (`colt-topic-selection`) — the paper may be drifting toward NeurIPS,118 ICML, or AISTATS.119120## Output format121122```text123[Numerics verdict] none needed / illustration justified / empirical core (re-route?)124[Regime match] simulation inside proved regime / exploratory (labeled?)125[Figure plan] <rate plot / separation / phase transition; caption contents>126[Statistical floor] seeds, replications, bars defined127[Honesty check] <any claim exceeding the theorems>128```