Colt Experiments

COLT Experiments

Start from the venue truth: COLT imposes no experiments requirement, and most COLT papers contain no experiments. The 2026 CFP (checked 2026-07-08) asks for theoretical machine-learning contributions and mentions no empirical-evaluation expectation; papers are accepted on theorems. The decision this skill supports is therefore whether to include numerics, and only then how.

Should this paper contain numerics at all?

If the answer is no, spend the pages on proof overviews instead; a decorative benchmark table in a COLT submission signals venue confusion and can lower reviewer confidence.

Design rules when numerics earn their place

• Simulate the theorem's exact regime: the assumptions, the adversary, the parameter ranges. An illustration outside the proved regime must be labeled exploratory.
• Prefer synthetic constructions where ground truth is computable; COLT illustrations are about visualizing mathematics, not about datasets.
• Overlay the proved bound on the empirical curve so the reader sees slope agreement, constant gap, and where finite-size effects fade.
• Report the estimator of variability (bars = standard error over R runs, stated in the caption), fixed seeds, and replication counts — theorists distrust single trajectories on principle.
• Keep total numerical content to roughly a figure or two; the appendix holds the procedure description (no code-upload channel existed in the 2026 cycle).

A rate-illustration recipe

The canonical COLT figure is a log-log rate check — empirical error or regret against the driving parameter, with the theoretical slope for reference:

import numpy as np

rng = np.random.default_rng(seed=2026)          # fixed, reported seed
ns, R = np.logspace(2, 5, 8).astype(int), 50    # sample sizes, replications

emp = np.array([[run_once(n, rng) for _ in range(R)] for n in ns])
mean, se = emp.mean(axis=1), emp.std(axis=1, ddof=1) / np.sqrt(R)

slope = np.polyfit(np.log(ns), np.log(mean), 1)[0]
print(f"fitted slope {slope:.3f} vs. theoretical -1/2")
# plot log-log with se bars; overlay C * n**(-0.5) reference line

The caption must state: the model matches Assumptions 1-2, R = 50 replications, bars are ±1 standard error, and the reference line is the Theorem 1 rate with fitted constant. A fitted slope of −0.48 against a proved −1/2 is a persuasive picture; a slope of −0.7 is a finding you must discuss (constants regime? bound loose? bug?) rather than hide.

Separation and phase-transition pictures

• For a separation between two models or algorithm classes, plot both behaviors on the same instance family so the divergence is visual, and print the instance-family parameters in the caption.
• For a threshold phenomenon (learnable iff α > α*), sweep across the threshold with enough resolution to show the transition sharpening as n grows — a single n is not a phase-transition picture.
• Adversarial constructions from lower-bound proofs often make the best instances: simulating your own hard instance shows it is concretely hard, not just asymptotically.

Placement and captioning inside the paper

• Body placement: at most one illustration figure in the 12-page body, positioned next to the theorem it visualizes, never in the introduction as decoration; the remaining panels and the full procedure live in a terminal appendix section (colt-supplementary keeps numerics out of the proof flow).
• A COLT-grade caption is self-sufficient: instance family and parameters, the assumptions regime, replication count, what the bars are, what the reference line is, and the one-clause takeaway ("empirical slope matches the Theorem 1 rate").
• In-text discussion of the figure belongs in a Remark or a short subsection titled for what it shows ("Illustration of the separation"), keeping the paper's logical skeleton purely deductive.
• If space runs out, the figure is the first cut — state the numerical finding in one sentence with a pointer to the appendix, and spend the recovered half page on a proof overview.

Honesty rules that theorist reviewers enforce

• Never claim empirical support for regimes the theorem does not cover without the word "conjecture" or "exploratory" attached.
• Never tune the illustration (seeds, instances) to flatter the bound; if fluctuations are visible, show them.
• If the observed constants are large, say so — "the constant in Theorem 1 is pessimistic; empirically C ≈ 3" is a respected sentence at this venue.
• A numerical section may not compensate for a proof gap, and reviewers will not trade one for the other.

Cycle-volatility warnings

• The no-experiments-expected norm is structural to COLT, but scope language and any code policy are re-stated each cycle in the CFP (待核实 annually).
• Reviewer appetite for numerics varies by subfield: optimization and RL-theory reviewers often welcome a good rate plot; pure statistical-learning reviewers frequently prefer the pages spent on proofs. Calibrate to your subject area.
• If your empirical component outgrows illustration into contribution, re-run venue selection (colt-topic-selection) — the paper may be drifting toward NeurIPS, ICML, or AISTATS.

Output format

[Numerics verdict] none needed / illustration justified / empirical core (re-route?)
[Regime match] simulation inside proved regime / exploratory (labeled?)
[Figure plan] <rate plot / separation / phase transition; caption contents>
[Statistical floor] seeds, replications, bars defined
[Honesty check] <any claim exceeding the theorems>

Source: brycewang-stanford/Awesome-Journal-Skills → COLT-Skills/skills/colt-experiments/SKILL.md