Iccv Experiments

ICCV Experiments

An ICCV experimental program is built under a fixed, unrepeatable date: the early-March deadline of an odd year (March 7 in 2025, with supplement due the same day). The program design problem is therefore sequencing under a deadline, on top of the usual question of what evidence convinces a vision reviewer. Both are handled here.

Start with the benchmark-drift audit

Because the venue is biennial, a project conceived after one ICCV and submitted to the next spans two years of field motion. Before designing experiments, audit what moved:

drift audit (fill once, in the autumn before the deadline)
  benchmarks:   which datasets did the last two years of CVPR/ECCV/ICCV papers
                in this area actually evaluate on? any new canonical benchmark?
  baselines:    leaderboard top-3 today vs when the project started
                → any baseline in your draft older than ~18 months is a red flag
  backbones:    what does current SOTA initialize from? (matching this defines
                "fair" for your comparisons)
  metrics:      any metric revision or new evaluation server since last cycle?
  protocols:    resolution/prompt/eval-harness conventions that changed

Papers rejected for "outdated comparisons" are usually not lazy — they froze their experiment matrix at project start and never re-based. Re-run the audit in January; two months before an ICCV deadline is exactly when the previous November's CVPR-cycle preprints flood arXiv.

The fairness ledger

Vision reviewers' most reliable objection is compute-and-pretraining confounds dressed as method wins. Make fairness auditable with a ledger column per comparison:

The foundation-model twist: when everything builds on the same giant checkpoint, data exposure replaces architecture as the confound reviewers hunt. If your improvement could plausibly come from the pretrain having seen the test domain, run the decontamination or cross-domain check before a reviewer asks for it in a window when you have one rebuttal page to respond.

Ablations that isolate, not decorate

Structure the ablation grid so every row flips one switch, and include the two rows that distinguish a mechanism from a lucky configuration: the transplant (your module inserted into a baseline — does the gain travel?) and the sensitivity sweep (is the headline number a plateau or a spike?). Rows argued from in the text belong in the body; the full grid goes to the same-day supplement. If the core ablation shows the mechanism is not doing the work, that is an October discovery you want in November — which is why it runs first (see sequencing below).

Qualitative evidence with stated rules

At a venue that reviews with its eyes, image and video evidence carries real weight and attracts real skepticism. Three requirements: a declared selection rule on every grid ("first N val images", "random seed 0" — curation without a rule is what reviewers assume by default); side-by-sides against the two strongest baselines on identical inputs; and a failure-mode section with a taxonomy, previewed in the body and cataloged in the supplement. For temporal or 3D claims, the supplement video is the primary exhibit — packaging in iccv-supplementary.

Sequencing runs toward March

The scarce resource is calendar, not GPUs. Order the program by decision value per week:

1. Falsifiers first (autumn): the core ablation and the strongest baseline comparison. If the idea dies, it dies while retargeting to CVPR-November is still possible (iccv-workflow's autumn fork).
2. Headline table (winter): full benchmark suite, matched settings, seeds on the cheap rows.
3. Breadth pass (January): second domain, transfer, robustness suite — whatever supports the claim's outer scope.
4. Freeze margin (mid-February): the main table freezes ~2–3 weeks out; the deadline is same-day for the supplement, so late results have no legal landing zone except the rebuttal — and reviewers may not request major new experiments anyway.
5. Rebuttal reserve: hold 10–20% of compute for May; the most common winnable rebuttal item is a small matched re-run reported as a mini-table (iccv-author-response).

# deadline_math.py — sanity-check the plan against the calendar
runs = {"core_ablation": 6, "main_table": 21, "breadth": 10}   # GPU-days each
gpu = 8; days_left = (deadline - today).days - 18              # freeze margin
assert sum(runs.values()) / gpu <= days_left, "cut scope now, not in February"

The four questions any ICCV review silently asks

Does it work (main table, matched)? Why does it work (isolating ablations + transplant)? Where does it break (failure taxonomy, honest transfer results)? What does it cost (params, latency on named hardware, training GPU-hours — volunteered, since no form mandates it; see iccv-reproducibility)? Draft the experiments section as answers to these four, in this order, and the reviewer's checklist fills itself.

Reverify each cycle

• The 2027 deadline chain — sequencing above is calendar-shaped and the calendar is 待核实 until posted.
• Whether supplements stay same-day (changes step 4).
• Evaluation-server rules and submission budgets on your benchmarks.
• Any new ethics/human-data documentation the 2027 forms may require.

Output format

[Drift audit] run on <date>; stale baselines found: <list>
[Fairness ledger] axes matched or disclosed per comparison: n/m
[Ablation] one-switch rows: <n>; transplant + sensitivity present: yes/no
[Qualitative] selection rules stated · failure taxonomy drafted
[Sequencing] falsifiers scheduled before <date>; rebuttal reserve: <GPU-days>
[Cut candidates] <lowest decision-value runs if the calendar slips>

Source: brycewang-stanford/Awesome-Journal-Skills → ICCV-Skills/skills/iccv-experiments/SKILL.md