Emnlp Experiments

EMNLP Experiments

Use this while the experimental grid is still cheap to change. EMNLP's reviewing culture was partly built by papers criticizing NLP's own evaluation habits, so the venue audits experiments the way a security reviewer audits inputs: assume the design will be probed for the easiest way to make the headline number lie.

The five probes reviewers run

1. Baseline fairness. Did the comparison systems get the same tuning budget, data, and prompt engineering effort as yours? A win over an under-tuned baseline is the most common EMNLP soundness objection.
2. Coverage-claim match. Do the datasets, domains, and languages span the claim? English-only evidence supports English-only sentences.
3. Contamination. Could the pretrained or API model have seen the test data? For post-2020 NLP this is a default suspicion, not an exotic one.
4. Variance. Is the improvement larger than run-to-run noise? Single-run deltas of under a point convince no one here.
5. Mechanism. Does any experiment isolate why the method works, or only that it does? Ablations that remove the claimed ingredient are the minimum.

Design the grid so each probe has a prepared answer, and say where in the paper each answer lives.

Reporting floor by evidence type

Contamination audit, concretely

For every evaluation set: record its release date against the model's training cutoff; run n-gram or substring overlap between test instances and any accessible pretraining or fine-tuning corpora; where training data is closed (API models), state that directly and, when feasible, add a post-cutoff or perturbed test slice. Report the audit even when it finds nothing — "we checked" is evidence; silence is a reviewer question you chose to receive in July instead of answering in May.

Statistical practice that survives review

• Test at the right unit: sentence-level metrics on the same documents are not independent samples; use paired tests over documents or systems-by-item designs.
• Paired bootstrap or approximate randomization are the community's defaults for system comparison; report the number of resamples.
• Multiple metrics × multiple datasets × multiple models is a multiple-comparisons machine — state how many comparisons the paper makes and correct or temper claims accordingly.
• Power matters in the negative direction too: a "no difference" claim from 200 test items is an underpowered shrug, not a negative result.

Prompt sensitivity is an ablation, not an afterthought

Any result mediated by prompts inherits their variance. The minimum grid:

For each headline LLM result:
  - k ≥ 3 semantically equivalent prompt paraphrases  -> report mean ± spread
  - few-shot exemplar reshuffles (if applicable)      -> report order sensitivity
  - decoding: fixed and disclosed (temp, top_p)        -> no silent temperature 0.8
  - exact prompt text                                  -> appendix, verbatim
If the ranking of systems flips across paraphrases, the paper's claim is about
prompts, not systems — and the paper must say so.

Error analysis as an experiment

EMNLP error analysis is a designed study, not a paragraph: sample failures under a documented scheme (random within strata beats hand-picked), define error categories with two annotators and report agreement on the categorization itself, then connect categories to mechanism — which category does the proposed component reduce, and which does it leave untouched? A good error analysis generates follow-up experiments; a decorative one generates adjectives.

Human evaluation, designed like an experiment

When automatic metrics cannot measure the construct (adequacy, coherence, harm), the human study inherits the full burden of experimental design, and EMNLP reviewers grade it as one:

• Define the judgment as a question annotators could disagree about meaningfully — then measure that disagreement (report the agreement statistic and what level you consider acceptable for this construct, since chance-corrected agreement on skewed labels behaves badly).
• Sample instances for rating by a documented rule; rating each system's outputs on different items is a design error, not a shortcut.
• Blind raters to system identity and randomize presentation order; order effects in side-by-side preference ratings are large and well known.
• Report rater recruitment, training, compensation, and count — the Responsible NLP checklist requires it, and reviewers cross-check the two.
• Analyze ratings with models matching their structure (ordinal, per-rater variance), or at minimum report per-item aggregation rules; averaging Likert scales across raters and items without comment is the field's most tolerated bad habit, and its tolerance is expiring.

Grid economics under a deadline

When compute or time forces cuts, cut in this order: extra datasets confirming an already-shown effect first; extra model scales second; never cut the seeds/variance runs or the contamination audit — they are cheap relative to the review risk they retire. A smaller grid with variance beats a wider grid of single runs at this venue, every time.

Output format

[Probe readiness] <fairness / coverage / contamination / variance / mechanism: ready or gap>
[Reporting-floor gaps] <evidence type -> missing item>
[Statistical plan] <test, unit, corrections, power posture>
[Prompt-sensitivity status] <done / needed / not applicable>
[Error-analysis design] <sampling, categories, agreement, mechanism link>

Source: brycewang-stanford/Awesome-Journal-Skills → EMNLP-Skills/skills/emnlp-experiments/SKILL.md