yuangao-tum NLP Rebuttal Scenario Playbook
Translated and adapted from MLNLP-World/Paper-Rebuttal-Tips. 28 recurring reviewer-concern scenarios. Each has four parts: the concern, a bad answer that backfires, a recommended answer template, and the takeaway.
Good rebuttal = Respect + Evidence + Clarity.
How to use
- Classify each reviewer comment with the router below.
- Load ONLY the reference file(s) for the matched tips.
- Adapt the recommended-answer template: replace every placeholder (XXX, A/B/C, Table X) with the paper's real content and freshly computed numbers. Never ship a template verbatim.
- For overall response structure, ordering, and tone, use the
write-rebuttalskill (itemize → brain-dump → draft → revise, the 18 tactics, the neutral-third-party test). These two skills compose: that one shapes the whole response, this one shapes each answer.
Concern router
| Reviewer concern sounds like | Tip | Reference |
|---|---|---|
| "Too complex", "bag of tricks", "which component matters?" | 1 | innovation-theory.md |
| "Not novel", "similar to prior work A" | 2 | innovation-theory.md |
| "Just a combination of existing techniques" | 3 | innovation-theory.md |
| "Contributions unclear" | 4 | innovation-theory.md |
| "Motivation unclear", "why is this problem important?" | 5 | innovation-theory.md |
| "No theoretical analysis", "why does it work?" | 6 | innovation-theory.md |
| "Limitations discussion is superficial" | 7 | innovation-theory.md |
| "Related work missing/insufficient" | 8 | communication-writing.md |
| "Writing/notation unclear" | 9 | communication-writing.md |
| Reviewer misunderstood the method | 10 | communication-writing.md |
| Vague, low-quality negative review | 11 | communication-writing.md |
| Tempted to reply "we will add..." | 12 | communication-writing.md |
| "Missing/weak baselines" | 13 | experiments-evidence.md |
| "Improvements are marginal" | 14 | experiments-evidence.md |
| "Unfair experimental setup" | 15 | experiments-evidence.md |
| "Missing ablations" | 16 | experiments-evidence.md |
| "Too much computational overhead" | 17 | experiments-evidence.md |
| Asked for experiments too large for the rebuttal window | 18 | experiments-evidence.md |
| "Dataset too small" | 19 | experiments-evidence.md |
| "Generalization not shown" (few datasets/models/tasks) | 20 | experiments-evidence.md |
| "No variance / significance / seeds" | 21 | experiments-evidence.md |
| "Possible train/test leakage or contamination" | 22 | experiments-evidence.md |
| "Hyperparameter sensitivity?" ("why k=40?") | 23 | experiments-evidence.md |
| "Wrong/missing evaluation metrics" | 24 | experiments-evidence.md |
| "No human evaluation" | 25 | experiments-evidence.md |
| "Intermediate outputs never evaluated directly" | 26 | experiments-evidence.md |
| Claims "continual/online" but experiments are one-shot offline | 27 | experiments-evidence.md |
| "No code, seeds, or hyperparameters — not reproducible" | 28 | experiments-evidence.md |
A single comment often maps to several tips (e.g. "marginal gains and no significance testing" = 14 + 21). Load all matches and merge their strategies into one answer.
Cross-cutting rules (from the 28 scenarios)
- Act, don't promise (Tip 12): run the number/analysis now and put it in the rebuttal. "We will add X in the revision" alone convinces nobody.
- Never blame the reviewer (Tips 4, 9, 10): if they misread, the fix is a clarification plus a pointer to the line, stated neutrally.
- Answer head-on (Tips 2, 3): name exactly where the difference or novelty lies — motivation, mechanism, role — not just that it exists.
- Evidence over adjectives (Tips 13-28): every disputed claim gets a table, an ablation, a test, or an honest statement of infeasibility with a scaled-down proxy result (Tip 18).
- Concede real weaknesses gracefully (Tips 7, 14, 19): bound the claim, show the trend, explain what the paper still establishes.
Anti-patterns (never do)
- Asserting all components are necessary without per-component ablation (Tip 1).
- "We are the first to apply X to Y" as the whole novelty defense (Tip 3).
- Repeating the introduction as the answer to a motivation question (Tip 5).
- "Experiments show it works" as the answer to a theory question (Tip 6).
- Calling the setup fair without matching compute/tuning budgets (Tip 15).
- Dismissing a metric request instead of adding the metric (Tip 24).