Research Paper Writing

Research Paper Writing Pipeline

End-to-end pipeline for producing publication-ready ML/AI research papers targeting NeurIPS, ICML, ICLR, ACL, AAAI, and COLM. This skill covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission.

This is not a linear pipeline — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops.

┌─────────────────────────────────────────────────────────────┐
│                    RESEARCH PAPER PIPELINE                  │
│                                                             │
│  Phase 0: Project Setup ──► Phase 1: Literature Review      │
│       │                          │                          │
│       ▼                          ▼                          │
│  Phase 2: Experiment     Phase 5: Paper Drafting ◄──┐      │
│       Design                     │                   │      │
│       │                          ▼                   │      │
│       ▼                    Phase 6: Self-Review      │      │
│  Phase 3: Execution &           & Revision ──────────┘      │
│       Monitoring                 │                          │
│       │                          ▼                          │
│       ▼                    Phase 7: Submission               │
│  Phase 4: Analysis ─────► (feeds back to Phase 2 or 5)     │
│                                                             │
└─────────────────────────────────────────────────────────────┘

When To Use This Skill

Use this skill when:

• Starting a new research paper from an existing codebase or idea
• Designing and running experiments to support paper claims
• Writing or revising any section of a research paper
• Preparing for submission to a specific conference or workshop
• Responding to reviews with additional experiments or revisions
• Converting a paper between conference formats
• Writing non-empirical papers — theory, survey, benchmark, or position papers (see Paper Types Beyond Empirical ML)
• Designing human evaluations for NLP, HCI, or alignment research
• Preparing post-acceptance deliverables — posters, talks, code releases

Core Philosophy

1. Be proactive. Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate.
2. Never hallucinate citations. AI-generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as [CITATION NEEDED].
3. Paper is a story, not a collection of experiments. Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready.
4. Experiments serve claims. Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative.
5. Commit early, commit often. Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history.

Proactivity and Collaboration

Default: Be proactive. Draft first, ask with the draft.

Block for input only when: target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first.

Phase 0: Project Setup

Goal: Establish the workspace, understand existing work, identify the contribution.

Step 0.1: Explore the Repository

# Understand project structure
ls -la
find . -name "*.py" | head -30
find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding"

Look for:

• README.md — project overview and claims
• results/, outputs/, experiments/ — existing findings
• configs/ — experimental settings
• .bib files — existing citations
• Draft documents or notes

Step 0.2: Organize the Workspace

Establish a consistent workspace structure:

workspace/
  paper/               # LaTeX source, figures, compiled PDFs
  experiments/         # Experiment runner scripts
  code/                # Core method implementation
  results/             # Raw experiment results (auto-generated)
  tasks/               # Task/benchmark definitions
  human_eval/          # Human evaluation materials (if needed)

Step 0.3: Set Up Version Control

git init  # if not already
git remote add origin <repo-url>
git checkout -b paper-draft  # or main

Git discipline: Every completed experiment batch gets committed with a descriptive message. Example:

Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task)
Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier

Step 0.4: Identify the Contribution

Before writing anything, articulate:

• The What: What is the single thing this paper contributes?
• The Why: What evidence supports it?
• The So What: Why should readers care?

Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?"

Step 0.5: Create a TODO List

Use the todo tool to create a structured project plan:

Research Paper TODO:
- [ ] Define one-sentence contribution
- [ ] Literature review (related work + baselines)
- [ ] Design core experiments
- [ ] Run experiments
- [ ] Analyze results
- [ ] Write first draft
- [ ] Self-review (simulate reviewers)
- [ ] Revise based on review
- [ ] Submission prep

Update this throughout the project. It serves as the persistent state across sessions.

Step 0.6: Estimate Compute Budget

Before running experiments, estimate total cost and time:

Compute Budget Checklist:
- [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs)
- [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds)
- [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate)
- [ ] Total budget ceiling and contingency (add 30-50% for reruns)

Track actual spend as experiments run:

# Simple cost tracker pattern
import json, os
from datetime import datetime

COST_LOG = "results/cost_log.jsonl"

def log_cost(experiment: str, model: str, input_tokens: int, output_tokens: int, cost_usd: float):
    entry = {
        "timestamp": datetime.now().isoformat(),
        "experiment": experiment,
        "model": model,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "cost_usd": cost_usd,
    }
    with open(COST_LOG, "a") as f:
        f.write(json.dumps(entry) + "\n")

When budget is tight: Run pilot experiments (1-2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs.

Step 0.7: Multi-Author Coordination

Most papers have 3-10 authors. Establish workflows early:

Section ownership: Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency.

Author Coordination Checklist:
- [ ] Agree on section ownership (who writes what)
- [ ] Set up shared workspace (Overleaf or git repo)
- [ ] Establish notation conventions (before anyone writes)
- [ ] Schedule internal review rounds (not just at the end)
- [ ] Designate one person for final formatting pass
- [ ] Agree on figure style (colors, fonts, sizes) before creating figures

LaTeX conventions to agree on early:

• \method{} macro for consistent method naming
• Citation style: \citet{} vs \citep{} usage
• Math notation: lowercase bold for vectors, uppercase bold for matrices, etc.
• British vs American spelling

Phase 1: Literature Review

Goal: Find related work, identify baselines, gather citations.

Step 1.1: Identify Seed Papers

Start from papers already referenced in the codebase:

# Via terminal:
grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py"
find . -name "*.bib"

Step 1.2: Search for Related Work

Load the arxiv skill for structured paper discovery: skill_view("arxiv"). It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation.

Use web_search for broad discovery, web_extract for fetching specific papers:

# Via web_search:
web_search("[main technique] + [application domain] site:arxiv.org")
web_search("[baseline method] comparison ICML NeurIPS 2024")

# Via web_extract (for specific papers):
web_extract("https://arxiv.org/abs/2303.17651")

Additional search queries to try:

Search queries:
- "[main technique] + [application domain]"
- "[baseline method] comparison"
- "[problem name] state-of-the-art"
- Author names from existing citations

Recommended: Install Exa MCP for real-time academic search:

claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp"

Step 1.2b: Deepen the Search (Breadth-First, Then Depth)

A flat search (one round of queries) typically misses important related work. Use an iterative breadth-then-depth pattern inspired by deep research pipelines:

Iterative Literature Search:

Round 1 (Breadth): 4-6 parallel queries covering different angles
  - "[method] + [domain]"
  - "[problem name] state-of-the-art 2024 2025"
  - "[baseline method] comparison"
  - "[alternative approach] vs [your approach]"
  → Collect papers, extract key concepts and terminology

Round 2 (Depth): Generate follow-up queries from Round 1 learnings
  - New terminology discovered in Round 1 papers
  - Papers cited by the most relevant Round 1 results
  - Contradictory findings that need investigation
  → Collect papers, identify remaining gaps

Round 3 (Targeted): Fill specific gaps
  - Missing baselines identified in Rounds 1-2
  - Concurrent work (last 6 months, same problem)
  - Key negative results or failed approaches
  → Stop when new queries return mostly papers you've already seen

When to stop: If a round returns >80% papers already in your collection, the search is saturated. Typically 2-3 rounds suffice. For survey papers, expect 4-5 rounds.

For agent-based workflows: Delegate each round's queries in parallel via delegate_task. Collect results, deduplicate, then generate the next round's queries from the combined learnings.

Step 1.3: Verify Every Citation

NEVER generate BibTeX from memory. ALWAYS fetch programmatically.

For each citation, follow the mandatory 5-step process:

Citation Verification (MANDATORY per citation):
1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords
2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef)
3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory)
4. VALIDATE → Confirm the claim you're citing actually appears in the paper
5. ADD → Add verified BibTeX to bibliography
If ANY step fails → mark as [CITATION NEEDED], inform scientist

# Fetch BibTeX via DOI
import requests

def doi_to_bibtex(doi: str) -> str:
    response = requests.get(
        f"https://doi.org/{doi}",
        headers={"Accept": "application/x-bibtex"}
    )
    response.raise_for_status()
    return response.text

If you cannot verify a citation:

\cite{PLACEHOLDER_author2024_verify_this}  % TODO: Verify this citation exists

Always tell the scientist: "I've marked [X] citations as placeholders that need verification."

See references/citation-workflow.md for complete API documentation and the full CitationManager class.

Step 1.4: Organize Related Work

Group papers by methodology, not paper-by-paper:

Good: "One line of work uses X's assumption [refs] whereas we use Y's assumption because..." Bad: "Smith et al. introduced X. Jones et al. introduced Y. We combine both."

Phase 2: Experiment Design

Goal: Design experiments that directly support paper claims. Every experiment must answer a specific question.

Step 2.1: Map Claims to Experiments

Create an explicit mapping:

Rule: If an experiment doesn't map to a claim, don't run it.

Step 2.2: Design Baselines

Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?"

Standard baseline categories:

• Naive baseline: Simplest possible approach
• Strong baseline: Best known existing method
• Ablation baselines: Your method minus one component
• Compute-matched baselines: Same compute budget, different allocation

Step 2.3: Define Evaluation Protocol

Before running anything, specify:

• Metrics: What you're measuring, direction symbols (higher/lower better)
• Aggregation: How results are combined across runs/tasks
• Statistical tests: What tests will establish significance
• Sample sizes: How many runs/problems/tasks

Step 2.4: Write Experiment Scripts

Follow these patterns from successful research pipelines:

Incremental saving — save results after each step for crash recovery:

# Save after each problem/task
result_path = f"results/{task}/{strategy}/result.json"
if os.path.exists(result_path):
    continue  # Skip already-completed work
# ... run experiment ...
with open(result_path, 'w') as f:
    json.dump(result, f, indent=2)

Artifact preservation — save all intermediate outputs:

results/<experiment>/
  <task>/
    <strategy>/
      final_output.md          # Final result
      history.json             # Full trajectory
      pass_01/                 # Per-iteration artifacts
        version_a.md
        version_b.md
        critic.md

Separation of concerns — keep generation, evaluation, and visualization separate:

run_experiment.py              # Core experiment runner
run_baselines.py               # Baseline comparison
run_comparison_judge.py        # Blind evaluation
analyze_results.py             # Statistical analysis
make_charts.py                 # Visualization

See references/experiment-patterns.md for complete design patterns, cron monitoring, and error recovery.

Step 2.5: Design Human Evaluation (If Applicable)

Many NLP, HCI, and alignment papers require human evaluation as primary or complementary evidence. Design this before running automated experiments — human eval often has longer lead times (IRB approval, annotator recruitment).

When human evaluation is needed:

• Automated metrics don't capture what you care about (fluency, helpfulness, safety)
• Your contribution is about human-facing qualities (readability, preference, trust)
• Reviewers at NLP venues (ACL, EMNLP) expect it for generation tasks

Key design decisions:

Annotation guideline checklist:

- [ ] Clear task description with examples (good AND bad)
- [ ] Decision criteria for ambiguous cases
- [ ] At least 2 worked examples per category
- [ ] Attention checks / gold standard items (10-15% of total)
- [ ] Qualification task or screening round
- [ ] Estimated time per item and fair compensation (>= local minimum wage)
- [ ] IRB/ethics review if required by your institution

Reporting requirements (reviewers check all of these):

• Number of annotators and their qualifications
• Inter-annotator agreement with specific metric and value
• Compensation details (amount, estimated hourly rate)
• Annotation interface description or screenshot (appendix)
• Total annotation time

See references/human-evaluation.md for complete guide including statistical tests for human eval data, crowdsourcing quality control patterns, and IRB guidance.

Phase 3: Experiment Execution & Monitoring

Goal: Run experiments reliably, monitor progress, recover from failures.

Step 3.1: Launch Experiments

Use nohup for long-running experiments:

nohup python run_experiment.py --config config.yaml > logs/experiment_01.log 2>&1 &
echo $!  # Record the PID

Parallel execution: Run independent experiments simultaneously, but be aware of API rate limits. 4+ concurrent experiments on the same API will slow each down.

Step 3.2: Set Up Monitoring (Cron Pattern)

For long-running experiments, set up periodic status checks. The cron prompt should follow this template:

Monitor Prompt Template:
1. Check if process is still running: ps aux | grep <pattern>
2. Read last 30 lines of log: tail -30 <logfile>
3. Check for completed results: ls <result_dir>
4. If results exist, read and report: cat <result_file>
5. If all done, commit: git add -A && git commit -m "<descriptive message>" && git push
6. Report in structured format (tables with key metrics)
7. Answer the key analytical question for this experiment

Silent mode: If nothing has changed since the last check, respond with [SILENT] to suppress notification to the user. Only report when there's news.

Step 3.3: Handle Failures

Common failure modes and recovery:

Key: Scripts should always check for existing results and skip completed work. This makes re-runs safe and efficient.

Step 3.4: Commit Completed Results

After each experiment batch completes:

git add -A
git commit -m "Add <experiment name>: <key finding in 1 line>"
git push

Step 3.5: Maintain an Experiment Journal

Git commits track what happened, but not the exploration tree — the decisions about what to try next based on what you learned. Maintain a structured experiment journal that captures this tree:

// experiment_journal.jsonl — append one entry per experiment attempt
{
  "id": "exp_003",
  "parent": "exp_001",
  "timestamp": "2025-05-10T14:30:00Z",
  "hypothesis": "Adding scope constraints will fix convergence failure from exp_001",
  "plan": "Re-run autoreason with max_tokens=2000 and fixed structure template",
  "config": {"model": "haiku", "strategy": "autoreason", "max_tokens": 2000},
  "status": "completed",
  "result_path": "results/exp_003/",
  "key_metrics": {"win_rate": 0.85, "convergence_rounds": 3},
  "analysis": "Scope constraints fixed convergence. Win rate jumped from 0.42 to 0.85.",
  "next_steps": ["Try same constraints on Sonnet", "Test without structure template"],
  "figures": ["figures/exp003_convergence.pdf"]
}

Why a journal, not just git? Git tracks file changes. The journal tracks the reasoning: why you tried X, what you learned, and what that implies for the next experiment. When writing the paper, this tree is invaluable for the Methods section ("we observed X, which motivated Y") and for honest failure reporting.

Selecting the best path: When the journal shows a branching tree (exp_001 → exp_002a, exp_002b, exp_003), identify the path that best supports the paper's claims. Document dead-end branches in the appendix as ablations or negative results.

Snapshot code per experiment: Copy the experiment script after each run:

cp experiment.py results/exp_003/experiment_snapshot.py

This enables exact reproduction even after subsequent code changes.

Phase 4: Result Analysis

Goal: Extract findings, compute statistics, identify the story.

Step 4.1: Aggregate Results

Write analysis scripts that:

1. Load all result files from a batch
2. Compute per-task and aggregate metrics
3. Generate summary tables

# Standard analysis pattern
import json, os
from pathlib import Path

results = {}
for result_file in Path("results/").rglob("result.json"):
    data = json.loads(result_file.read_text())
    strategy = result_file.parent.name
    task = result_file.parent.parent.name
    results.setdefault(strategy, {})[task] = data

# Compute aggregate metrics
for strategy, tasks in results.items():
    scores = [t["score"] for t in tasks.values()]
    print(f"{strategy}: mean={np.mean(scores):.1f}, std={np.std(scores):.1f}")

Step 4.2: Statistical Significance

Always compute:

• Error bars: Standard deviation or standard error, specify which
• Confidence intervals: 95% CI for key results
• Pairwise tests: McNemar's test for comparing two methods
• Effect sizes: Cohen's d or h for practical significance

See references/experiment-patterns.md for complete implementations of McNemar's test, bootstrapped CIs, and Cohen's h.

Step 4.3: Identify the Story

After analysis, explicitly answer:

1. What is the main finding? State it in one sentence.
2. What surprised you? Unexpected results often make the best papers.
3. What failed? Failed experiments can be the most informative. Honest reporting of failures strengthens the paper.
4. What follow-up experiments are needed? Results often raise new questions.

Handling Negative or Null Results

When your hypothesis was wrong or results are inconclusive, you have three options:

How to write a negative results paper:

• Lead with what the community believes and why it matters to test it
• Describe your rigorous methodology (must be airtight — reviewers will scrutinize harder)
• Present the null result clearly with statistical evidence
• Analyze why the expected result didn't materialize
• Discuss implications for the field

Venues that explicitly welcome negative results: NeurIPS (Datasets & Benchmarks track), TMLR, ML Reproducibility Challenge, workshops at major conferences. Some workshops specifically call for negative results.

Step 4.4: Create Figures and Tables

Figures:

• Use vector graphics (PDF) for all plots: plt.savefig('fig.pdf')
• Colorblind-safe palettes (Okabe-Ito or Paul Tol)
• Self-contained captions — reader should understand without main text
• No title inside figure — the caption serves this function

Tables:

• Use booktabs LaTeX package
• Bold best value per metric
• Include direction symbols (higher/lower better)
• Consistent decimal precision

\usepackage{booktabs}
\begin{tabular}{lcc}
\toprule
Method & Accuracy $\uparrow$ & Latency $\downarrow$ \\
\midrule
Baseline & 85.2 & 45ms \\
\textbf{Ours} & \textbf{92.1} & 38ms \\
\bottomrule
\end{tabular}

Step 4.5: Decide: More Experiments or Write?

Step 4.6: Write the Experiment Log (Bridge to Writeup)

Before moving to paper writing, create a structured experiment log that bridges results to prose. This is the single most important connective tissue between experiments and the writeup — without it, the writing agent has to re-derive the story from raw result files.

Create experiment_log.md with the following structure:

# Experiment Log

## Contribution (one sentence)
[The paper's main claim]

## Experiments Run

### Experiment 1: [Name]
- **Claim tested**: [Which paper claim this supports]
- **Setup**: [Model, dataset, config, number of runs]
- **Key result**: [One sentence with the number]
- **Result files**: results/exp1/final_info.json
- **Figures generated**: figures/exp1_comparison.pdf
- **Surprising findings**: [Anything unexpected]

### Experiment 2: [Name]
...

## Figures
| Filename | Description | Which section it belongs in |
|----------|-------------|---------------------------|
| figures/main_comparison.pdf | Bar chart comparing all methods on benchmark X | Results, Figure 2 |
| figures/ablation.pdf | Ablation removing components A, B, C | Results, Figure 3 |
...

## Failed Experiments (document for honesty)
- [What was tried, why it failed, what it tells us]

## Open Questions
- [Anything the results raised that the paper should address]

Why this matters: When drafting, the agent (or a delegated sub-agent) can load experiment_log.md alongside the LaTeX template and produce a first draft grounded in actual results. Without this bridge, the writing agent must parse raw JSON/CSV files and infer the story — a common source of hallucinated or misreported numbers.

Git discipline: Commit this log alongside the results it describes.

Iterative Refinement: Strategy Selection

Any output in this pipeline — paper drafts, experiment scripts, analysis — can be iteratively refined. The autoreason research provides empirical evidence for when each refinement strategy works and when it fails. Use this section to choose the right approach.

Quick Decision Table

The Generation-Evaluation Gap

Core insight: Autoreason's value depends on the gap between a model's generation capability and its self-evaluation capability.

Model Tier        │ Generation │ Self-Eval │ Gap    │ Autoreason Value
──────────────────┼────────────┼───────────┼────────┼─────────────────
Weak (Llama 8B)   │ Poor       │ Poor      │ Small  │ None — can't generate diverse candidates
Mid (Haiku 3.5)   │ Decent     │ Poor      │ LARGE  │ MAXIMUM — 42/42 perfect Borda
Mid (Gemini Flash)│ Decent     │ Moderate  │ Large  │ High — wins 2/3
Strong (Sonnet 4) │ Good       │ Decent    │ Medium │ Moderate — wins 3/5
Frontier (S4.6)   │ Excellent  │ Good      │ Small  │ Only with constraints

This gap is structural, not temporary. As costs drop, today's frontier becomes tomorrow's mid-tier. The sweet spot moves but never disappears.

Autoreason Loop (Summary)

Each pass produces three candidates from fresh, isolated agents:

1. Critic → finds problems in incumbent A (no fixes)
2. Author B → revises A based on critique
3. Synthesizer → merges A and B (randomized labels)
4. Judge Panel → 3 blind CoT judges rank A, B, AB via Borda count
5. Convergence → A wins k=2 consecutive passes → done

Key parameters:

• k=2 convergence (k=1 premature, k=3 too expensive, no quality gain)
• CoT judges always (3x faster convergence)
• Temperature 0.8 authors, 0.3 judges
• Conservative tiebreak: incumbent wins ties
• Every role is a fresh agent with no shared context

Applying to Paper Drafts

When refining the paper itself through autoreason:

• Provide ground truth to the critic: actual experimental data, result JSONs, statistical outputs. Without this, models hallucinate fabricated ablation studies and fake confidence intervals.
• Use 3 working judges minimum: A broken judge parser doesn't add noise — it prevents equilibrium entirely.
• Scope constrain the revision: "Address these specific weaknesses" not "improve the paper."

Failure Modes

See references/autoreason-methodology.md for complete prompts, Borda scoring details, model selection guide, scope constraint design patterns, and compute budget reference.

Phase 5: Paper Drafting

The complete drafting procedure (section-by-section order, LaTeX scaffolding, figure/table conventions, abstract and intro formulas, related-work positioning) lives in references/phase5-paper-drafting.md — load it with read_file when you reach this phase. Pair it with references/writing-guide.md for prose-level style rules.

Phase 6: Self-Review & Revision

Goal: Simulate the review process before submission. Catch weaknesses early.

Step 6.1: Simulate Reviews (Ensemble Pattern)

Generate reviews from multiple perspectives. The key insight from automated research pipelines (notably SakanaAI's AI-Scientist): ensemble reviewing with a meta-reviewer produces far more calibrated feedback than a single review pass.

Step 1: Generate N independent reviews (N=3-5)

Use different models or temperature settings. Each reviewer sees only the paper, not other reviews. Default to negative bias — LLMs have well-documented positivity bias in evaluation.

You are an expert reviewer for [VENUE]. You are critical and thorough.
If a paper has weaknesses or you are unsure about a claim, flag it clearly
and reflect that in your scores. Do not give the benefit of the doubt.

Review this paper according to the official reviewer guidelines. Evaluate:

1. Soundness (are claims well-supported? are baselines fair and strong?)
2. Clarity (is the paper well-written? could an expert reproduce it?)
3. Significance (does this matter to the community?)
4. Originality (new insights, not just incremental combination?)

Provide your review as structured JSON:
{
  "summary": "2-3 sentence summary",
  "strengths": ["strength 1", "strength 2", ...],
  "weaknesses": ["weakness 1 (most critical)", "weakness 2", ...],
  "questions": ["question for authors 1", ...],
  "missing_references": ["paper that should be cited", ...],
  "soundness": 1-4,
  "presentation": 1-4,
  "contribution": 1-4,
  "overall": 1-10,
  "confidence": 1-5
}

Step 2: Meta-review (Area Chair aggregation)

Feed all N reviews to a meta-reviewer:

You are an Area Chair at [VENUE]. You have received [N] independent reviews
of a paper. Your job is to:

1. Identify consensus strengths and weaknesses across reviewers
2. Resolve disagreements by examining the paper directly
3. Produce a meta-review that represents the aggregate judgment
4. Use AVERAGED numerical scores across all reviews

Be conservative: if reviewers disagree on whether a weakness is serious,
treat it as serious until the authors address it.

Reviews:
[review_1]
[review_2]
...

Step 3: Reflection loop (optional, 2-3 rounds)

Each reviewer can refine their review after seeing the meta-review. Use an early termination sentinel: if the reviewer responds "I am done" (no changes), stop iterating.

Model selection for reviewing: Reviewing is best done with the strongest available model, even if you wrote the paper with a cheaper one. The reviewer model should be chosen independently from the writing model.

Few-shot calibration: If available, include 1-2 real published reviews from the target venue as examples. This dramatically improves score calibration. See references/reviewer-guidelines.md for example reviews.

Step 6.1b: Visual Review Pass (VLM)

Text-only review misses an entire class of problems: figure quality, layout issues, visual consistency. If you have access to a vision-capable model, run a separate visual review on the compiled PDF:

You are reviewing the visual presentation of this research paper PDF.
Check for:
1. Figure quality: Are plots readable? Labels legible? Colors distinguishable?
2. Figure-caption alignment: Does each caption accurately describe its figure?
3. Layout issues: Orphaned section headers, awkward page breaks, figures far from their references
4. Table formatting: Aligned columns, consistent decimal precision, bold for best results
5. Visual consistency: Same color scheme across all figures, consistent font sizes
6. Grayscale readability: Would the figures be understandable if printed in B&W?

For each issue, specify the page number and exact location.

This catches problems that text-based review cannot: a plot with illegible axis labels, a figure placed 3 pages from its first reference, inconsistent color palettes between Figure 2 and Figure 5, or a table that's clearly wider than the column width.

Step 6.1c: Claim Verification Pass

After simulated reviews, run a separate verification pass. This catches factual errors that reviewers might miss:

Claim Verification Protocol:
1. Extract every factual claim from the paper (numbers, comparisons, trends)
2. For each claim, trace it to the specific experiment/result that supports it
3. Verify the number in the paper matches the actual result file
4. Flag any claim without a traceable source as [VERIFY]

For agent-based workflows: delegate verification to a fresh sub-agent that receives only the paper text and the raw result files. The fresh context prevents confirmation bias — the verifier doesn't "remember" what the results were supposed to be.

Step 6.2: Prioritize Feedback

After collecting reviews, categorize:

Step 6.3: Revision Cycle

For each critical/high issue:

1. Identify the specific section(s) affected
2. Draft the fix
3. Verify the fix doesn't break other claims
4. Update the paper
5. Re-check against the reviewer's concern

Step 6.4: Rebuttal Writing

When responding to actual reviews (post-submission), rebuttals are a distinct skill from revision:

Format: Point-by-point. For each reviewer concern:

> R1-W1: "The paper lacks comparison with Method X."

We thank the reviewer for this suggestion. We have added a comparison with 
Method X in Table 3 (revised). Our method outperforms X by 3.2pp on [metric] 
(p<0.05). We note that X requires 2x our compute budget.

Rules:

• Address every concern — reviewers notice if you skip one
• Lead with the strongest responses
• Be concise and direct — reviewers read dozens of rebuttals
• Include new results if you ran experiments during the rebuttal period
• Never be defensive or dismissive, even of weak criticisms
• Use latexdiff to generate a marked-up PDF showing changes (see Professional LaTeX Tooling section)
• Thank reviewers for specific, actionable feedback (not generic praise)

What NOT to do: "We respectfully disagree" without evidence. "This is out of scope" without explanation. Ignoring a weakness by only responding to strengths.

Step 6.5: Paper Evolution Tracking

Save snapshots at key milestones:

paper/
  paper.tex                    # Current working version
  paper_v1_first_draft.tex     # First complete draft
  paper_v2_post_review.tex     # After simulated review
  paper_v3_pre_submission.tex  # Final before submission
  paper_v4_camera_ready.tex    # Post-acceptance final

Phase 7: Submission Preparation

Goal: Final checks, formatting, and submission.

Step 7.1: Conference Checklist

Every venue has mandatory checklists. Complete them carefully — incomplete checklists can result in desk rejection.

See references/checklists.md for:

• NeurIPS 16-item paper checklist
• ICML broader impact + reproducibility
• ICLR LLM disclosure policy
• ACL mandatory limitations section
• Universal pre-submission checklist

Step 7.2: Anonymization Checklist

Double-blind review means reviewer

…(truncated)