Arcdeck Eval

arcdeck-eval

Narrative-Driven Paper-to-Slide Generation via ArcDeck — Ozden et al. (2026) (arXiv:2604.11969, 2026)

What this evaluates

Evaluates the ability of LLM/VLM systems to generate high-quality, narrative-coherent presentation slides from academic papers. It probes content coverage, rhetorical structure preservation, textual fluency, and visual layout quality compared to human-authored references.

Datasets

• ArcBench — total 100; splits: test (100)

Metrics

• VLM-based Q/A Quiz Accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions generated from the source paper and answered using only the generated slides. Calculated as (correct / 25) * 100 across four categories: Story, Visuals, Hard, and Depth.
• VLM-as-Judge Score — range: [1, 100]
  • Score from 1 to 100 per dimension (Text Quality, Narrative Flow, Visual Layout, Visual Thematic), derived from a 10-item checklist where each satisfied criterion contributes to the final score.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated slide text and the source paper, measuring sequence-level content coverage.
• Perplexity — range: other
  • Linguistic fluency score computed using LLaMA-3-8B, representing the model's uncertainty over the generated slide text.
• VLM Pairwise Preference Win Rate — range: percent
  • Percentage of times a generated deck is preferred over a baseline or author-prepared deck by a VLM judge, averaged over 11 randomized runs.

Input / output format

Input: Source paper (text & figures) and optionally generated slide decks or multiple-choice questions. For pairwise tests, two slide decks plus the source paper.

Output: Generated slides in 13.33×7.5-inch layout (HTML/CSS converted to PDF), or multiple-choice answers, or judge scores (1–100), or pairwise preference selections.

Scoring recipe

# VLM Quiz Accuracy
correct = sum(1 for q, a in zip(questions, answers) if a == q.gold)
accuracy = (correct / 25) * 100

# VLM-as-Judge Score
judge_score = sum(1 for criterion in checklist if judge.satisfies(criterion)) * 10

# ROUGE-L
rouge_l = rouge_l_score(generated_text, source_text)

# Pairwise Win Rate
wins = sum(1 for run in range(11) if judge.prefers(deck_A, deck_B, source))
win_rate = (wins / 11) * 100

Common pitfalls

• Single-model judge bias is mitigated by using both closed (GPT-5) and open-source (Qwen3-VL) judges.
• Slide stylistic variations are controlled by evaluating all methods under fixed themes to ensure preference reflects content/structure.
• Quiz questions are generated by one VLM from the source paper and answered by a separate VLM from the slides, introducing potential generation/answering model mismatch.
• Pairwise evaluations are repeated 11 times with randomized order to account for position bias and stochastic judge behavior.

Evidence (verbatim from paper)

We complement the VLM-based evaluations with two standard text metrics reported in Tab.5: ROUGE-L, measuring sequence overlap between the generated slide text and the source paper to quantify content coverage, and Perplexity (computed via LLaMA-3-8B), assessing linguistic fluency of the slide text.

Citation

@misc{ozden2026arcdeck,
  title={Narrative-Driven Paper-to-Slide Generation via ArcDeck},
  author={Ozden et al. (2026)},
  year={2026},
  note={arXiv:2604.11969}
}

• arXiv: 2604.11969