Text2distbench Eval

text2distbench-eval

Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models — Guo et al. (2026) (arXiv:2604.06201, 2026)

What this evaluates

Evaluates large language models' ability to infer population-level statistics (e.g., sentiment proportions, topic frequencies) from aggregated natural language text. It probes marginal, conditional, and joint distribution estimation over discrete categories derived from real-world comments.

Datasets

• Text2DistBench — total 540; splits: test (540); repo https://github.com/0Frett/Text2DistBench

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy: 1 if the model's generated answer exactly matches the ground truth answer, 0 otherwise. Averaged across all 540 questions.

Input / output format

Input: A zero-shot prompt containing definitions of sentiment and topic, entity metadata, a randomly sampled set of 50 associated comments, and a distributional question (marginal, conditional, or joint).

Output: A natural language answer to the distributional question, generated in text-only mode without tool calling or retrieval.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(gold_answers)

Common pitfalls

• Models may rely on pretraining knowledge rather than the provided comments, violating the zero-shot/no-retrieval constraint.
• The benchmark uses a single sample per question with default decoding parameters, so results may vary significantly with temperature or sampling settings.
• Distributional questions require aggregating information across 50 comments, which can exceed context windows or lead to attention dilution.

Evidence (verbatim from paper)

resulting in a total of 540 evaluation questions across the three tasks. For each entity, we randomly sample 50 comments from the associated videos to balance question token length. Each question is sampled once using the model’s default decoding parameters, including temperature and maximum token limits.

Citation

@misc{guo2026beyondfacts,
  title={Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.06201}
}

• arXiv: 2604.06201