Longgenbench Eval

longgenbench-eval

LongGenBench: Long-context Generation Benchmark — Liu et al. (2024) (arXiv:2410.04199, 2024)

What this evaluates

Evaluates the ability of LLMs to maintain accuracy and logical consistency when generating long-text responses that answer multiple sequential questions from GSM8K or MMLU in a single pass. It specifically probes performance degradation as the number of generated questions increases.

Datasets

• LongGenBench-GSM8K — total ?; splits: test (-1); repo https://github.com/Dominic789654/LongGenBench
• LongGenBench-MMLU — total ?; splits: test (-1); repo https://github.com/Dominic789654/LongGenBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total K questions in the long-context prompt. Calculated as (correct_count / K) * 100.
• delta — range: percent
  • Performance degradation calculated as Baseline Accuracy minus LongGenBench Accuracy. Negative values indicate a drop in performance.

Input / output format

Input: A long-context prompt containing K sequential questions from GSM8K or MMLU, requiring the model to generate a single continuous text response answering all questions.

Output: A single generated text response containing answers to all K questions in sequence.

Scoring recipe

def compute_accuracy(predictions, gold, k):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / k) * 100

Common pitfalls

• Confusing baseline accuracy (single-question evaluation) with LongGenBench accuracy (multi-question sequential generation).
• Assuming accuracy degradation is linear across question indices; the paper shows non-linear drops dependent on model architecture and size.
• Overlooking that the model must generate all K answers in a single pass without intermediate stopping or re-prompting.

Evidence (verbatim from paper)

Figure [3] shows the accuracy distribution of API accessed models in LongGenBench-GSM8K. The x-axis represents the question index within a single long-text response, with the maximum index being $K$. The y-axis indicates the accuracy of the model’s responses to these questions.

Citation

@misc{liu2024longgenbench,
  title={LongGenBench: Long-context Generation Benchmark},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.04199}
}

• arXiv: 2410.04199