qhjqhj00 longbench-v2-eval
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai et al. (2024) (arXiv:2412.15204, 2024)
What this evaluates
Evaluates large language models' ability to comprehend and reason over realistic, extremely long contexts (up to 2M words) across six multitask domains. It probes deep understanding rather than shallow extraction by using challenging multiple-choice questions that require extended reasoning and careful reading.
Datasets
- LongBench v2 — total 503; splits: test (503); repo https://github.com/THUDM/LongBench
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions. Invalid or non-compliant model responses are compensated by assigning a 25% accuracy score (random guessing baseline) instead of being discarded.
Input / output format
Input: Long context document(s) (up to 2M words, truncated to model's max context window if needed) concatenated with a multiple-choice question and its options. For RAG evaluation, the context is split into 512-token chunks and top-N most similar chunks are retrieved.
Output: A single letter/option corresponding to the correct answer. In the CoT setting, the model first generates a chain of thought followed by the final answer choice.
Scoring recipe
def compute_accuracy(predictions, golds):
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
if pred is a valid choice:
if pred == gold:
correct += 1
else:
correct += 0.25 # Compensate invalid responses with random guess baseline
return (correct / total) * 100
Common pitfalls
- Invalid model outputs must be compensated with a 25% score (random baseline) rather than dropped or counted as zero.
- Contexts exceeding the model's context window require middle truncation to ensure fair comparison.
- CoT and zero-shot settings yield different results; mixing evaluation settings without explicit labeling invalidates comparisons.
Evidence (verbatim from paper)
To account for model responses and human responses that do not yield a valid choice, we report the compensated results in Table 5, where these cases are counted towards the accuracy with a random probability of 25%.
Citation
@misc{bai2024longbenchv2,
title={LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks},
author={Yushi Bai et al. (2024)},
year={2024},
note={arXiv:2412.15204}
}
- arXiv: 2412.15204