Longbench Pro Eval

longbench-pro-eval

LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark — Chen et al. (2026) (arXiv:2601.02872, 2026)

What this evaluates

Evaluates long-context understanding and reasoning capabilities of LLMs across bilingual (English/Chinese) tasks. It probes retrieval, ranking, ordering, multiple-choice, information extraction, and summarization under varying difficulty levels and context lengths.

Datasets

• LongBench Pro — total ?; splits: (unstated)

Metrics

• LongBench Pro Score (primary) — range: percent
  • Average of task-specific metric scores across all samples, multiplied by 100. Task-specific metrics include NDCG@k, pairwise accuracy, accuracy, F1, SubEM, or a composite Summary Score (0.5 * max_i SemSim + 0.5 * max_i ROUGE-L). All individual metrics range [0, 1].
• NDCG@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k, used for retrieval and ranking tasks.

Input / output format

Input: Long context documents (English or Chinese) with task-specific instructions. If document length exceeds model context length, it is truncated from the middle to (model context length - output length).

Output: Task-dependent generations: ranked lists, ordered sequences, selected options, extracted text spans, or summaries. Output length is set to 32k for thinking mode and 1k for non-thinking mode.

Scoring recipe

def compute_longbench_pro_score(predictions, golds, task_types):
    task_scores = []
    for pred, gold, task in zip(predictions, golds, task_types):
        if task == "T1": score = ndcg_at_k(pred, gold)
        elif task in ["T2", "T6.3"]: score = pairwise_accuracy(pred, gold)
        elif task in ["T3", "T11"]: score = accuracy(pred, gold)
        elif task in ["T5", "T6.2", "T7", "T9"]: score = f1_score(pred, gold)
        elif task in ["T6.1", "T8", "T10"]: score = subem_score(pred, gold)
        elif task == "T4": 
            scores = [0.5 * semsim(pred, ref) + 0.5 * rouge_l(pred, ref) for ref in gold]
            score = max(scores)
        task_scores.append(score)
    return sum(task_scores) / len(task_scores) * 100

Common pitfalls

• Truncation is applied from the middle of the context when it exceeds the model's context length, which may discard crucial information.
• Thinking and non-thinking scores are reported separately; models must be evaluated under specific prompt modes (thinking vs non-thinking) with different output length constraints (32k vs 1k).
• For summarization tasks, three reference summaries are provided, and the metric takes the maximum score across references rather than averaging them.

Evidence (verbatim from paper)

We use task-specific metrics summarized in Table 2. T1 (Retrieval & Ranking) is evaluated by NDCG@k. T2 and T6.3 (ordering-style tasks) use pairwise accuracy based on rank consistency. T3 and T11 are multiple-choice and use accuracy. For tasks with potentially multiple answer components extracted from the source text (T5, T6.2, T7, and T9), we use F1 to penalize spurious components. For tasks with a single canonical answer that is not directly copied from the source (T6.1, T8, and T10), we use SubEM. For T4 (Summary), we combine semantic similarity (SemSim) and ROUGE-L to balance semantic faithfulness and coverage. Each summarization sample includes three reference summaries. The metrics are first computed between the generated summary and each reference summary individually, and the maximum value for each metric is taken to reflect consistency with the best-matching reference. The final weighted score is calculated as: ... All metrics have a value range of [0,1]. We report the average score over all samples and multiply by 100.

Citation

@misc{chen2026longbenchpro,
  title={LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2601.02872}
}

• arXiv: 2601.02872