Longbench Eval

longbench-eval

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding — Bai et al. (2023) (arXiv:2308.14508, 2023)

What this evaluates

Evaluates large language models' ability to understand and process long contexts across bilingual (English and Chinese) multitask scenarios, including single/multi-document QA, summarization, few-shot learning, code completion, and synthetic tasks.

Datasets

• LongBench — total ?; splits: test (-1); repo https://github.com/THUDM/LongBench

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall over token overlap between prediction and gold answer. Used primarily for QA and few-shot tasks.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence (LCS) between the generated summary and the reference summary, capturing both precision and recall.
• Edit Sim — range: [0, 1]
  • Normalized Levenshtein distance measuring the minimum number of single-character edits (insertions, deletions, substitutions) required to change one code snippet into another.

Input / output format

Input: Long context document(s) concatenated with an instruction or question. If input length L exceeds the model's maximum context length M, the sequence is truncated from the middle: [S_1:floor(M/2); S_L-floor(M/2)-1:L]. Few-shot tasks include examples within the context. Chat-style prompts are explicitly omitted for few-shot and code completion tasks.

Output: Free-form text generation using greedy decoding. For few-shot learning and code completion tasks, only the first line of the model's response is extracted for evaluation.

Scoring recipe

def evaluate(predictions, golds, task_types):
    scores = []
    for pred, gold, task in zip(predictions, golds, task_types):
        if task in ['fewshot', 'code']:
            pred = pred.split('\n')[0]  # Extract first line
        if task == 'qa':
            scores.append(f1_score(pred, gold))
        elif task == 'summarization':
            scores.append(rouge_l_score(pred, gold))
        elif task == 'code':
            scores.append(edit_similarity(pred, gold))
        else:
            scores.append(exact_match(pred, gold))
    return sum(scores) / len(scores)

Common pitfalls

• Middle truncation is used when context exceeds model limits, which differs from standard prefix truncation and can discard crucial middle information.
• Chat-style prompts are deliberately avoided for few-shot and code tasks to force completion-style generation; applying standard chat templates will invalidate scores.
• Memorization confounds results on Wikipedia-derived datasets; readers must compare against the 'w/o context' baseline to isolate true long-context understanding.

Evidence (verbatim from paper)

F1 and ROUGE-L (Lin, 2004) are two popular N-gram based metrics widely adopted in QA and summarization tasks. Edit Sim (Levenshtein distance) is popularly used in code generation evaluation (Svyatkovskiy et al., 2020). For the few-shot learning tasks, we extract the first line of the response. For the two code completion tasks, we extract the first line of model generation that is not comment.

Citation

@misc{bai2023longbench,
  title={LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding},
  author={Bai et al. (2023)},
  year={2023},
  note={arXiv:2308.14508}
}

• arXiv: 2308.14508