Longalign Eval

longalign-eval

LongAlign: A Recipe for Long Context Alignment of Large Language Models — Bai et al. (2024) (arXiv:2401.18058, 2024)

What this evaluates

Evaluates large language models' ability to follow instructions and retrieve information in long-context scenarios (up to 64k tokens), while also measuring their general capabilities and instruction-following performance in short-context settings.

Datasets

• LongBench-Chat — total ?; splits: test (-1)
• LongBench — total ?; splits: test (-1)
• MT-Bench — total ?; splits: test (-1)
• ARC — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• GPT-4 rating (1-10) (primary) — range: other
  • GPT-4 rates the model's output based on alignment with ground truth on a 1-10 scale. Scores are averaged across all test instances. For other datasets, standard metrics (accuracy, F1, ROUGE) are normalized to a 0-100 scale.

Input / output format

Input: Instruction prompts paired with long or short context documents/texts.

Output: Model-generated text responses.

Scoring recipe

def compute_metric(predictions, gold, dataset_type):
    if dataset_type in ['LongBench-Chat', 'MT-Bench']:
        scores = [gpt4_rate(pred, gold) for pred, gold in zip(predictions, gold)]
        return sum(scores) / len(scores)
    else:
        raw = compute_standard_metric(predictions, gold)
        return normalize_0_100(raw)

Common pitfalls

• Using ROUGE/F1 for LongBench instead of GPT-4, as the paper explicitly notes aligned models produce longer responses that penalize traditional metrics.
• Not averaging multiple GPT-4 ratings; the protocol specifies scoring twice and averaging for stability.
• Confusing LongBench-Chat (instruction-following) with LongBench (general QA/summarization), which use different evaluation focuses.

Evidence (verbatim from paper)

For LongBench-Chat and MT-Bench, the reported results are averaged over GPT-4's rating (1-10) across all test instances, while results on other datasets are normalized between 0-100.

Citation

@misc{bai2024longalign,
  title={LongAlign: A Recipe for Long Context Alignment of Large Language Models},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2401.18058}
}

• arXiv: 2401.18058