Longbench Write Eval

longbench-write-eval

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs — Bai et al. (2024) (arXiv:2408.07055, 2024)

What this evaluates

Evaluates an LLM's ability to generate ultra-long, coherent, and high-quality text (up to 20k words) while strictly adhering to explicit length constraints. It probes long-context generation capabilities, structural coherence over extended outputs, and instruction-following for length requirements.

Datasets

• LongBench-Write — total ?; splits: test (-1); repo https://github.com/THUDM/LongWriter

Metrics

• Sq (Quality Score) (primary) — range: [0, 100]
  • Overall quality score rated on a 0-100 scale, aggregating sub-dimensions (Relevance, Accuracy, Coherence, Clarity, Breadth and Depth, Reading Experience). Higher values indicate better generation quality.
• Sl (Length Score) — range: [0, 100]
  • Score measuring adherence to the required output length constraint, rated on a 0-100 scale. Evaluated separately across four output length brackets: [0, 500), [500, 2k), [2k, 4k), [4k, 20k).

Input / output format

Input: A writing prompt specifying a topic and a required output length (e.g., 'Write a 20,000-word article about...').

Output: A single continuous text generation of arbitrary length.

Scoring recipe

# For each sample in LongBench-Write test set:
predictions = model.generate(prompt)
gold_length = extract_required_length(prompt)
actual_length = count_words(predictions)

# Quality & Length scores (0-100) are obtained via human/LLM annotation per sub-dimension
sq = compute_quality_score(predictions)
sl = compute_length_adherence_score(predictions, gold_length)

# Aggregate by length bracket
bracket = assign_bracket(actual_length)
record(sq, sl, actual_length, bracket)

# Final metric: mean Sq and Sl per bracket, plus overall mean
overall_sq = mean(sq for all samples)
overall_sl = mean(sl for all samples)

Common pitfalls

• Models frequently truncate outputs well before the required length, especially beyond 4k words.
• Quality scores (Sq) often degrade significantly in longer brackets due to coherence loss, even when length is met.
• Length constraints in prompts are sometimes ignored or misinterpreted, leading to severe under-generation.

Evidence (verbatim from paper)

Table 7: Evaluation results on English samples in LongBench-Write.

Citation

@misc{bai2024longwriter,
  title={LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2408.07055}
}

• arXiv: 2408.07055