# Longbench Write Eval > Evaluates an LLM's ability to generate ultra-long, coherent, and high-quality text (up to 20k words) while strictly adhering to explicit length constraints. It probes long-context generation capabilities, structural coherence over extended outputs, and instruction-following for length requirements. Use when the user wants to benchmark on LongBench-Write, or asks about evaluating this task. Reports Sq (Quality Score). - Skill: `qhjqhj00/longbench-write-eval` (Agent Skill) - Install (CLI): `npx skillmds add qhjqhj00/longbench-write-eval` - Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/longbench-write-eval/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/qhjqhj00/longbench-write-eval --- # longbench-write-eval > LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs — Bai et al. (2024) (arXiv:2408.07055, 2024) ## What this evaluates Evaluates an LLM's ability to generate ultra-long, coherent, and high-quality text (up to 20k words) while strictly adhering to explicit length constraints. It probes long-context generation capabilities, structural coherence over extended outputs, and instruction-following for length requirements. ## Datasets - **LongBench-Write** — total ?; splits: test (-1); repo https://github.com/THUDM/LongWriter ## Metrics - `Sq (Quality Score)` **(primary)** — range: [0, 100] - Overall quality score rated on a 0-100 scale, aggregating sub-dimensions (Relevance, Accuracy, Coherence, Clarity, Breadth and Depth, Reading Experience). Higher values indicate better generation quality. - `Sl (Length Score)` — range: [0, 100] - Score measuring adherence to the required output length constraint, rated on a 0-100 scale. Evaluated separately across four output length brackets: [0, 500), [500, 2k), [2k, 4k), [4k, 20k). ## Input / output format **Input**: A writing prompt specifying a topic and a required output length (e.g., 'Write a 20,000-word article about...'). **Output**: A single continuous text generation of arbitrary length. ## Scoring recipe ```python # For each sample in LongBench-Write test set: predictions = model.generate(prompt) gold_length = extract_required_length(prompt) actual_length = count_words(predictions) # Quality & Length scores (0-100) are obtained via human/LLM annotation per sub-dimension sq = compute_quality_score(predictions) sl = compute_length_adherence_score(predictions, gold_length) # Aggregate by length bracket bracket = assign_bracket(actual_length) record(sq, sl, actual_length, bracket) # Final metric: mean Sq and Sl per bracket, plus overall mean overall_sq = mean(sq for all samples) overall_sl = mean(sl for all samples) ``` ## Common pitfalls - Models frequently truncate outputs well before the required length, especially beyond 4k words. - Quality scores (Sq) often degrade significantly in longer brackets due to coherence loss, even when length is met. - Length constraints in prompts are sometimes ignored or misinterpreted, leading to severe under-generation. ## Evidence (verbatim from paper) > Table 7: Evaluation results on English samples in LongBench-Write. ## Citation ```bibtex @misc{bai2024longwriter, title={LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs}, author={Bai et al. (2024)}, year={2024}, note={arXiv:2408.07055} } ``` - arXiv: 2408.07055
Overall[0, 500)[500, 2k)[2k, 4k)[4k, 20k)
S̅SlSqSlSqSlSqSlSqSlSq