Scholawrite Eval

scholawrite-eval

ScholaWrite: A Dataset of End-to-End Scholarly Writing Process — Linghe Wang et al. (arXiv:2502.02904, 2025)

What this evaluates

Evaluates an LLM's ability to predict human scholarly writing intentions from a LaTeX draft and to iteratively edit the draft according to those intentions. It measures lexical diversity, topic consistency, and intention coverage across a 100-iteration self-writing process.

Datasets

• SCHOLAWRITE — total ?; splits: train (-1), test (-1)

Metrics

• lexical diversity — range: [0, 1]
  • Number of unique tokens divided by total tokens generated in the final iteration.
• topic consistency — range: other
  • Cosine similarity between the embedding of the seed document and the embedding of the output from the final iteration.
• intention coverage (primary) — range: [0, 1]
  • Number of unique intention labels predicted across the entire 100 iterations divided by the 15 available labels in the taxonomy.

Input / output format

Input: LaTeX draft text ('before_text') concatenated with task instructions and intention label definitions/verbalizers.

Output: For prediction: a single intention label from the 15-class taxonomy. For generation: revised LaTeX text (plain draft without delimiters, or token-wrapped diffs for training).

Scoring recipe

def score_lexical_diversity(final_output):
    tokens = final_output.split()
    return len(set(tokens)) / len(tokens) if tokens else 0.0

def score_topic_consistency(seed_doc, final_output):
    return cosine_similarity(embed(seed_doc), embed(final_output))

def score_intention_coverage(predicted_labels):
    return len(set(predicted_labels)) / 15

Common pitfalls

• Test set sampling is limited to 300 keystroke entries per intention label due to budget constraints, not the full 20% split.
• GPT-4o baselines advance iterations regardless of intention prediction matches, while Llama models wait for a different intention, creating an unfair comparison in the iterative loop.
• Metrics are computed only on the final iteration's output (lexical diversity, topic consistency) or over the full 100 iterations (intention coverage).

Evidence (verbatim from paper)

• Lexical diversity: Assess the unique tokens model generated in the final iteration of writing, measured by the number of unique tokens divided by the total tokens generated. • Topic consistency: Cosine similarity between the seed document and output from the final iteration of writing. • Intention coverage: Assess the diversity of the model’s writing intention, measured by the number of unique labels predicted through the entire 100 iterations divided by all 15 intended labels available in our taxonomy.

Citation

@misc{wang2025scholawrite,
  title={ScholaWrite: A Dataset of End-to-End Scholarly Writing Process},
  author={Linghe Wang et al.},
  year={2025},
  note={arXiv:2502.02904}
}

• arXiv: 2502.02904