Freshwiki Article Eval

freshwiki-article-eval

Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models — Shao et al. (2024) (arXiv:2402.14207, 2024)

What this evaluates

Evaluates the ability of LLMs to generate comprehensive, well-organized, and verifiable Wikipedia-like articles from a given topic. It probes outline planning, factual coverage, structural coherence, and source grounding.

Datasets

• FreshWiki — total 100; splits: test (100)

Metrics

• Heading soft recall — range: [0, 100] percent
  • Calculates the proportion of human-written article headings that are semantically matched in the generated outline.
• Heading entity recall — range: [0, 100] percent
  • Measures the proportion of named entities present in human headings that are also present in the generated outline headings.
• ROUGE-1 (primary) — range: [0, 100] percent
  • Standard unigram overlap between the generated article and the human-written reference text.
• ROUGE-L — range: [0, 100] percent
  • Longest common subsequence overlap between the generated article and the human reference.
• Entity Recall — range: [0, 100] percent
  • Proportion of named entities in the human article correctly recalled in the generated article, computed using FLAIR NER.
• Rubric Scores (Interest Level, Coherence/Organization, Relevance/Focus, Coverage) — range: [1, 5]
  • 1-5 point scores assigned by a 13B Prometheus LLM evaluator based on a rubric developed with Wikipedia editors.
• Verifiability (Citation Recall/Precision) — range: [0, 100] percent
  • Assesses whether cited passages entail the generated sentences, evaluated using Mistral 7B-Instruct.

Input / output format

Input: A topic name (from FreshWiki dataset). The model generates an outline first, then a full-length article with citations.

Output: Generated outline (list of headings) and a full-length article (≤4000 tokens) with inline citations referencing retrieved passages.

Scoring recipe

def score_instance(pred_outline, pred_article, gold_article):
    heading_soft_recall = compute_semantic_match(pred_outline.headings, gold_article.headings)
    heading_entity_recall = compute_entity_overlap(pred_outline.headings, gold_article.headings)
    rouge1 = rouge_score(pred_article.text, gold_article.text, rouge_type='1')
    rougeL = rouge_score(pred_article.text, gold_article.text, rouge_type='L')
    entity_recall = compute_ner_recall(pred_article.text, gold_article.text, ner_model='FLAIR')
    rubric_scores = prometheus_13b.evaluate(pred_article.text, gold_article.text, rubric=['Interest', 'Coherence', 'Relevance', 'Coverage'])
    verifiability = mistral_7b.check_entailment(pred_article.citations, pred_article.sentences)
    return {
        'heading_soft_recall': heading_soft_recall,
        'heading_entity_recall': heading_entity_recall,
        'ROUGE-1': rouge1, 'ROUGE-L': rougeL,
        'Entity Recall': entity_recall,
        'Rubric': rubric_scores,
        'Verifiability': verifiability
    }

Common pitfalls

• The evaluation explicitly limits output to 4000 tokens (~3000 words), which may truncate longer topics and artificially lower coverage/recall scores.
• The ground truth Wikipedia article is excluded from the search results during generation, so metrics reflect retrieval-augmented generation without data leakage.
• LLM-based evaluators (Prometheus, Mistral) are used for rubric and verifiability scores, which may introduce bias or inconsistency compared to human judges.

Evidence (verbatim from paper)

As discussed in §2.2, we evaluate the outline quality to assess the pre-writing stage by calculating the heading soft recall and heading entity recall. A higher recall score signifies a more comprehensive outline relative to the human-written article. To assess the full-length article quality, we adopt ROUGE scores (Lin, 2004) and compute the entity recall in the article level based on FLAIR NER results. Moreover, based on Wikipedia criteria, we evaluate the article from the aspects of (1) Interest Level, (2) Coherence and Organization, (3) Relevance and Focus, (4) Coverage, and (5) Verifiability.

Citation

@misc{shao2024storm,
  title={Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models},
  author={Shao et al. (2024)},
  year={2024},
  note={arXiv:2402.14207}
}

• arXiv: 2402.14207