Literaryqa Eval

literaryqa-eval

LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA — Bonomo et al. (2025) (arXiv:2510.13494, 2025)

What this evaluates

Evaluates long-context language models on their ability to answer complex, abstractive questions about entire literary works. It probes narrative event understanding and semantic correctness, measuring how well generated answers align with human judgment rather than just matching reference strings.

Datasets

• LiteraryQA — total ?; splits: test (-1); repo https://github.com/sapienzaNLP/literaryQA

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Longest common subsequence between reference and candidate, normalized by reference length.
• METEOR — range: [0, 1]
  • F-measure based on unigram matching, incorporating stemming and synonymy.
• F1 — range: [0, 1]
  • Token-level F1 score (harmonic mean of precision and recall).
• EM — range: [0, 1]
  • Exact match: 1 if candidate exactly equals reference, 0 otherwise.
• BERTScore — range: [0, 1]
  • Semantic similarity score computed using contextual embeddings from a DeBERTa-XLarge model finetuned for NLI.
• LLM-as-a-judge — range: other
  • Score generated by an LLM (e.g., Prometheus 2 7B, GPT-4.1, Claude 3.7 Sonnet) following a system prompt and rubric, typically on a 1–5 scale.

Input / output format

Input: Question, reference answers, and the full book context (or book summary for LLM-as-a-judge evaluation).

Output: Numerical score (0–1) for ROUGE-L, METEOR, F1, BERTScore; binary flag for EM; or rubric-based score (1–5) for LLM-as-a-judge.

Scoring recipe

def compute_metrics(predictions, references):
    scores = {}
    for pred, ref in zip(predictions, references):
        scores['ROUGE-L'] = rouge_l_f1(pred, ref)
        scores['METEOR'] = meteor_score(pred, ref)
        scores['F1'] = token_level_f1(pred, ref)
        scores['EM'] = 1.0 if pred == ref else 0.0
        scores['BERTScore'] = bertscore_similarity(pred, ref, model='DeBERTa-XLarge')
    return scores

Common pitfalls

• n-gram metrics (ROUGE-L, F1, EM) are highly sensitive to length mismatches and reference noise, yielding poor correlation with human judgment.
• LLM-as-a-judge scores vary drastically depending on whether reference answers or book summaries are provided as context.
• System-level correlation (Kendall’s τ) is reported instead of per-instance accuracy, which can obscure individual question failures.

Evidence (verbatim from paper)

We include metrics that have been used in literature to evaluate answers on NarrativeQA, namely: ROUGE-L*(Lin, [2004]), METEOR(Banerjee and Lavie, [2005]), token-level F1 (F1) and exact-match (EM) taken from extractive QA(Yang et al., [2018]). As our neural-based metric, we use BERTScore(Zhang et al., [2020])*, which provides a score between 0 and 1 that represents the semantical similarity of two pieces of text.

Citation

@misc{bonomo2025literaryqa,
  title={LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA},
  author={Bonomo et al. (2025)},
  year={2025},
  note={arXiv:2510.13494}
}

• arXiv: 2510.13494