Qags Questeval Eval

qags-questeval-eval

QFS-Composer: Query-focused summarization pipeline for less resourced languages — Đuranović et al. (2026) (arXiv:2604.10687, 2026)

What this evaluates

Evaluates the factual consistency and information coverage of query-focused summaries for less-resourced languages without reference texts. It probes whether LLMs can preserve source details and align with user intent by comparing answers generated from the source versus answers generated from the candidate summary.

Datasets

• Slovene News Summarization Corpus (MOCHA translation) — total 21; splits: test (21)

Metrics

• Exact match (EM) — range: [0, 1]
  • Binary score of 1 if source-conditioned and summary-conditioned answers are identical strings, 0 otherwise.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall computed from overlapping unigrams between source-conditioned answers and summary-conditioned answers.
• Edit Distance — range: other
  • Levenshtein distance counting character substitutions, insertions, and deletions needed to transform one answer into the other.
• BERTScore F1 — range: [0, 1]
  • F1 score computed from contextual embeddings of source-conditioned and summary-conditioned answers to capture semantic similarity.
• QuestEval F1 (primary) — range: [0, 1]
  • F1 score combining precision (consistency of answers when conditioned on summary vs source) and recall (coverage of important source information weighted by a pre-trained Weighter component and adjusted for answerability confidence).

Input / output format

Input: Source document text, user query, and optionally an augmented prompt containing decomposed sub-questions or extracted named entities.

Output: A single abstractive summary text generated by the LLM.

Scoring recipe

def compute_qags_f1(source_doc, summary, qa_model, qg_model):
    entities = extract_entities(summary)
    questions = [qg_model.generate(q, context=summary) for q in entities]
    ans_source = [qa_model.answer(q, context=source_doc) for q in questions]
    ans_summary = [qa_model.answer(q, context=summary) for q in questions]
    scores = [unigram_f1(a_s, a_sum) for a_s, a_sum in zip(ans_source, ans_summary)]
    return sum(scores) / len(scores)

Common pitfalls

• QuestEval heavily penalizes paraphrased but factually correct summaries due to its reliance on exact answer matches.
• The evaluation caps the number of questions at 10 for computational reasons, which may not capture full document coverage.
• QAGS does not penalize summary length, so higher scores can reflect verbosity rather than true information density.
• QA model performance degrades on long documents due to chunking strategies that may misrank relevant text segments.

Evidence (verbatim from paper)

The core of QA-based methods is to measure the similarity between answers generated using the entire original document as context and answers producing using the candidate summary as only context. In this work, the similarity between answers is measured using four different metrics: Exact match score (EM), $F_{1}$-score, Edit Distance and BERTScore $F_{1}$.

Citation

@misc{duranovic2026qfscomposer,
  title={QFS-Composer: Query-focused summarization pipeline for less resourced languages},
  author={Đuranović et al. (2026)},
  year={2026},
  note={arXiv:2604.10687}
}

• arXiv: 2604.10687