Qg Bench Eval

qg-bench-eval

Generative Language Models for Paragraph-Level Question Generation — Ushio et al. (2022) (arXiv:2210.03992, 2022)

What this evaluates

Evaluates the ability of generative language models to produce paragraph-level questions conditioned on a target answer and a context sentence. It probes domain adaptability and multilingual generalization across diverse extractive QA datasets.

Datasets

• SQuAD v1.1 — total 98169; splits: train (75722), val (10570), test (11877)
• SQuADShifts — total ?; splits: train (-1), val (-1), test (-1)
• SubjQA — total ?; splits: train (-1), val (-1), test (-1)
• Multilingual QA (JAQuAD, GerQuAD, SberQuAD, KorQuAD, FQuAD, Spanish SQuAD, Italian SQuAD) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• automatic evaluation metrics (primary) — range: [0, 1]
  • Standard sequence-to-sequence generation metrics (e.g., BLEU, ROUGE, BERTScore) are used to compare generated questions against gold questions. Specific formulas are not detailed in this section.
• manual assessment for answerability and coherence — range: percent
  • Human evaluators rate generated questions on answerability (whether the question can be answered by the context) and coherence (fluency and logical consistency).

Input / output format

Input: paragraph, sentence, answer

Output: question (constrained to be a substring of the provided sentence)

Scoring recipe

# Pseudo-code for evaluation
# 1. Generate question from (paragraph, sentence, answer)
# 2. Compute automatic metrics (e.g., BLEU/ROUGE/BERTScore) against gold question
# 3. Conduct manual evaluation:
#    - Rate answerability: Can the question be answered by the paragraph?
#    - Rate coherence: Is the question fluent and logically sound?
# 4. Aggregate scores across domains and languages

Common pitfalls

• Many multilingual datasets lack official test sets; test splits are sampled from training data with no paragraph overlap.
• Automatic metrics alone are insufficient; manual evaluation is required to assess answerability and coherence.
• Domain-specific and low-resource language settings show significant performance degradation compared to general English domains.

Evidence (verbatim from paper)

The benchmark enables fair comparison of generative language models through fine-tuning on diverse domains and languages, revealing limitations in automatic evaluation metrics and highlighting the importance of manual assessment for answerability and coherence.

Citation

@misc{ushio2022generative,
  title={Generative Language Models for Paragraph-Level Question Generation},
  author={Ushio et al. (2022)},
  year={2022},
  note={arXiv:2210.03992}
}

• arXiv: 2210.03992