Qgeval Eval

qgeval-eval

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation — Fu et al. (2024) (arXiv:2406.05707, 2024)

What this evaluates

Evaluates the quality of generated questions across seven dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. It measures how well automatic metrics and LLMs align with human judgments on these dimensions.

Datasets

• QGEval — total 3000; splits: test (3000); repo https://github.com/WeipingFu/QGEval

Metrics

• Pearson correlation (primary) — range: [-1, 1]
  • Calculates the Pearson product-moment correlation coefficient between the vector of human annotation scores and the vector of automatic/LLM scores for each of the seven evaluation dimensions. Higher absolute values indicate stronger linear alignment with human judgments.

Input / output format

Input: Passage, ground-truth answer, and generated question. For reference-based metrics, a reference question is also provided. Human annotation scores (1–5 Likert scale) per dimension, averaged over three annotators, serve as the gold standard.

Output: Per-dimension scores (1–5 scale) for each of the seven dimensions. Automatic metrics output a single composite score or per-dimension scores depending on the metric type.

Scoring recipe

for dim in ['fluency', 'clarity', 'conciseness', 'relevance', 'consistency', 'answerability', 'answer consistency']:
    human_scores = [get_human_score(q, dim) for q in questions]
    auto_scores = [get_auto_score(q, dim) for q in questions]
    pearson_r = pearsonr(human_scores, auto_scores)
    print(f'{dim}: {pearson_r:.3f}')

Common pitfalls

• Ceiling effects in human scores for fluency, clarity, relevance, and consistency cause automatic metrics to show poor alignment despite high human ratings.
• Reference-based vs reference-free scoring types drastically change scores for metrics like BARTScore and GPTScore; the paper explicitly distinguishes ref-hypo and src-hypo configurations.
• The benchmark shows limited discriminative power among top-performing models on most dimensions, making it difficult to rank state-of-the-art QG models using standard t-tests.

Evidence (verbatim from paper)

We evaluate the agreement between the automatic metrics and human annotation scores by calculating the Pearson correlation over each dimension, results are shown in Table[5], with the three highest and lowest absolute coefficients bolded and underlined respectively.

Citation

@misc{fu2024qgeval,
  title={QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation},
  author={Fu et al. (2024)},
  year={2024},
  note={arXiv:2406.05707}
}

• arXiv: 2406.05707