Mixqg Qg Eval

mixqg-qg-eval

MixQG: Neural Question Generation with Mixed Answer Types — Murakhovs'ka et al. (2021) (arXiv:2110.08175, 2021)

What this evaluates

Evaluates a neural question generation model's ability to produce fluent, relevant questions conditioned on a target answer and context. It probes both automatic n-gram/embedding-based similarity metrics and human-rated utility for educational quiz creation.

Datasets

• SQuAD — total ?; splits: train (-1), dev (-1)
• NQ — total ?; splits: train (-1), dev (-1)
• Quoref — total ?; splits: train (-1), test (-1)
• DROP — total ?; splits: train (-1), dev (-1)

Metrics

• question approval rate (primary) — range: percent
  • Percentage of generated questions approved by human teachers for inclusion in a quiz, calculated as approved_count / total_generated_questions.
• BLEU — range: [0, 1]
  • Standard n-gram precision metric with brevity penalty, measuring lexical overlap between generated and reference questions.
• ROUGE — range: [0, 1]
  • Recall-oriented n-gram overlap metric (typically ROUGE-L or ROUGE-1/2), measuring how much reference text is covered by the generation.
• METEOR — range: [0, 1]
  • Metric that aligns generated and reference words using synonymy and stemming, combining precision and recall with a penalty for fragmentation.
• BERTScore — range: [0, 1]
  • Computes cosine similarity between contextual embeddings of generated and reference tokens, then aggregates precision, recall, and F1 scores.

Input / output format

Input: Context (e.g., Wikipedia article or sentence) and a target answer (quiz concept or answer span).

Output: A single generated question string.

Scoring recipe

def score(predictions, golds, human_approvals=None):
    scores = {}
    scores['BLEU'] = compute_bleu(predictions, golds)
    scores['ROUGE'] = compute_rouge(predictions, golds)
    scores['METEOR'] = compute_meteor(predictions, golds)
    scores['BERTScore'] = compute_bertscore(predictions, golds)
    if human_approvals is not None:
        approved = sum(1 for a in human_approvals if a)
        scores['question approval rate'] = (approved / len(predictions)) * 100
    return scores

Common pitfalls

• n-gram based metrics (BLEU/ROUGE) may not correlate well with human judgments of question quality and fluency.
• Models trained solely on a target dataset can outperform joint-trained models in zero-shot settings, but fine-tuning the joint model yields the best overall performance.
• Human evaluation requires teachers to explicitly select a quiz concept (target answer) from the context to ensure generated questions match the intended answer specificity.

Evidence (verbatim from paper)

We report the commonly-used metrics applied in the QG research: BLEU, ROUGE, and METEOR scores. We also report BERTScore, which relies on contextual embeddings to produce the final score. The success of a QG model depends on its question approval rate. We compare seven QG models and collect 3,164 human-annotated samples from 10 recruited teachers. Teachers can then approve a generated question to be included on the quiz or reject it and provide a reason for rejection.

Citation

@misc{murakhovska2021mixqg,
  title={MixQG: Neural Question Generation with Mixed Answer Types},
  author={Murakhovs'ka et al. (2021)},
  year={2021},
  note={arXiv:2110.08175}
}

• arXiv: 2110.08175