Squad Eval

squad-eval

Confidence Regularized Masked Language Modeling using Text Length — Ji et al. (2025) (arXiv:2504.06037, 2025)

What this evaluates

Measures a model's ability to extract precise answer spans from a given context paragraph in response to a natural language question, testing reading comprehension and span prediction.

Datasets

• SQuAD 1.1/2.0 — total ?; splits: test (-1); HF squad

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score between predicted answer span and ground truth answer span, calculated as the harmonic mean of precision and recall across all examples.

Input / output format

Input: Question and context paragraph tokenized together, truncated to 512 tokens.

Output: Predicted start and end token indices for the answer span.

Scoring recipe

def compute_f1(pred_spans, gold_spans):
    precisions, recalls = [], []
    for pred, gold in zip(pred_spans, gold_spans):
        p = len(set(pred) & set(gold)) / max(len(pred), 1)
        r = len(set(pred) & set(gold)) / max(len(gold), 1)
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        precisions.append(p)
        recalls.append(r)
    return sum(precisions) / len(precisions), sum(recalls) / len(recalls)

Common pitfalls

• SQuAD 2.0 includes unanswerable questions, requiring a null prediction strategy that affects F1 calculation.
• Not averaging results over multiple random seeds as the paper reports averages of 7 trials.

Evidence (verbatim from paper)

We evaluated methods on the GLUE benchmark (Wang et al., 2019) and SQuAD 1.1/2.0 datasets (Rajpurkar et al., 2016, 2018). Following Devlin et al. (2019), we excluded WNLI from tasks of GLUE benchmark. We reported Matthew's correlation score for CoLA, Pearson correlations for STS-b, F1 score for SQuAD 1.1/2.0, and accuracy scores for the other tasks.

Citation

@misc{ji2025confidenceregularized,
  title={Confidence Regularized Masked Language Modeling using Text Length},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.06037}
}

• arXiv: 2504.06037