Duorc Eval

duorc-eval

Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC — Quijano et al. (2021) (arXiv:2101.06326, 2021)

What this evaluates

Evaluates reading comprehension and long-form text understanding by asking models to answer questions about movie plots. It specifically probes sensitivity to narrative length and semantic shifts between short and paraphrased long versions of the same story.

Datasets

• DuoRC — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score between predicted and gold answers. Computed as 2 * (precision * recall) / (precision + recall).
• Accuracy — range: [0, 1]
  • Exact-match accuracy: 1 if the predicted answer exactly matches the gold answer, 0 otherwise.

Input / output format

Input: Context (movie plot, either short or long paraphrased version) and a question.

Output: Predicted answer text or span.

Scoring recipe

def compute_metrics(predictions, golds):
    f1_scores = []
    exact_matches = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = set(normalize(pred))
        gold_tokens = set(normalize(gold))
        if not gold_tokens:
            f1_scores.append(0.0)
            exact_matches.append(0.0)
            continue
        intersection = pred_tokens & gold_tokens
        precision = len(intersection) / len(pred_tokens) if pred_tokens else 0
        recall = len(intersection) / len(gold_tokens)
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
        exact_matches.append(1.0 if pred == gold else 0.0)
    return {'F1': sum(f1_scores) / len(f1_scores), 'Accuracy': sum(exact_matches) / len(exact_matches)}

Common pitfalls

• Effective batch size is tbs × number of GPUs, not just tbs.
• Hyperparameters are grid-searched per dataset subset, so optimal settings do not transfer between SelfRC and ParaphraseRC.
• Model selection is based on validation F1, but final results are reported on the test set.

Evidence (verbatim from paper)

Table 2 shows that the best model is the ALBERT model pretrained with the SQuAD2 dataset. The best F1 score is 76.4 and the accuracy score is 68.52 using the validation set. The results also indicate that increasing the hyperparameters improves the performance of the models.

Citation

@misc{quijano2021gridsearch,
  title={Grid Search Hyperparameter Benchmarking of BERT, ALBERT, and LongFormer on DuoRC},
  author={Quijano et al. (2021)},
  year={2021},
  note={arXiv:2101.06326}
}

• arXiv: 2101.06326