Concurrence Eval

concurrence-eval

Do Question Answering Modeling Improvements Hold Across Benchmarks? — Liu et al. (2021) (arXiv:2102.01065, 2021)

What this evaluates

Measures how consistently a modeling approach's performance ranking holds across different question answering benchmarks. It probes whether improvements in QA models generalize across datasets with varying data collection procedures, passage/question distributions, and targeted linguistic phenomena.

Datasets

• SQuAD — total ?; splits: test (-1)
• NewsQA — total ?; splits: test (-1)
• NaturalQuestions — total ?; splits: test (-1)
• DROP — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• QAMR — total ?; splits: test (-1)

Metrics

• concurrence (Spearman's τ) (primary) — range: [-1, 1]
  • Spearman's rank correlation coefficient computed over the performance scores of a fixed set of models evaluated on two different benchmarks.

Input / output format

Input: Performance scores of a fixed set of QA models on two distinct benchmarks.

Output: A single Spearman's rank correlation coefficient (τ) quantifying the agreement between the two benchmarks' model rankings.

Scoring recipe

def compute_concurrence(scores_A, scores_B):
    models = list(scores_A.keys())
    assert models == list(scores_B.keys())
    vals_A = [scores_A[m] for m in models]
    vals_B = [scores_B[m] for m in models]
    ranks_A = rankdata(vals_A)
    ranks_B = rankdata(vals_B)
    return spearmanr(ranks_A, ranks_B).correlation

Common pitfalls

• Concurrence evaluates the correlation of model rankings, not absolute score differences or dataset overlap.
• Requires the exact same model suite to be evaluated on both benchmarks; missing models break the correlation calculation.
• The paper uses MRQA-formatted versions of NewsQA, NQ, DROP, and HotpotQA, which differ from their original releases (e.g., DROP subset excludes non-extractive answers).

Evidence (verbatim from paper)

We study the concurrence between six human-constructed benchmarks: SQuAD, NewsQA, NaturalQuestions, DROP (Dua et al., 2019), HotpotQA (Yang et al., 2018), and QAMR (Michael et al., 2018). Despite differences in benchmark crowdsourcing setups, passage and questions distributions, and even linguistic phenomena of interest, modeling improvements generally hold across human-constructed benchmarks (Table 1).

Citation

@misc{liu2021concurrence,
  title={Do Question Answering Modeling Improvements Hold Across Benchmarks?},
  author={Liu et al. (2021)},
  year={2021},
  note={arXiv:2102.01065}
}

• arXiv: 2102.01065