qhjqhj00 conll2012-coref-eval
End-to-end Neural Coreference Resolution — Lee et al. (2017) (ACL 2017, 2017)
What this evaluates
Evaluates a model's ability to jointly detect mentions and resolve coreferential chains in English text without relying on external syntactic parsers or hand-crafted features. It measures how well the model groups word spans into entity clusters based on contextual and structural cues.
Datasets
- CoNLL-2012 (English) — total 3493; splits: train (2802), dev (343), test (348)
Metrics
F1(primary) — range: percent- Average F1 score of three cluster-level metrics: MUC, B³, and CEAFφ4. Computed over both mention detection and coreference clustering, following the official CoNLL-2012 evaluation protocol.
Input / output format
Input: Raw English documents (paragraphs of text).
Output: A set of coreference clusters, where each cluster contains a list of mention spans (start, end indices) that refer to the same entity.
Scoring recipe
def compute_f1(pred_clusters, gold_clusters):
muc_f1 = muc_score(pred_clusters, gold_clusters)
b3_f1 = b3_score(pred_clusters, gold_clusters)
ceaf_f1 = ceaf_phi4_score(pred_clusters, gold_clusters)
return (muc_f1 + b3_f1 + ceaf_f1) / 3.0
Common pitfalls
- Aggressive pruning (max span width 10, max antecedents 250) can hurt recall on long documents if not replicated exactly.
- Official CoNLL scorer requires exact mention boundary matching; slight boundary shifts penalize F1 heavily.
- Ensemble averaging of scores vs. majority voting changes results significantly; the paper averages mention and antecedent scores across 5 models.
Evidence (verbatim from paper)
We use the English coreference resolution data from the CoNLL-2012 shared task in our experiments. This dataset contains 2802 training documents, 343 development documents, and 348 test documents. Achieves +1.5 F1 on OntoNotes and +3.1 F1 with a 5-model ensemble by directly optimizing marginal likelihood of gold clusters.
Citation
@misc{lee2017endtoend,
title={End-to-end Neural Coreference Resolution},
author={Lee et al. (2017)},
year={2017},
note={ACL 2017}
}
- arXiv: 1707.07045