Docred Eval

docred-eval

DocRED: A Large-Scale Document-Level Relation Extraction Dataset — Yao et al. (2019) (arXiv:1906.06127, 2019)

What this evaluates

Evaluates document-level relation extraction systems on multi-sentence reasoning, entity coreference resolution, and long-range dependency modeling. It measures how well models can predict relational facts between entity pairs across entire documents, including cases requiring evidence from multiple sentences.

Datasets

• DocRED — total 5053; splits: train (-1), dev (-1), test (-1); repo https://github.com/thunlp/DocRED

Metrics

• F1 (primary) — range: percent
  • Macro-averaged F1 score for multi-label relation prediction across all entity pairs in a document.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve for multi-label relation prediction.
• Ign F1 — range: percent
  • F1 score computed after excluding relational facts that overlap between the training and dev/test sets to mitigate evaluation bias from memorization.
• Ign AUC — range: [0, 1]
  • AUC score computed after excluding relational facts that overlap between the training and dev/test sets.

Input / output format

Input: A document consisting of n words, with word-level features (GloVe embeddings, entity type embeddings, coreference embeddings), named entity mentions, and entity pair definitions with relative distances.

Output: For each entity pair (e_i, e_j), a probability score for each relation type r, treated as a multi-label classification problem.

Scoring recipe

tp, fp, fn = 0, 0, 0
for (e_i, e_j), gold_rels in gold.items():
    for r in gold_rels:
        if r in preds[(e_i, e_j)] and preds[(e_i, e_j)][r] > 0.5:
            tp += 1
        else:
            fn += 1
    for r in preds[(e_i, e_j)]:
        if r not in gold_rels:
            fp += 1
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Models may memorize relational facts that appear in both training and dev/test sets, inflating standard F1/AUC scores. The paper mandates reporting Ign F1/AUC to correct this bias.
• Precision cannot be reliably computed for the 'Performance vs. Supporting Evidence Types' breakdown because wrong predictions cannot be classified into evidence subsets; only recall is reported for that specific analysis.
• Entity representations are computed by averaging word-level hidden states for all mentions of an entity, requiring accurate coreference resolution before scoring.

Evidence (verbatim from paper)

Two widely used metrics F1 and AUC are used in our experiments. However, some relational facts present in both the training and dev/test sets, thus a model may memorize their relations during training and achieve a better performance on the dev/test set in an undesirable way, introducing evaluation bias. However, the overlap in relational facts between the training and dev/test sets is inevitable, since many common relational facts are likely to be shared in different documents. Therefore, we also report the F1 and AUC scores excluding those relational facts shared by the training and dev/test sets, denoted as Ign F1 and Ign AUC, respectively.

Citation

@misc{yao2019docred,
  title={DocRED: A Large-Scale Document-Level Relation Extraction Dataset},
  author={Yao et al. (2019)},
  year={2019},
  note={arXiv:1906.06127}
}

• arXiv: 1906.06127