Fredo Eval

fredo-eval

Few-Shot Document-Level Relation Extraction — Popovic et al. (2022) (arXiv:2205.02048, 2022)

What this evaluates

Evaluates few-shot document-level relation extraction by testing a model's ability to identify relations between entity pairs across documents using limited support examples. It specifically probes domain adaptation capabilities, handling of class imbalance, and robustness to NOTA (none-of-the-above) distributions in realistic document-level settings.

Datasets

• FREDo — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/nicpopovic/FREDo

Metrics

• macro F1 (primary) — range: percent
  • Macro-averaged F1 score computed across all relation types. Precision, recall, and F1 are calculated per relation type and then averaged.

Input / output format

Input: A query document containing multiple entity mentions and candidate pairs, alongside a support set of K labeled documents for each relation type (including NOTA).

Output: A predicted relation type label for each candidate entity pair in the query document.

Scoring recipe

f1_scores = []
for rel in all_relations:
    tp = sum(1 for p, g in zip(preds, golds) if p == rel and g == rel)
    fp = sum(1 for p, g in zip(preds, golds) if p == rel and g != rel)
    fn = sum(1 for p, g in zip(preds, golds) if p != rel and g == rel)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    f1_scores.append(f1)
macro_f1 = sum(f1_scores) / len(f1_scores)

Common pitfalls

• NOTA (none-of-the-above) is treated as a relation type, requiring explicit handling of class imbalance and negative examples rather than standard binary classification.
• Cross-domain evaluation involves distribution shift between support and query documents, invalidating standard prototype averaging assumptions.
• Repeated entity mentions in a single document require explicit pooling (e.g., mean pooling) before relation embedding formation, which significantly impacts performance.

Evidence (verbatim from paper)

In order to get sufficient coverage to calculate representative macro $F_{1}$ scores on the development set, we sample 4k episodes. ... Results for FREDo in-domain task. Reported results are macro averages across relation types.

Citation

@misc{popovic2022fewshotdocumentlevelrelationextraction,
  title={Few-Shot Document-Level Relation Extraction},
  author={Popovic et al. (2022)},
  year={2022},
  note={arXiv:2205.02048}
}

• arXiv: 2205.02048