qhjqhj00 refind-relation-extraction-eval
REFiND: Relation Extraction Financial Dataset — Kaur et al. (2023) (arXiv:2305.18322, 2023)
What this evaluates
This benchmark evaluates a model's ability to perform relation extraction on complex, domain-specific financial documents (SEC 10-X filings). It specifically probes challenges such as numerical inference, semantic ambiguity between similar relation types, and directional dependency resolution in long-range financial text.
Datasets
- REFiND — total 29000; splits: train (-1), val (-1), test (-1)
Metrics
micro-F1(primary) — range: percent- Standard micro-averaged F1 score calculated globally across all entity pair groups and the entire dataset.
macro-F1— range: percent- Standard macro-averaged F1 score calculated globally across all entity pair groups and the entire dataset.
Input / output format
Input: Financial text sentences with entity mentions marked by special tokens: [E1], / / E1, [E2], and / / E2, along with an ordered pair of entity mentions (e1, e2).
Output: A single relation label from a predefined set of 22 relations across 8 entity types.
Scoring recipe
def compute_f1(preds, gold, average='micro'):
# preds and gold are lists of relation labels
# Calculate precision and recall per class or globally based on average
# Return F1 score as percentage
tp = sum(1 for p, g in zip(preds, gold) if p == g)
fp = sum(1 for p, g in zip(preds, gold) if p != g and p in gold)
fn = sum(1 for p, g in zip(preds, gold) if p != g and g in preds)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1 * 100
Common pitfalls
- Models often fail numerical inference, treating percentages (e.g., 31% vs 100% equity) as regular tokens rather than performing logical checks for acquisition thresholds.
- Semantic ambiguity causes confusion between closely related relations like member_of, employee_of, and founder_of for the same entity pairs.
- Directional ambiguity leads to errors swapping acquired_by and subsidiary_of despite entity ordering cues.
Evidence (verbatim from paper)
To ensure a comprehensive evaluation of the benchmarks, we report both micro- and macro-F1 score metrics (evaluation details in Appendix A.3). We also evaluate the performance of each model on each entity pair group and the entire REFinD dataset to obtain a more detailed understanding of the models' strengths and weaknesses.
Citation
@misc{kaur2023refind,
title={REFiND: Relation Extraction Financial Dataset},
author={Kaur et al. (2023)},
year={2023},
note={arXiv:2305.18322}
}
- arXiv: 2305.18322