Fact Based Oie Eval

fact-based-oie-eval

AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark — Friedrich et al. (2021) (arXiv:2109.07464, 2021)

What this evaluates

Evaluates Open Information Extraction systems on their ability to correctly extract complete facts from sentences, moving beyond token-level overlap to fact-level exact matching against exhaustive gold synsets. It measures whether a system can identify all surface realizations of a fact and penalizes extractions that contain correct tokens but express incorrect or incomplete facts.

Datasets

• CaRB — total ?; splits: (unstated)

Metrics

• Precision, Recall, F1 score (fact-based) (primary) — range: [0, 1]
  • Precision = TP / (TP + FP), Recall = TP / (TP + FN), F1 = 2 * (P * R) / (P + R). TP is the number of fact synsets covered by at least one system extraction. FN is the number of fact synsets not covered by any system extraction. FP is the number of system extractions that do not exactly match any gold triple.

Input / output format

Input: Natural language sentence.

Output: List of Open Information Extraction triples (subject; predicate; object) extracted by the system.

Scoring recipe

covered_synsets = set()
for sys_ext in system_extractions:
    for synset in gold_synsets:
        if sys_ext in synset:
            covered_synsets.add(id(synset))
            break
TP = len(covered_synsets)
FN = len(gold_synsets) - TP
FP = len([e for e in system_extractions if not any(e in s for s in gold_synsets)])
Precision = TP / (TP + FP) if (TP + FP) > 0 else 0
Recall = TP / (TP + FN) if (TP + FN) > 0 else 0
F1 = 2 * Precision * Recall / (Precision + Recall) if (Precision + Recall) > 0 else 0

Common pitfalls

• Token-overlap metrics incorrectly reward extractions that contain all gold tokens but express a different or incomplete fact.
• Counting multiple system extractions of the same fact as multiple true positives instead of rewarding the fact only once.
• Failing to account for entity coreference or paraphrased surface realizations when defining gold fact synsets.

Evidence (verbatim from paper)

Because benchmarks based on fact synsets are supposed to be complete, a system OIE extraction is considered correct if and only if it exactly matches any of the gold extractions from any of the fact synsets. The number of true positives (TPs) is the number of fact synsets (i.e., different facts) “covered” by at least one system extraction. This way, a system that extracts N different triples of the same fact, will be rewarded only once for the correct extraction of the fact. False negatives (FNs) are then fact synsets not covered by any of the system extractions. Finally, each system extraction that does not exactly match any gold triple (from any synset) is counted as a false positive (FP). We then compute Precision, Recall, and F1 score from TP, FP, and FN in the standard fashion.

Citation

@misc{friedrich2021annie,
  title={AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark},
  author={Friedrich et al. (2021)},
  year={2021},
  note={arXiv:2109.07464}
}

• arXiv: 2109.07464