Benchie Fl Eval

benchie-fl-eval

$\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark — Lamarche et al. (2024) (arXiv:2407.16860, 2024)

What this evaluates

Evaluates Open Information Extraction (OIE) systems on their ability to extract fact-based triples from text. It uses a conservative exact-matching function with synset-based clustering to penalize non-informative copies and reward precise fact extraction, while also measuring correlation with downstream QA and knowledge base tasks.

Datasets

• BenchIE^FL — total ?; splits: test (-1); repo https://github.com/rali-udem/benchie_fl.git

Metrics

• exact-match (primary) — range: [0, 1]
  • 1 if a predicted triple exactly matches a gold triple or its synset cluster, else 0. Averaged over instances.
• Mean Reciprocal Rank — range: [0, 1]
  • Average of 1/rank for the first correct answer across queries.

Input / output format

Input: Raw text passages or sentences.

Output: Extracted fact-based triples (subject, relation, object).

Scoring recipe

def compute_score(predictions, gold):
    correct = 0
    for pred in predictions:
        if pred in gold or synset_match(pred, gold):
            correct += 1
    return correct / len(gold)

Common pitfalls

• Neural OIE models tend to copy large chunks of input text, producing non-informative tuples that inflate scores on older benchmarks.
• Optimizing extractors for specific benchmarks does not lead to significant performance gains but can skew system rankings.
• Lexical variation requires synset-based clustering for matching; strict string matching causes false negatives.

Evidence (verbatim from paper)

We then use QUEST to construct answers and report the scores measured by the Mean Reciprocal Rank.

Citation

@misc{lamarche2024benchiefl,
  title={$\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark},
  author={Lamarche et al. (2024)},
  year={2024},
  note={arXiv:2407.16860}
}

• arXiv: 2407.16860