Compactie Eval

compactie-eval

CompactIE: Compact Facts in Open Information Extraction — Fatahi Bayat et al. (2022) (arXiv:2205.02880, 2022)

What this evaluates

Evaluates Open Information Extraction systems on their ability to extract compact, clause-level facts from text. It measures precision, recall, and F1 using token-level matching against gold triples, with a focus on avoiding over-specific extractions and handling overlapping constituents.

Datasets

• CaRB — total 641; splits: test (577); repo https://github.com/dair-iitd/CaRB
• Wire57 — total 57; splits: test (56); repo https://github.com/rali-udem/WiRe57
• BenchIE — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Standard F1 score (harmonic mean of precision and recall) computed using dataset-specific token-level matching scoring functions.

Input / output format

Input: Input sentences (pre-processed to remove conjunctions).

Output: A set of Open Information Extraction triples (subject, relation, object).

Scoring recipe

def compute_f1(pred_triples, gold_triples):
    tp = sum(1 for p in pred_triples if any(token_match(p, g) for g in gold_triples))
    fp = len(pred_triples) - tp
    fn = len(gold_triples) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• CaRB and Wire57 are incomplete benchmarks; gold sets do not include all acceptable surface realizations, which can artificially suppress recall.
• Scoring functions explicitly penalize over-specific extractions, requiring models to balance compactness with recall.
• Pre-processing steps (e.g., removing conjunctions, excluding triples with clauses inside constituents) must be applied consistently for fair comparison.

Evidence (verbatim from paper)

We report precision (P), recall (R), and F1 computed by these scoring functions. Wire57 contains more fine-grained extractions than the CaRB dataset and its scoring function is more rigorous for compact facts since it penalizes over-specific extractions. However, both CaRB and Wire57 scoring functions are based on token-level matching of system extractions against ground truth facts.

Citation

@misc{fatahibayat2022compactie,
  title={CompactIE: Compact Facts in Open Information Extraction},
  author={Fatahi Bayat et al. (2022)},
  year={2022},
  note={arXiv:2205.02880}
}

• arXiv: 2205.02880