Openie Eval

openie-eval

A Survey on Neural Open Information Extraction: Current Status and Future Directions — Zhou et al. (2022) (arXiv:2205.11725, 2022)

What this evaluates

This evaluation probes a model's ability to perform Open Information Extraction (OpenIE), which involves identifying and extracting relational triples (subject, predicate, object) from unstructured text without relying on a predefined ontology or schema. It measures how well systems capture complete, correct, and minimal information spans across diverse domains like news and encyclopedias.

Datasets

• OIE2016 — total ?; splits: train (-1), dev (-1), test (-1)
• CaRB — total ?; splits: dev (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed over extracted tuples. Precision and recall are calculated based on tuple matching scores (e.g., lexical overlap, syntactic head, or token-level match) rather than exact string equality.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve, computed by varying the confidence threshold of extracted tuples and measuring precision and recall at each threshold.

Input / output format

Input: A single natural language sentence (typically from news or encyclopedia domains).

Output: A set of extracted relational tuples, each consisting of a subject span, a relation/predicate span, and an object span.

Scoring recipe

def compute_f1(predictions, gold, match_fn):
    tp, fp, fn = 0, 0, 0
    matched_gold = set()
    for pred in predictions:
        best_score, best_idx = -1, -1
        for i, g in enumerate(gold):
            if i not in matched_gold:
                score = match_fn(pred, g)
                if score > best_score:
                    best_score, best_idx = score, i
        if best_score > 0:
            tp += best_score
            matched_gold.add(best_idx)
        else:
            fp += 1
    fn = len(gold) - len(matched_gold)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Using the original OIE2016 lenient lexical overlap matching instead of the stricter syntactic-head matching inflates reported F1 scores significantly.
• Recall computation differs between one-to-one and multi-to-one mapping strategies; failing to specify which is used makes cross-paper comparisons invalid.
• Evaluating models trained on bootstrapped pseudo-labels from rule-based systems may underestimate true neural model capability due to label noise limits.

Evidence (verbatim from paper)

Commonly used measures are F1 and PR-AUC scores. ... OIE2016 proposes to follow the matching criteria introduced in He et al. (2015), and considers two tuples a match if both share the same grammatical head of all of the elements. However, Jiang et al. (2019) noted that the evaluation metric implemented in the public code of OIE2016 uses a more lenient lexical overlap instead. ... CaRB scorer uses token level match and it matches relation with relation, arguments with arguments. ... During precision computation, each extraction is matched exclusively to one gold tuple. ... During recall computation, CaRB scorer allows one extraction being matched by multiple gold tuples...

Citation

@misc{zhou2022survey,
  title={A Survey on Neural Open Information Extraction: Current Status and Future Directions},
  author={Zhou et al. (2022)},
  year={2022},
  note={arXiv:2205.11725}
}

• arXiv: 2205.11725