Openie6 Eval

openie6-eval

OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction — Keshav Kolluru et al. (2020) (arXiv:2010.03147, 2020)

What this evaluates

Evaluates Open Information Extraction systems on their ability to accurately identify and extract relational triples from natural language sentences. It measures precision, recall, and F1 using multiple reference-matching protocols, alongside throughput speed and confidence-threshold robustness (AUC).

Datasets

• CaRB — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (P * R) / (P + R). Computed using system-specific scoring functions (CaRB, OIE16-C, Wire57-C, or CaRB(1-1)).
• AUC — range: [0, 1]
  • Area under the Precision-Recall curve, computed by varying the confidence threshold for each extraction to generate multiple (P, R) points.
• Speed (sentences/sec) — range: other
  • Number of sentences processed per second.

Input / output format

Input: A single natural language sentence (typically from Wikipedia or the CaRB benchmark).

Output: A set of extracted relational triples (subject, relation, object), optionally accompanied by a confidence score for each extraction.

Scoring recipe

def evaluate(predictions, gold, scoring_func):
    # predictions: list of extractions with confidence scores
    # gold: list of reference extractions
    # scoring_func: CaRB, OIE16-C, Wire57-C, or CaRB(1-1)
    matched = scoring_func.align(predictions, gold)
    precision = len(matched) / len(predictions) if predictions else 0.0
    recall = len(matched) / len(gold) if gold else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    auc = compute_area_under_pr_curve(predictions, gold, scoring_func)
    return {'F1': f1, 'AUC': auc}

Common pitfalls

• Different scoring functions (CaRB, OIE16-C, Wire57-C, CaRB(1-1)) use different matching strategies (e.g., one-to-one vs. many-to-many), so results are not directly comparable across functions without noting the variant.
• Wire57-C does not support AUC calculation because its matching algorithm is not naturally compatible with confidence-threshold variation.
• Speed evaluation is run on a specific 3,200-sentence subset, not the full CaRB test set, and uses mixed hardware (V100 GPU + 4 CPU cores).

Evidence (verbatim from paper)

We evaluate all systems against CaRB's reference extractions, as they have higher coverage and quality compared to other datasets. Apart from CaRB's scoring function, we also use scoring functions of OIE16 and Wire57 benchmarks on the CaRB reference set, which we refer to as OIE16-C and Wire57-C. Additionally we use CaRB(1-1), a variant of CaRB that retains CaRB's similarity computation, but uses a one-to-one mapping for both precision and recall (similar to OIE16-C, Wire57-C). For each system, we report a final F1 score using precision and recall computed by these scoring functions. OpenIE systems typically associate a confidence value with each extraction, which can be varied to generate a precision-recall (P-R) curve. We also report the area under P-R curve (AUC) for all scoring functions except Wire57-C, as its matching algorithm is not naturally compatible with P-R curves.

Citation

@misc{kolluru2020openie6,
  title={OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction},
  author={Keshav Kolluru et al. (2020)},
  year={2020},
  note={arXiv:2010.03147}
}

• arXiv: 2010.03147