Openie Systems Eval

openie-systems-eval

When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications — Pei et al. (2022) (arXiv:2211.08228, 2022)

What this evaluates

Evaluates the performance and efficiency of state-of-the-art neural OpenIE models and training datasets across multiple standard benchmarks. It probes how model properties like N-ary relation support and inferred relation extraction capability align with benchmark characteristics and downstream task requirements.

Datasets

• OIE2016 — total ?; splits: test (-1)
• WiRE57 — total ?; splits: test (-1)
• ReOIE2016 — total ?; splits: test (-1)
• CaRB — total ?; splits: test (-1)
• LSOIE — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, calculated as the maximum F1 value across the precision-recall curve.
• Precision — range: [0, 1]
  • Proportion of extracted relations that exactly match gold standard relations.
• Recall — range: [0, 1]
  • Proportion of gold standard relations successfully extracted by the model.
• Sentences per second — range: other
  • Throughput metric measuring the average number of sentences processed per second across batches.

Input / output format

Input: Raw English sentences from benchmark datasets.

Output: Extracted relation tuples (subject, relation, object), with optional additional arguments for N-ary relations.

Scoring recipe

def compute_f1(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• Using AUC instead of F1 can artificially inflate scores for low-recall models due to the trapezoidal rule's assumption of a (recall=0, precision=1) anchor point.
• Training data preprocessing (e.g., stripping inferred relations or flattening N-ary relations) is applied to training sets but not test sets, potentially skewing performance expectations.
• Efficiency is measured as average sentences per second across batches, not per individual sentence, which can obscure latency variance during parallel execution.

Evidence (verbatim from paper)

When comparing OpenIE systems, we place a greater emphasis on F1 score than AUC. ... We compare performance using primarily F1 score to address HR and HP and sentences extracted per second to address FE.

Citation

@misc{pei2022whentousewhat,
  title={When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications},
  author={Pei et al. (2022)},
  year={2022},
  note={arXiv:2211.08228}
}

• arXiv: 2211.08228