Oie Error Analysis Eval

oie-error-analysis-eval

Analysing Errors of Open Information Extraction Systems — Rudolf Schneider et al. (2017) (arXiv:1707.07499, 2017)

What this evaluates

Evaluates Open Information Extraction (OIE) systems on their ability to correctly extract relational tuples from text, measuring precision, recall, and F2 scores under both strict and relaxed containment matching strategies. It also qualitatively classifies extraction errors to identify systematic failure modes like boundary mismatches and annotation style conflicts.

Datasets

• NYT-222 — total ?; splits: (unstated)
• WEB-500 — total ?; splits: (unstated)
• PENN-100 — total ?; splits: (unstated)
• OIE2016 — total 1768; splits: (unstated)

Metrics

• F2 (primary) — range: [0, 1]
  • F2 = (5 * P * R) / (4 * P + R), where P is precision and R is recall. Beta is set to 2 to weight recall twice as heavily as precision.

Input / output format

Input: Raw sentences from the dataset.

Output: Extracted relations formatted as predicate(arg1, arg2, ...) with variable arity (unary to n-ary).

Scoring recipe

def containment_match(pred, gold, relaxed=False):
    if relaxed:
        return pred.predicate == gold.predicate and set(pred.args) <= set(gold.args)
    return pred == gold  # exact span/string match

tp = sum(1 for p in preds if any(containment_match(p, g, relaxed=True) for g in golds))
fp = len(preds) - tp
fn = len(golds) - tp
P = tp / (tp + fp) if (tp + fp) > 0 else 0
R = tp / (tp + fn) if (tp + fn) > 0 else 0
F2 = (5 * P * R) / (4 * P + R) if (4 * P + R) > 0 else 0

Common pitfalls

• Strict containment matching penalizes systems heavily for minor boundary shifts or argument count differences, especially when comparing binary systems on n-ary gold standards.
• Annotation style mismatches between gold datasets and system outputs (e.g., verb-centric vs. nominal triggers) can cause false negatives that are not true extraction failures.
• Noisy text artifacts (HTML encodings, unfinished sentences) in datasets like WEB-500 disrupt intermediate parsing steps, artificially lowering recall.

Evidence (verbatim from paper)

In a quantitative evaluation we report precision, recall and $F_{2}$ scores on all four data sets. We conduct our experiments with an exact (a) and relaxed (b) containment match strategy.

Citation

@misc{schneider2017analysing,
  title={Analysing Errors of Open Information Extraction Systems},
  author={Rudolf Schneider et al. (2017)},
  year={2017},
  note={arXiv:1707.07499}
}

• arXiv: 1707.07499