Wire57 Eval

wire57-eval

WiRe57 : A Fine-Grained Benchmark for Open Information Extraction — Léchelle et al. (2018) (arXiv:1809.08962, 2018)

What this evaluates

Evaluates Open Information Extraction systems on their ability to accurately extract relational tuples from text. It probes token-level precision and recall by matching predicted arguments and relations against a fine-grained, manually annotated gold standard.

Datasets

• WiRe57 — total 347; splits: test (347); repo https://github.com/rali-udem/WiRe57

Metrics

• token-weighted F1 (primary) — range: [0, 1]
  • System-level precision and recall are computed as the sum of shared words across all matched tuple parts divided by the total length of predicted and gold tuples respectively. F1 is the harmonic mean of these system-level P and R. Matching is done greedily by max F1 score.

Input / output format

Input: A sentence and its corresponding gold standard OIE tuples.

Output: A list of predicted OIE tuples in the format (arg1; rel; arg2; arg3; arg4; arg5).

Scoring recipe

def compute_wire57_f1(preds, gold):
    matches = []
    used_gold = set()
    for p in preds:
        best_f1, best_g = -1, None
        for j, g in enumerate(gold):
            if j in used_gold: continue
            if shares_core_words(p, g): # arg1, rel, arg2
                p_score = shared_words(p, g) / len(p)
                g_score = shared_words(p, g) / len(g)
                f1 = 2 * p_score * g_score / (p_score + g_score)
                if f1 > best_f1: best_f1, best_g = f1, j
        if best_g is not None:
            matches.append((p, gold[best_g]))
            used_gold.add(best_g)
    total_pred_len = sum(len(p) for p in preds)
    total_gold_len = sum(len(g) for g in gold)
    shared = sum(shared_words(p, g) for p, g in matches)
    prec = shared / total_pred_len if total_pred_len else 0
    rec = shared / total_gold_len if total_gold_len else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) else 0

Common pitfalls

• Overlong extractions can artificially inflate recall if not penalized by token-level scoring.
• Coreference resolution and inferred words are explicitly excluded from the gold standard used for scoring, so systems attempting to resolve them will be unfairly penalized.
• Greedy matching order affects final scores; pairs with higher F1 are matched first, which may leave suboptimal matches for remaining tuples.

Evidence (verbatim from paper)

Our scorer computes precision and recall of a system’s predicted tuples at the token level. Precision is, briefly put, the proportion of extracted words that are found in the reference. Recall is the proportion of reference words found in the systems’ predictions. We match predicted tuples with reference ones by greedily removing from the potential match pool the pair with maximum F1 score, until no remaining tuples match. Hence, the overall performance metrics of an extractor are its token-weighted precision and recall over all tuples, i.e. precision_sys = ... recall_sys = ... F1_sys = ...

Citation

@misc{lechelle2018wire57,
  title={WiRe57 : A Fine-Grained Benchmark for Open Information Extraction},
  author={Léchelle et al. (2018)},
  year={2018},
  note={arXiv:1809.08962}
}

• arXiv: 1809.08962