Ielm Eval

ielm-eval

IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models — Wang et al. (2022) (arXiv:2210.14128, 2022)

What this evaluates

Evaluates the zero-shot open information extraction (OIE) capability of pre-trained language models by measuring their ability to extract subject-predicate-object triples from text without task-specific training or fine-tuning. It probes whether LMs inherently store rich, open-world relational knowledge that can be accessed via attention mechanisms.

Datasets

• CaRB — total ?; splits: test (-1)
• Re-OIE2016 — total ?; splits: test (-1)
• TAC KBP-OIE — total ?; splits: test (-1)
• Wikidata-OIE — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R).
• Precision — range: [0, 1]
  • Ratio of correctly extracted triples to all predicted triples.
• Recall — range: [0, 1]
  • Ratio of correctly extracted triples to all gold triples.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring ranking quality across thresholds.

Input / output format

Input: A sentence or document from which relational knowledge must be extracted.

Output: A set of triples (subject, predicate, object) representing the extracted relations.

Scoring recipe

def compute_f1(predictions, gold):
    pred_set = set(tuple(sorted(t)) for t in predictions)
    gold_set = set(tuple(sorted(t)) for t in gold)
    tp = len(pred_set & gold_set)
    p = tp / len(pred_set) if pred_set else 0
    r = tp / len(gold_set) if gold_set else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return f1

Common pitfalls

• spaCy noun chunker errors account for approximately 30% of argument extraction mistakes.
• Predicates located outside the argument pairs are not handled by the current extraction pipeline.
• Zero-shot evaluation uses no task-specific training or fine-tuning, measuring only inherent LM knowledge.
• Performance varies significantly between standard OIE datasets (CaRB, Re-OIE2016) and factual OIE datasets (TAC KBP-OIE, Wikidata-OIE).

Evidence (verbatim from paper)

The best zero-shot OIE system based on GPT-2XL obtains a +2.6% and a +3.1% absolute F1 improvement on TAC KBP-OIE and Wikidata-OIE respectively over the previous supervised state-of-the-art.

Citation

@misc{wang2022ielm,
  title={IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2210.14128}
}

• arXiv: 2210.14128