Lsoie Eval

lsoie-eval

LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction — Solawetz et al. (2021) (arXiv:2101.11177, 2021)

What this evaluates

Evaluates supervised open information extraction (OIE) models on extracting schema-free predicate-argument tuples from sentences. Probes the model's ability to correctly identify predicates and arguments while maintaining syntactic head alignment and argument ordering.

Datasets

• LSOIE — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for extracted tuples. A prediction matches a gold tuple if it shares the same predicate and contains the syntactic head of each gold argument, identified via Stanford CoreNLP.

Input / output format

Input: A single sentence tokenized into tokens, optionally augmented with features like POS embeddings or BERT Sentence A/B embeddings.

Output: A BIO tagging sequence per token, representing extractions. Extractions are decoded via Viterbi algorithm and thresholded using model confidence scores (mean log probability for non-CRF, sequence log probability for CRF).

Scoring recipe

def compute_f1(predictions, gold):
    matches = 0
    for pred in predictions:
        pred_heads = get_syntactic_heads(pred)  # via Stanford CoreNLP
        for gold_tuple in gold:
            if pred.predicate == gold_tuple.predicate and \
               all(head in pred_heads for head in gold_tuple.syntactic_heads):
                matches += 1
                break
    precision = matches / len(predictions) if predictions else 0
    recall = matches / len(gold) if gold else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Matching relies on syntactic heads rather than full lexical overlap, which can cause mismatches if the dependency parser misidentifies heads.
• Confidence scoring uses mean log probability instead of the product of inverse probabilities used in prior work, altering the precision-recall curve shape.
• BIO tagging format requires valid sequence decoding via Viterbi, which constrains valid extractions and may filter out high-probability but structurally invalid sequences.

Evidence (verbatim from paper)

Table 3: Modeling results on the LSOIE test sets. We evaluate our system's performance against the gold test data in LSOIE-wiki and LSOIE-sci by considering extractions to be a match if they contain the same predicate as the gold extraction and contain the syntactic head of each gold argument. Syntactic heads are extracted with the Stanford CoreNLP dependency parser (Chen and Manning, 2014).

Citation

@misc{solawetz2021lsoie,
  title={LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction},
  author={Solawetz et al. (2021)},
  year={2021},
  note={arXiv:2101.11177}
}

• arXiv: 2101.11177