Unsupervised Relation Extraction Eval

unsupervised-relation-extraction-eval

Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion — Goswami et al. (2020) (arXiv:2010.06804, 2020)

What this evaluates

Evaluates the ability of language models to perform unsupervised relation extraction by predicting relation labels or tokens from contextual text. It probes factual grounding and context-constrained generation capabilities across varying relation types and corpus sources.

Datasets

• T-REx — total 34039; splits: test (34039)
• Google-RE — total 5528; splits: test (5528)
• ZSRE — total 42635; splits: test (42635)
• TACRED — total 6357; splits: test (6357)

Metrics

• EM — range: [0, 1]
  • Exact Match (EM) is calculated as the fraction of instances where the predicted relation token(s) exactly match the gold label.
• F1 (primary) — range: [0, 1]
  • F1 score is the harmonic mean of precision and recall computed over token/label overlaps between predictions and gold labels. F1 = 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Context sentence containing entity mentions and a target relation type to predict.

Output: Predicted relation token(s) or label, constrained to vocabulary present in the supporting context.

Scoring recipe

def compute_metrics(preds, golds):
    em_scores = [1.0 if p == g else 0.0 for p, g in zip(preds, golds)]
    precisions, recalls, f1s = [], [], []
    for p, g in zip(preds, golds):
        p_set, g_set = set(p.split()), set(g.split())
        if not g_set: continue
        tp = len(p_set & g_set)
        precisions.append(tp / len(p_set) if p_set else 0)
        recalls.append(tp / len(g_set))
    f1s = [2 * (p * r) / (p + r) if p + r > 0 else 0 for p, r in zip(precisions, recalls)]
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1s)/len(f1s)}

Common pitfalls

• Assuming supervised fine-tuning is used; the protocol emphasizes unsupervised/constrained cloze completion.
• Overlooking the distinction between single-token and multi-token extraction types across benchmarks.
• Ignoring the information-theoretic context filtering step required before scoring.

Evidence (verbatim from paper)

Table 3: We consider four benchmarks that vary with respect to the type of target extractions, the quality of context to relation alignment, and the underlying corpus.
Table 4: Effect of token expansion on ZSRE dataset. Method | ZSRE (EM, F1) | TACRED (EM, F1) No expansion | 42.4 | 46.1 | 49.6 | 50.3

Citation

@misc{goswami2020unsupervised,
  title={Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion},
  author={Goswami et al. (2020)},
  year={2020},
  note={arXiv:2010.06804}
}

• arXiv: 2010.06804