Kbp Auc Eval

kbp-auc-eval

Populating Web Scale Knowledge Graphs using Distantly Supervised Relation Extraction and Validation — Dash et al. (2019) (arXiv:1908.08104, 2019)

What this evaluates

Evaluates the ability of distantly supervised relation extraction and knowledge base validation systems to correctly predict and rank triples in web-scale knowledge graphs. It probes how well global graph structure and confidence scoring can refine noisy extractions and reduce logical inconsistencies.

Datasets

• NYT-FB — total ?; splits: test (-1)
• CC-DBP — total ?; splits: test (-1)
• NELL-165 — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Precision-Recall curve computed across varying confidence thresholds. Precision = TP/(TP+FP), Recall = TP/(TP+FN). In this positive-unlabeled setting, recall measures the percentage of true triples correctly extracted by the IE system above a minimum confidence threshold.

Input / output format

Input: Sentences containing two entity mentions with relation labels (for RE), or candidate triples with initial confidence scores from an IE system (for KBV).

Output: A confidence score for each candidate triple, used to rank and filter predictions.

Scoring recipe

thresholds = sorted(set(predictions), reverse=True)
precisions, recalls = [], []
for t in thresholds:
    tp = sum(1 for p, g in zip(predictions, gold) if p >= t and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p >= t and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p < t and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    precisions.append(prec)
    recalls.append(rec)
auc = trapezoidal_integral(precisions, recalls)

Common pitfalls

• Precision is systematically underestimated because the evaluation uses a positive-unlabeled setting where ground truth only includes known KB triples, not all possible true triples.
• Recall is calculated only over triples extracted by the IE system above a minimum confidence threshold, not over the full set of possible true triples, which limits the scope of the evaluation.

Evidence (verbatim from paper)

In all cases, the recall is the correct percent of triples that were extracted by the IE system above minimum confidence. This recall basis is logical in the case of KBV, but note that KBV or KBV_IE could also be used to predict triples outside the set extracted by an IE system. Table 2 provides the AUC for all the systems described above.

Citation

@misc{dash2019populating,
  title={Populating Web Scale Knowledge Graphs using Distantly Supervised Relation Extraction and Validation},
  author={Dash et al. (2019)},
  year={2019},
  note={arXiv:1908.08104}
}

• arXiv: 1908.08104