Emerge Eval

emerge-eval

EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge — Zaporojets et al. (2025) (arXiv:2507.03617, 2025)

What this evaluates

Evaluates the ability of information extraction models to update knowledge graphs with emerging textual knowledge. It probes capabilities in extracting existing and new triples, linking emerging entities, and deprecating obsolete relations based on temporal text evidence.

Datasets

• EMERGE — total ?; splits: temporal_deltas_2019_2025 (-1); repo https://github.com/klimzaporojets/emerge

Metrics

• recall (primary) — range: [0, 1]
  • True positives divided by the sum of true positives and false negatives. Measures the fraction of gold triples successfully extracted or identified for deprecation.
• completeness score — range: [0, 1]
  • Cosine similarity between the embedding of predicted triples and the embedding of ground truth triples. Used for models that do not perform explicit entity/relation alignment.

Input / output format

Input: Text passages (subsampled from Wikipedia) paired with a temporal KG snapshot. For closed IE, the model receives entity and relation dictionaries from the snapshot; for open IE, only relation types or raw text.

Output: A set of extracted triples (subject, relation, object) or a list of existing KG triples flagged for deprecation based on textual evidence.

Scoring recipe

def compute_metrics(predictions, gold, is_closed_ie):
    if is_closed_ie:
        tp = len(set(predictions) & set(gold))
        fn = len(set(gold) - set(predictions))
        return tp / (tp + fn) if (tp + fn) > 0 else 0.0
    else:
        pred_vec = mean_pool(triple_embeddings(predictions))
        gold_vec = mean_pool(triple_embeddings(gold))
        return cosine_similarity(pred_vec, gold_vec)

Common pitfalls

• Using precision or F1 scores instead of recall/completeness, which misrepresents performance under the open-world assumption due to inherent KG incompleteness.
• Applying exact-match evaluation to models that do not link extracted mentions to canonical KG entities, requiring cosine similarity instead.
• Evaluating on the full dataset without accounting for the paper's subsampling protocol (200 passages per temporal delta) due to computational constraints.

Evidence (verbatim from paper)

In order to evaluate the extraction and deprecation of triples based on emerging knowledge in text, we use recall as the primary metric (see Appendix [A]) to evaluate performance. We do not use precision and F1 scores, as these metrics might be misleading under the open-world assumption (Razniewski et al., [2024]). Under this assumption, the model may generate correct triple predictions that are incorrectly classified as false positives due to the inherently incomplete nature of KGs, which do not necessarily capture the full set of valid triples.

Citation

@misc{zaporojets2025emerge,
  title={EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge},
  author={Zaporojets et al. (2025)},
  year={2025},
  note={arXiv:2507.03617}
}

• arXiv: 2507.03617