Chimera Extraction Eval

chimera-extraction-eval

CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation — Sternlicht et al. (2025) (arXiv:2505.20779, 2025)

What this evaluates

Evaluates a model's ability to identify scientific idea recombinations in research abstracts, extract the involved scientific concepts (entities), and classify the type of recombination relation (inspiration or blend) between them.

Datasets

• CHIMERA — total 36464; splits: train (25317), validation (530), test (10617); repo https://github.com/noy-sternlicht/CHIMERA-KB

Metrics

• Precision, Recall, F1 (primary) — range: [0, 1]
  • Standard precision, recall, and F1 computed under soft matching for entities (semantic similarity judged by GPT-4o-mini, 1-to-1 constraint) and partial matching for relations (TP count proportional to correctly matched entities in gold relations of the same type).
• Cohen's kappa — range: [0, 1]
  • Inter-annotator agreement metric for classification and extraction tasks, calculated between human annotators.
• H@K, MRR, MedR — range: [0, 1] | other
  • Retrieval metrics for recombination prediction: Hit rate at K, Mean Reciprocal Rank, and Median Rank.

Input / output format

Input: Research paper abstracts

Output: Binary classification label (recombination present/absent), list of extracted scientific concept entities, and relation type (inspiration or blend) linking the entities.

Scoring recipe

def compute_metrics(preds, golds):
    # 1. Soft entity matching (1-to-1)
    matched_p, matched_g = set(), set()
    for p in preds.entities:
        best_g = max(golds.entities, key=lambda g: gpt4o_similarity(p, g))
        if best_g not in matched_g and best_g not in matched_p:
            matched_p.add(p); matched_g.add(best_g)
    # 2. Partial relation scoring
    tp = 0
    for p_rel in preds.relations:
        for g_rel in golds.relations:
            if p_rel.type == g_rel.type:
                tp += len([e for e in p_rel.ents if e in matched_g]) / len(g_rel.ents)
    # 3. P/R/F1
    prec = tp / len(preds.relations) if preds.relations else 0
    rec = tp / len(golds.relations) if golds.relations else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1

Common pitfalls

• Soft matching relies on an external LLM judge (GPT-4o-mini) for semantic similarity, which may introduce non-determinism or bias if the prompt/version changes.
• Relation scoring uses partial matching where TPs are proportional to matched entities, making exact score reproduction difficult without the exact matching script.
• Year-based split (<2024 train/val, >=2024 test) is critical to prevent data contamination from future publications; standard random splits will leak information.

Evidence (verbatim from paper)

For abstract classification, we report precision, recall, and F1. For entity and relation extraction, we adopt a soft matching approach: two entities of the same type match if they refer to semantically similar concepts. We use GPT-4o-mini to judge similarity (see prompt and details in Appendix[B.4]). A predicted entity may match at most one gold entity, and vice versa; extra matches are ignored. We compute precision, recall, and F1 under this soft matching. For relations, we use partial matching: a predicted relation contributes to the true positive count proportionally to the number of correctly matched entities in a gold relation of the same type.

Citation

@misc{sternlicht2025chimera,
  title={CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation},
  author={Sternlicht et al. (2025)},
  year={2025},
  note={arXiv:2505.20779}
}

• arXiv: 2505.20779