# Chimera Extraction Eval

> Evaluates a model's ability to identify scientific idea recombinations in research abstracts, extract the involved scientific concepts (entities), and classify the type of recombination relation (inspiration or blend) between them. Use when the user wants to benchmark on CHIMERA, or asks about evaluating this task. Reports Precision, Recall, F1.

- Skill: `qhjqhj00/chimera-extraction-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/chimera-extraction-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/chimera-extraction-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/chimera-extraction-eval

---


# chimera-extraction-eval

> CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation — Sternlicht et al. (2025) (arXiv:2505.20779, 2025)

## What this evaluates

Evaluates a model's ability to identify scientific idea recombinations in research abstracts, extract the involved scientific concepts (entities), and classify the type of recombination relation (inspiration or blend) between them.

## Datasets

- **CHIMERA** — total 36464; splits: train (25317), validation (530), test (10617); repo https://github.com/noy-sternlicht/CHIMERA-KB

## Metrics

- `Precision, Recall, F1` **(primary)** — range: [0, 1]
  - Standard precision, recall, and F1 computed under soft matching for entities (semantic similarity judged by GPT-4o-mini, 1-to-1 constraint) and partial matching for relations (TP count proportional to correctly matched entities in gold relations of the same type).
- `Cohen's kappa` — range: [0, 1]
  - Inter-annotator agreement metric for classification and extraction tasks, calculated between human annotators.
- `H@K, MRR, MedR` — range: [0, 1] | other
  - Retrieval metrics for recombination prediction: Hit rate at K, Mean Reciprocal Rank, and Median Rank.

## Input / output format

**Input**: Research paper abstracts

**Output**: Binary classification label (recombination present/absent), list of extracted scientific concept entities, and relation type (inspiration or blend) linking the entities.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    # 1. Soft entity matching (1-to-1)
    matched_p, matched_g = set(), set()
    for p in preds.entities:
        best_g = max(golds.entities, key=lambda g: gpt4o_similarity(p, g))
        if best_g not in matched_g and best_g not in matched_p:
            matched_p.add(p); matched_g.add(best_g)
    # 2. Partial relation scoring
    tp = 0
    for p_rel in preds.relations:
        for g_rel in golds.relations:
            if p_rel.type == g_rel.type:
                tp += len([e for e in p_rel.ents if e in matched_g]) / len(g_rel.ents)
    # 3. P/R/F1
    prec = tp / len(preds.relations) if preds.relations else 0
    rec = tp / len(golds.relations) if golds.relations else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1
```

## Common pitfalls

- Soft matching relies on an external LLM judge (GPT-4o-mini) for semantic similarity, which may introduce non-determinism or bias if the prompt/version changes.
- Relation scoring uses partial matching where TPs are proportional to matched entities, making exact score reproduction difficult without the exact matching script.
- Year-based split (<2024 train/val, >=2024 test) is critical to prevent data contamination from future publications; standard random splits will leak information.

## Evidence (verbatim from paper)

> For abstract classification, we report precision, recall, and F1. For entity and relation extraction, we adopt a soft matching approach: two entities of the same type match if they refer to semantically similar concepts. We use GPT-4o-mini to judge similarity (see prompt and details in Appendix[B.4]). A predicted entity may match at most one gold entity, and vice versa; extra matches are ignored. We compute precision, recall, and F1 under this soft matching. For relations, we use partial matching: a predicted relation contributes to the true positive count proportionally to the number of correctly matched entities in a gold relation of the same type.

## Citation

```bibtex
@misc{sternlicht2025chimera,
  title={CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation},
  author={Sternlicht et al. (2025)},
  year={2025},
  note={arXiv:2505.20779}
}
```

- arXiv: 2505.20779