Resel Scientific Ie Eval

resel-scientific-ie-eval

ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select — Zhuang et al. (2022) (arXiv:2210.14427, 2022)

What this evaluates

Evaluates a model's ability to perform N-ary relation extraction from scientific documents by first retrieving relevant text/table components and then selecting the correct entities within those components.

Datasets

• SciREX — total 438; splits: test (-1)
• PubMed — total 5688; splits: test (-1)
• NLP-TDMS (Full) — total 332; splits: test (-1)

Metrics

• Accuracy (Acc) (primary) — range: percent
  • Fraction of instances where the predicted component or entity exactly matches the ground truth.
• Mean Reciprocal Rank (MRR) — range: [0, 1]
  • Average of 1/rank for the correct answer across all queries, where rank is the position of the ground truth in the ranked list.
• Top-k Hit Rate (Hit@K) — range: percent
  • Fraction of queries where the correct answer appears within the top K ranked results (k=2, 3, 5).

Input / output format

Input: A query specifying the relation to extract, paired with a full scientific document containing text and tables (extracted from LaTeX/PDF).

Output: A ranked list of retrieved document components (paragraphs/tables) and a set of selected target entities within the retrieved components.

Scoring recipe

def compute_metrics(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    mrr = sum(1.0 / (rank + 1) for p, g in zip(predictions, gold) for rank, pred in enumerate(p) if pred == g) / len(gold)
    hit_rates = {k: sum(1 for p, g in zip(predictions, gold) if g in p[:k]) / len(gold) for k in [2, 3, 5]}
    return acc, mrr, hit_rates

Common pitfalls

• Using general-domain BERT instead of domain-specific encoders (SciBERT/ClinicalBERT) significantly degrades performance.
• Graph-based methods are prone to over-smoothing if the number of GAT layers (L) is increased too much.
• BERT embeddings alone struggle to discriminate numeric values and adjacent cells in tables without structural graph features.

Evidence (verbatim from paper)

Following existing works (Karpukhin et al., 2020), we use (1) Accuracy (Acc), (2) Mean Reciprocal Rank (MRR), and (3) Top-k Hit Rate (Hit@K) with $k = 2, 3, 5$ for evaluating both high- and low-level models (see Appendix D).

Citation

@misc{zhuang2022resel,
  title={ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select},
  author={Zhuang et al. (2022)},
  year={2022},
  note={arXiv:2210.14427}
}

• arXiv: 2210.14427