Scientific Embedding Eval

scientific-embedding-eval

Evaluating Embedding Frameworks for Scientific Domain — Ahmed et al. (2025) (arXiv:2510.06244, 2025)

What this evaluates

Evaluates static (Word2Vec, FastText) and transformer-based (SciBERT, RoBERTa) embedding models on scientific text using intrinsic (word/sentence similarity) and extrinsic (NER, document classification) tasks. Probes the impact of domain-specific pretraining and sub-word tokenization on representation quality and downstream performance.

Datasets

• UNMSRS — total ?; splits: test (-1)
• SemEval — total ?; splits: test (-1)
• Clinical STS 2018 — total ?; splits: test (-1)
• Clinical STS 2019 — total ?; splits: test (-1)
• Conll 2003 — total ?; splits: test (-1)
• CHEMDNER — total ?; splits: test (-1)
• SciERC — total ?; splits: test (-1)
• Reuters 12 — total ?; splits: test (-1)
• BioChem 8 — total ?; splits: test (-1)

Metrics

• Pearson (primary) — range: [-1, 1]
  • Pearson correlation coefficient measuring the linear relationship between predicted and gold similarity scores. Computed as the covariance of the two variables divided by the product of their standard deviations.
• F-Beta (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for entity or class prediction, typically with β=1. Calculated as (1 + β²) * (precision * recall) / ((β² * precision) + recall).

Input / output format

Input: Pairs of words or sentences for similarity tasks; raw text spans or documents for NER and classification tasks.

Output: Continuous similarity scores for pairs; discrete entity tags or document class labels for NER and classification.

Scoring recipe

# Pearson correlation for similarity tasks
pearson_r = np.corrcoef(gold_scores, pred_scores)[0, 1]

# F-Beta score for NER/Classification tasks
tp = sum(1 for g, p in zip(gold_labels, pred_labels) if g == p and g != 0)
fp = sum(1 for g, p in zip(gold_labels, pred_labels) if g != p and p != 0)
fn = sum(1 for g, p in zip(gold_labels, pred_labels) if g != p and g != 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f_beta = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall)

Common pitfalls

• Pearson correlation calculation ignores NaN values produced by word-based models on OOV tokens, artificially inflating their scores compared to sub-word models that handle OOVs.
• Sub-word tokenization does not universally improve performance; word-based tokenizers often outperform on sentence similarity and document classification due to large vocabulary sizes mitigating OOV issues.

Evidence (verbatim from paper)

It is able to generate a relatively good pearson score for in vocabulary tokens though.

Citation

@misc{ahmed2025evaluatingembedding,
  title={Evaluating Embedding Frameworks for Scientific Domain},
  author={Ahmed et al. (2025)},
  year={2025},
  note={arXiv:2510.06244}
}

• arXiv: 2510.06244