Bionpars Bench Eval

bionpars-bench-eval

BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining — Baqer M. Merzah et al. (2025) (arXiv:2506.21567, 2025)

What this evaluates

Evaluates a Persian biomedical large language model's ability to generate accurate, domain-specific long-form answers and summaries. It probes subject-specific knowledge acquisition, knowledge synthesis, and evidence-based reasoning by comparing model outputs against human-written biomedical references.

Datasets

• BioPars-BENCH — total 5231; splits: test (5231); repo https://github.com/amirap80/BioPars

Metrics

• BERTScore (primary) — range: [0, 1]
  • Computes token-level cosine similarity using BERT embeddings (layer 9). Precision and Recall are averaged across tokens, and F1 is calculated as 2·P·R/(P+R). Scores are scaled to [0,1] using random text baselines.

Input / output format

Input: Persian biomedical question or text prompt requiring a long-form answer.

Output: Generated Persian text response (long-form answer/summary).

Scoring recipe

def evaluate(pred, refs):
    # BERTScore F1
    R = mean(max(cos_sim(x, y) for y in refs) for x in pred)
    P = mean(max(cos_sim(y, x) for x in pred) for y in refs)
    bert_f1 = 2 * P * R / (P + R)
    # ROUGE-L
    rouge_l = (1 + beta**2) * (R_lcs * P_lcs) / (R_lcs + beta**2 * P_lcs)
    # MoverScore (EMD on IDF-weighted n-gram embeddings)
    moverscore = optimal_transport_cost(pred, refs)
    # BLEURT
    bleurt = bert_model.predict(pred, refs)
    return {'bertscore': bert_f1, 'rouge_l': rouge_l, 'moverscore': moverscore, 'bleurt': bleurt}

Common pitfalls

• ROUGE metrics rely on exact n-gram overlap and may penalize valid semantic paraphrases common in biomedical text.
• BERTScore uses layer 9 of BERT and requires scaling to [0,1] via random text baselines for proper calibration.
• MoverScore computes Earth Mover’s Distance on IDF-weighted n-gram embeddings, making it computationally expensive and sensitive to embedding layer choices.

Evidence (verbatim from paper)

Four criteria were used to evaluate the results: BERTScore[[49]], ROUGE[[50]], Mover Score[[51]], and BLEURT[[52]]. The following is a general definition of these criteria.

Citation

@misc{merzah2025bionpars,
  title={BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining},
  author={Baqer M. Merzah et al. (2025)},
  year={2025},
  note={arXiv:2506.21567}
}

• arXiv: 2506.21567