Jmedbench Eval

jmedbench-eval

JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models — Jiang et al. (2024) (arXiv:2409.13317, 2024)

What this evaluates

Evaluates Japanese biomedical large language models across five tasks: multiple-choice question answering, named entity recognition, machine translation, document classification, and semantic text similarity. It probes domain-specific knowledge, multilingual comprehension, and in-context learning capabilities.

Datasets

• JMedBench — total 38130; splits: test (-1)

Metrics

• F1-entity (primary) — range: percent
  • Micro-averaged F1 score for entity recognition, computed over predicted and gold entity spans.
• Accuracy (primary) — range: percent
  • Percentage of correctly predicted labels or multiple-choice answers.
• BLEU — range: percent
  • Standard BLEU score for machine translation evaluation.
• Pearson correlation — range: [-1, 1]
  • Pearson correlation coefficient between predicted and gold similarity scores.

Input / output format

Input: Text prompt formatted with one of four templates (Standard, English-centric, Instructed, Minimal), optionally containing zero-shot or few-shot demonstrations.

Output: Model-generated text response (e.g., selected option, entity tags, translated sentence, class label, or numerical similarity score).

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'NER':
        return micro_f1(predictions, golds)
    elif task in ['MCQA', 'DC']:
        return accuracy(predictions, golds)
    elif task == 'MT':
        return bleu_score(predictions, golds)
    elif task == 'STS':
        return pearson_corr(predictions, golds)

Common pitfalls

• Few-shot NER evaluation is restricted to 1 shot for long-text datasets (MRNER, NRNER) to avoid exceeding model context windows.
• Continual pre-training on biomedical or Japanese text can cause catastrophic forgetting, degrading multilingual performance (e.g., Meditron).
• Machine translation performance correlates more with language generation ability than understanding, especially for cross-lingual directions.

Evidence (verbatim from paper)

We report the highest F1-entity score among four prompt templates as discussed in Section [3.3]. The best and second-best performances are highlighted in bold and underlined, respectively.

Citation

@misc{jiang2024jmedbench,
  title={JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2409.13317}
}

• arXiv: 2409.13317