Multilingual Medical Benchmarks Eval

multilingual-medical-benchmarks-eval

Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain — García-Ferrero et al. (2024) (arXiv:2404.07613, 2024)

What this evaluates

Evaluates multilingual text-to-text models on medical argument mining (sequence labeling) and abstractive question answering across English, Spanish, French, and Italian.

Datasets

• AbstRCT — total ?; splits: train (-1), test (-1)
• BioASQ 6B — total ?; splits: test (-1)

Metrics

• sequence-level F1 (primary) — range: [0, 1]
  • Standard sequence-level F1 score for exact span matching (Tjong Kim Sang & De Meulder, 2003).
• QA exact-match — range: [0, 1]
  • Assessed by comparing generated answers against a set of ideal gold answers provided for each question.

Input / output format

Input: Argument Mining: Raw medical and scientific text. QA: A biomedical question accompanied by a set of relevant text snippets.

Output: Argument Mining: Predicted spans labeled as 'Claim' or 'Premise'. QA: A generated natural language answer.

Scoring recipe

def compute_seq_f1(pred_spans, gold_spans):
    pred_set = set(pred_spans)
    gold_set = set(gold_spans)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

# For QA: compare generated answer to each gold answer, take max exact match or F1, then average across questions.

Common pitfalls

• Using token-level F1 instead of the strictly required sequence-level F1 for argument mining.
• Treating glaucoma and mixed splits as in-domain; they are explicitly cross-domain evaluation splits.
• Assuming single gold answer per question for QA; the dataset provides a set of ideal gold answers.

Evidence (verbatim from paper)

However, in this paper we report results using the standard sequence level $F_{1}$ score Tjong Kim Sang and De Meulder ([2003]), a much more strict metric, which explains the lower results for all the models. ... Given a biomedical question and a set of snippets of text with relevant information about the question, the model must generate the ideal answer. A set of ideal gold answers are provided to assess the performance of the models.

Citation

@misc{garciaferrero2024medicalmt5,
  title={Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain},
  author={García-Ferrero et al. (2024)},
  year={2024},
  note={arXiv:2404.07613}
}

• arXiv: 2404.07613