Admedtagger Medical Eval

admedtagger-medical-eval

ADMEDTAGGER: an annotation framework for distillation of expert knowledge for the Polish medical language — Górski et al. (2025) (arXiv:2601.09722, 2025)

What this evaluates

Evaluates the ability of lightweight BERT-based models to classify Polish medical texts into five clinical categories. The benchmark tests knowledge distillation from a large LLM teacher to smaller classifiers, with ground truth curated by medical experts.

Datasets

• ADMEDTAGGER Physician-Validated Test Sets — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard F1 score calculated per clinical category, likely macro-averaged across the five categories. Combines precision and recall into a single harmonic mean metric.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of test instances.

Input / output format

Input: Polish medical text snippets representing specific clinical scenarios.

Output: One of five clinical category labels per text instance.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    f1 = f1_score(gold_labels, predictions, average='macro')
    return {'accuracy': accuracy, 'f1': f1}

Common pitfalls

• Metrics are reported per clinical scenario rather than globally aggregated across all categories.
• Ground truth is physician-validated rather than raw, limiting direct comparison with uncurated real-world benchmarks.
• The exact averaging strategy (macro vs. micro) for F1 is not explicitly stated in the text.

Evidence (verbatim from paper)

Finally, each trained BERT classifier was evaluated on the physician-validated test sets. DistilBERT achieves F1 scores >0.80 across all five clinical categories and >0.93 for three, outperforming others in accuracy while being 500× smaller and 300× more memory-efficient than the teacher LLM...

Citation

@misc{gorski2025admedtagger,
  title={ADMEDTAGGER: an annotation framework for distillation of expert knowledge for the Polish medical language},
  author={Górski et al. (2025)},
  year={2025},
  note={arXiv:2601.09722}
}

• arXiv: 2601.09722