Vietmed Ner Eval

vietmed-ner-eval

Medical Spoken Named Entity Recognition — Khai Le-Duc et al. (2024) (arXiv:2406.13337, 2024)

What this evaluates

Evaluates the ability of NER models to identify and classify medically defined entity spans in Vietnamese spoken text. It specifically probes robustness to ASR-generated noise and compares monolingual vs. multilingual, encoder vs. seq2seq architectures.

Datasets

• VietMed-NER — total ?; splits: test (-1)

Metrics

• micro F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, micro-averaged across all entity instances and types. Calculated as 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Raw text sequences, either ground-truth reference transcripts or ASR-generated transcripts.

Output: Sequence of token-level entity labels indicating the start and end of named entity spans with their corresponding medical entity types.

Scoring recipe

def compute_ner_metrics(predictions, gold):
    pred_spans = extract_spans(predictions)
    gold_spans = extract_spans(gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• ASR output introduces significant noise, causing a consistent performance drop across all models compared to reference text.
• Multilingual models may suffer from capacity dilution, where per-language performance decreases as the number of pre-trained languages increases.
• Seq2seq generative models underperform encoder-only models for NER, likely because generation is less suited for span classification.

Evidence (verbatim from paper)

Table 4: NER results on ASR output of test set for different NER and ASR models. Metrics shown are Precision, Recall, and overall micro F1 score.

Citation

@misc{leduc2024medicalspokenner,
  title={Medical Spoken Named Entity Recognition},
  author={Khai Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2406.13337}
}

• arXiv: 2406.13337