Biouner Eval

biouner-eval

BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition — Wazir Ali et al. (2026) (arXiv:2604.02904, 2026)

What this evaluates

Evaluates the ability of models to perform clinical named entity recognition in Urdu, specifically identifying and classifying biomedical entities like diseases, genes, and proteins within clinical text sequences. It probes sequence labeling capabilities in a low-resource, domain-specific language setting.

Datasets

• BioUNER — total 153000; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard token-level F1 score for sequence labeling, calculated as the harmonic mean of precision and recall over all correctly predicted entity tokens.

Input / output format

Input: A sequence of tokens X = {x_1, x_2, ..., x_N} representing a clinical Urdu sentence.

Output: A sequence of labels Y = {y_1, y_2, ..., y_N} where each y_i is a BIES-tagged entity class (e.g., Disease, Gene, Protein) or O.

Scoring recipe

def calculate_f1(predictions, gold):
    pred_spans = extract_entity_spans(predictions)
    gold_spans = extract_entity_spans(gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Token-level vs. entity-level matching: NER evaluation requires exact span matching rather than simple token-level accuracy.
• BIES labeling scheme must be strictly followed; misinterpreting B (Begin), I (Inside), E (End), S (Single) tags leads to incorrect entity boundary detection.
• Low-resource language effects: Urdu's script and morphology may cause tokenization mismatches if subword tokenizers are not aligned with the annotation guidelines.

Evidence (verbatim from paper)

Formally, the task is to learn a mapping from the input sequence X to an output label sequence Y = {y_1,y_2,...,y_N}, where each label y_i ∈ V and V denotes a predefined set of NE labels such as Disease, Gene, and Protein. In order to comprehensively evaluate the newly proposed BioUNER dataset, we exploit Conditional Random Fields (CRF)... Support Vector Machines (SVM)... LSTM... mBERT... and XLM-RoBERTa... Models are evaluated using F1 score.

Citation

@misc{wazirali2026biouner,
  title={BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition},
  author={Wazir Ali et al. (2026)},
  year={2026},
  note={arXiv:2604.02904}
}

• arXiv: 2604.02904