Urdu Mner Eval

urdu-mner-eval

A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition — Ahmad et al. (2025) (arXiv:2505.05148, 2025)

What this evaluates

Evaluates the ability of models to recognize named entities (Person, Location, Organization, Miscellaneous) in Urdu social media posts by jointly processing textual and visual inputs. It probes cross-modal alignment, handling of low-resource language morphological complexity, and ambiguity resolution using visual context.

Datasets

• Twitter2015-Urdu — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 score (primary) — range: percent
  • Standard token-level or span-level precision, recall, and F1 for named entity recognition. F1 = 2 * (Precision * Recall) / (Precision + Recall). Evaluated per entity type and overall.

Input / output format

Input: Paired Urdu tweet text and associated image. Text is tokenized to a maximum of 128 tokens; images are processed via a fixed-weight ResNet-152.

Output: Sequence of BIO/IOB entity labels for each token, corresponding to classes: PER, LOC, ORG, MISC, and O.

Scoring recipe

def compute_ner_f1(preds, golds):
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        if set(p) == set(g) and p:
            tp += 1
        elif p and g:
            fp += 1
        elif g:
            fn += 1
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return f1

Common pitfalls

• Urdu lacks capitalization, making entity boundary detection significantly harder without visual context or character-level features.
• Visual noise (e.g., concert images for 'Google') can mislead models that lack proper cross-modal gating or attention mechanisms.
• Low-resource nature requires careful fine-tuning of pre-trained models (e.g., Urdu-BERT) rather than using standard English BERT out-of-the-box.

Evidence (verbatim from paper)

To assess the performance of the MNER models, we utilize the F1 score for each entity type, along with the overall precision (P), recall (R), and F1 score (F1). These metrics provide a comprehensive evaluation of model accuracy and effectiveness.

Citation

@misc{ahmad2025urdu,
  title={A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition},
  author={Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2505.05148}
}

• arXiv: 2505.05148