Naamapadam Eval

naamapadam-eval

Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages — Mhaske et al. (2022) (arXiv:2212.10168, 2022)

What this evaluates

Evaluates Named Entity Recognition (NER) capabilities across 11 Indic languages. It probes a model's ability to identify and classify PERSON, LOCATION, and ORGANIZATION entities in low-resource and multilingual settings using projection-based and fine-tuned approaches.

Datasets

• Naamapadam — total 5700000; splits: train (-1), test (-1)

Metrics

• F1 (primary) — range: percent
  • Standard entity-level F1 score computed over correctly predicted entity spans (PERSON, LOCATION, ORGANIZATION). Precision is the ratio of correct predictions to all predicted spans, and recall is the ratio of correct predictions to all gold spans.

Input / output format

Input: Sequence of sub-word tokens generated by the mBERT tokenizer.

Output: Token-level entity labels predicted via a linear layer followed by softmax over BIO-style entity tags.

Scoring recipe

def compute_f1(pred_spans, gold_spans):
    tp = len(set(pred_spans) & set(gold_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return (2 * prec * rec / (prec + rec)) * 100 if (prec + rec) > 0 else 0.0

Common pitfalls

• Evaluates only PERSON, LOCATION, and ORGANIZATION entities; other entity types in baseline datasets are explicitly ignored or treated as non-entities.
• Results are averaged over 5 random seeds with 2 epochs of fine-tuning, so single-run scores may vary significantly.
• Zero-shot baseline uses a specific HuggingFace model (Davlan/bert-base-multilingual-cased-ner-hrl) fine-tuned on 10 high-resource languages, not a generic multilingual model.

Evidence (verbatim from paper)

We tune hyper-parameters based on F1-Score on the validation set. We use the following range of values for selecting the best hyper-parameter. Once we obtain the best hyper-parameter, we fine-tune the model for 2 epochs with 5 different random seeds. We report the mean and standard deviation of the 5 runs.

Citation

@misc{mhaske2022naamapadam,
  title={Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages},
  author={Mhaske et al. (2022)},
  year={2022},
  note={arXiv:2212.10168}
}

• arXiv: 2212.10168