Persian Ner Eval

persian-ner-eval

Exploring the Potential of Machine Translation for Generating Named Entity Datasets: A Case Study between Persian and English — Amir Sartipi et al. (2023) (arXiv:2302.09611, 2023)

What this evaluates

Evaluates the quality and cross-lingual transferability of machine-translated Persian named entity recognition datasets by measuring model performance against original English benchmarks. It probes how well translation-based dataset generation preserves entity boundaries and labels across languages with different scripts and linguistic structures.

Datasets

• CoNLL 2003 — total ?; splits: test (-1)
• OntoNotes 5.0 — total ?; splits: test (-1)
• NCBI Disease — total ?; splits: test (-1)
• WNUT 2017 — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of precision and recall: 2 * (P * R) / (P + R). Reported as a percentage in the results table.
• Precision — range: percent
  • Ratio of correctly predicted named entities to all predicted entities.
• Recall — range: percent
  • Ratio of correctly predicted named entities to all actual named entities in the gold standard.

Input / output format

Input: Raw text sequences (sentences/documents) in English or Persian.

Output: Token-level or span-level named entity labels (e.g., B-PER, I-LOC, O).

Scoring recipe

def compute_ner_f1(pred_spans, gold_spans):
    pred_set = set(pred_spans)
    gold_set = set(gold_spans)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100

Common pitfalls

• Translation-based dataset generation often introduces surface form mismatches and translation errors that disproportionately hurt performance on noisy or novel entity types (e.g., WNUT 2017).
• Performance varies significantly between monolingual (Pars-Bert) and multilingual (xlm-roberta-base) models across datasets, making cross-model comparisons non-trivial without controlling for language coverage.
• F1 scores are reported as percentages in the results table, which can cause confusion if compared against decimal-scale benchmarks or if not explicitly converted.

Evidence (verbatim from paper)

Table 2 presents the results of our evaluation for both the English and translated versions of the main datasets. The rows where the Model column is Δ tar-src show the difference between the English and Persian datasets. Our results indicate that the largest discrepancy between the source and target datasets in terms of F1 score was observed for the WNUT 2017 dataset, while the smallest difference was noted for the NCBI Disease dataset.

Citation

@misc{sartipi2023mtner,
  title={Exploring the Potential of Machine Translation for Generating Named Entity Datasets: A Case Study between Persian and English},
  author={Amir Sartipi et al. (2023)},
  year={2023},
  note={arXiv:2302.09611}
}

• arXiv: 2302.09611