Tlunified Ner Eval

tlunified-ner-eval

Developing a Named Entity Recognition Dataset for Tagalog — Miranda (2023) (arXiv:2311.07161, 2023)

What this evaluates

Evaluates Named Entity Recognition (NER) capabilities on Tagalog news text, specifically measuring performance across Person, Organization, and Location entities using supervised learning and zero-shot LLM prompting.

Datasets

• TLUNIFIED-NER — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ljvmiranda921/calamanCy

Metrics

• F1-score (primary) — range: [0, 1]
  • Standard token-level F1-score for exact span matching, computed per entity class (Person, Organization, Location) and overall. Calculated as the harmonic mean of precision and recall.

Input / output format

Input: Raw Tagalog text sequences from news documents.

Output: BILUO sequence encoding scheme (e.g., B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, O) for each token.

Scoring recipe

def compute_f1(gold_spans, pred_spans):
    tp = len(gold_spans & pred_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Models frequently confuse lexical vs. semantic entity tags (e.g., 'Ombudsman' as Person vs Organization).
• Zero-shot LLM prompting exhibits high variance and consistently underperforms supervised baselines on this corpus.
• Evaluation relies on exact span matching; boundary mismatches or incorrect BIO/BILUO formatting will artificially deflate scores.

Evidence (verbatim from paper)

Table 4 reports the F1-score on the test set across three trials. We trained several NER models using spaCy's transition-based parser (Honnibal et al., 2020). The state transitions are based on the BILUO sequence encoding scheme and the actions are decided by a convolutional neural network with a maxout (Goodfellow et al., 2013) activation function.

Citation

@misc{miranda2023developing,
  title={Developing a Named Entity Recognition Dataset for Tagalog},
  author={Miranda (2023)},
  year={2023},
  note={arXiv:2311.07161}
}

• arXiv: 2311.07161