hiner-ner-eval
HiNER: A Large Hindi Named Entity Recognition Dataset — Murthy et al. (2022) (arXiv:2204.13743, 2022)
What this evaluates
This benchmark evaluates a model's ability to perform Named Entity Recognition (NER) on Hindi text. It probes the model's capacity to identify and classify entity spans (e.g., Person, Location, Organization, and others) in a language characterized by free word order, lack of capitalization, and spelling variations.
Datasets
- HiNER — total ?; splits: train (-1), dev (-1), test (-1)
Metrics
F1-Score (primary) — range: percent
- Entity-level F1 score computed via exact match of BIO-tagged spans. Precision is the fraction of predicted entities that are correct, Recall is the fraction of gold entities that are predicted, and F1 is their harmonic mean. Reported as Micro, Macro, and Weighted averages across 11 entity tags.
Input / output format
Input: Tokenized Hindi sentences formatted with I-O-B (BIO) sequence labels.
Output: Sequence of BIO tags (e.g., B-PER, I-LOC, O) corresponding to each input token.
Scoring recipe
from seqeval.metrics import f1_score
# predictions and gold are lists of lists of BIO tags per sentence
micro_f1 = f1_score(gold, predictions, average='micro')
macro_f1 = f1_score(gold, predictions, average='macro')
weighted_f1 = f1_score(gold, predictions, average='weighted')
Common pitfalls
- Hindi lacks capitalization and has free word order, making entity boundary detection significantly harder than in English.
- The dataset supports 11 entity tags, but many baselines collapse them to 3 (Person, Location, Organization); results must be reported for both configurations.
- Evaluation uses entity-level exact matching via Seqeval, not token-level accuracy, so partial span matches are not counted as correct.
Evidence (verbatim from paper)
We use the I-O-B encoding as input format for model training and report the results using Seqeval (Nakayama, 2018) generate evaluation statistics. Table 5: Test Set F1-Score of various pre-trained LMs on our HiNER dataset. This table reports a mean F1-score and its standard deviation over 5 runs.
Citation
@misc{murthy2022hiner,
title={HiNER: A Large Hindi Named Entity Recognition Dataset},
author={Murthy et al. (2022)},
year={2022},
note={arXiv:2204.13743}
}
1---2name: hiner-ner-eval3description: This benchmark evaluates a model's ability to perform Named Entity Recognition (NER) on Hindi text. It probes the model's capacity to identify and classify entity spans (e.g., Person, Location, Organization, and others) in a language characterized by free word order, lack of capitalization, and spelling variations. Use when the user wants to benchmark on HiNER, or asks about evaluating this task. Reports F1-Score.4---56# hiner-ner-eval78> HiNER: A Large Hindi Named Entity Recognition Dataset — Murthy et al. (2022) (arXiv:2204.13743, 2022)910## What this evaluates1112This benchmark evaluates a model's ability to perform Named Entity Recognition (NER) on Hindi text. It probes the model's capacity to identify and classify entity spans (e.g., Person, Location, Organization, and others) in a language characterized by free word order, lack of capitalization, and spelling variations.1314## Datasets1516- **HiNER** — total ?; splits: train (-1), dev (-1), test (-1)1718## Metrics1920- `F1-Score` **(primary)** — range: percent21 - Entity-level F1 score computed via exact match of BIO-tagged spans. Precision is the fraction of predicted entities that are correct, Recall is the fraction of gold entities that are predicted, and F1 is their harmonic mean. Reported as Micro, Macro, and Weighted averages across 11 entity tags.2223## Input / output format2425**Input**: Tokenized Hindi sentences formatted with I-O-B (BIO) sequence labels.2627**Output**: Sequence of BIO tags (e.g., B-PER, I-LOC, O) corresponding to each input token.2829## Scoring recipe3031```python32from seqeval.metrics import f1_score33# predictions and gold are lists of lists of BIO tags per sentence34micro_f1 = f1_score(gold, predictions, average='micro')35macro_f1 = f1_score(gold, predictions, average='macro')36weighted_f1 = f1_score(gold, predictions, average='weighted')37```3839## Common pitfalls4041- Hindi lacks capitalization and has free word order, making entity boundary detection significantly harder than in English.42- The dataset supports 11 entity tags, but many baselines collapse them to 3 (Person, Location, Organization); results must be reported for both configurations.43- Evaluation uses entity-level exact matching via Seqeval, not token-level accuracy, so partial span matches are not counted as correct.4445## Evidence (verbatim from paper)4647> We use the I-O-B encoding as input format for model training and report the results using Seqeval (Nakayama, 2018) generate evaluation statistics. Table 5: Test Set F1-Score of various pre-trained LMs on our HiNER dataset. This table reports a mean F1-score and its standard deviation over 5 runs.4849## Citation5051```bibtex52@misc{murthy2022hiner,53 title={HiNER: A Large Hindi Named Entity Recognition Dataset},54 author={Murthy et al. (2022)},55 year={2022},56 note={arXiv:2204.13743}57}58```5960- arXiv: 2204.13743
qhjqhj00