qhjqhj00 fgner-eval
AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers — Kaushik et al. (2026) (arXiv:2601.10161, 2026)
What this evaluates
Evaluates fine-grained named entity recognition (FgNER) capabilities across multiple languages by measuring how accurately models identify and classify specific entity types within text sequences. The protocol assesses sequence labeling performance using standard span-based metrics.
Datasets
- Various NER datasets (cited in text) — total ?; splits: (unstated)
Metrics
F1-score(primary) — range: [0, 1]- Standard sequence labeling F1-score computed over exact match of predicted and gold entity spans and types, following the evaluation protocol in Golde et al. (2025).
Input / output format
Input: Tokenized text sequences with corresponding fine-grained entity labels for sequence labeling.
Output: Predicted fine-grained entity spans and type labels per token/sequence.
Scoring recipe
def compute_f1(predictions, gold):
pred_spans = extract_spans(predictions)
gold_spans = extract_spans(gold)
tp = len(pred_spans & gold_spans)
fp = len(pred_spans - gold_spans)
fn = len(gold_spans - pred_spans)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Exact metric computation details are deferred to an external citation (Golde et al., 2025) rather than being defined inline.
- Specific NER datasets used for training/evaluation are only referenced via citations and not explicitly named or sized in the text.
Evidence (verbatim from paper)
Training was performed on an NVIDIA A100 GPU, with evaluation based on SeqEval metrics, and the best performance determined by the F1-score following Golde et al. ([2025]).
Citation
@misc{kaushik2026awedfiner,
title={AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers},
author={Kaushik et al. (2026)},
year={2026},
note={arXiv:2601.10161}
}
- arXiv: 2601.10161