Cv 18 Ner Eval

cv-18-ner-eval

CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech — Saidi et al. (2026) (arXiv:2604.02209, 2026)

What this evaluates

Evaluates end-to-end and cascaded named entity recognition from Arabic speech. It probes a model's ability to jointly transcribe spoken Arabic and predict fine-grained entity types (21 categories) using BIO-style tagging, as well as extract entity spans and values.

Datasets

• CV-18 NER — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• CoER (primary) — range: percent
  • Concept Error Rate. Computed as (S+D+I)/N, where S, D, I are substitutions, deletions, and insertions, and N is the number of reference entity labels. Evaluates correctness of entity labels (e.g., B-PERS, I-ORG) independently of lexical content.
• CVER — range: percent
  • Concept-Value Error Rate. Computed as (S+D+I)/N over tag-value pairs. Evaluates both entity type and entity span by comparing extracted pairs from predicted and reference transcripts.
• WER — range: percent
  • Word Error Rate. Computed as (S+D+I)/N over words, after removing special entity tokens from model outputs.
• Micro-averaged F1-score — range: percent
  • Standard micro-averaged F1 computed over token-level BIO predictions for text-based NER systems.

Input / output format

Input: Audio signals resampled to 16 kHz for speech models; normalized text transcriptions (punctuation/diacritics removed, Eastern numerals transliterated) for text-based NER models.

Output: For speech models: concatenated transcription with entity tags as single atomic special tokens (e.g., B-PERS, I-ORG). For text models: BIO-style token-level labels aligned with input sentences.

Scoring recipe

def compute_error_rate(pred_tokens, ref_tokens):
    S, D, I = edit_distance_operations(pred_tokens, ref_tokens)
    N = len(ref_tokens)
    return (S + D + I) / N

def compute_coer(pred_tags, ref_tags):
    return compute_error_rate(pred_tags, ref_tags)

def compute_cver(pred_pairs, ref_pairs):
    return compute_error_rate(pred_pairs, ref_pairs)

# WER is computed on word tokens after stripping entity tags.
# F1 is computed via standard micro-averaging over token labels.

Common pitfalls

• Transcription errors in ASR directly degrade downstream NER performance, but lower WER does not guarantee better CoER/CVER due to differences in error types.
• Entity tags are treated as single atomic tokens in the vocabulary; they must be removed before computing WER to avoid skewing results.
• Performance drops sharply for low-frequency entity types (e.g., PRODUCT, LAW) compared to high-frequency ones (e.g., LANGUAGE, GPE).

Evidence (verbatim from paper)

Named entity recognition performance is evaluated using Concept Error Rate (CoER) and Concept-Value Error Rate (CVER), which are adapted from spoken language understanding evaluation. WER, CoER, and CVER follow the standard error-rate formulation: Error Rate = (S+D+I)/N, where S, D, and I denote substitutions, deletions, and insertions, and N is the number of reference units. CoER evaluates the correctness of entity labels only (e.g., B-PERS, I-ORG), independently of the lexical content. CVER evaluates both entity type and entity span by comparing tag–value pairs extracted from the annotated transcripts.

Citation

@misc{saidi2026cv18ner,
  title={CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech},
  author={Saidi et al. (2026)},
  year={2026},
  note={arXiv:2604.02209}
}

• arXiv: 2604.02209