Contextual Earnings 22 Eval

contextual-earnings-22-eval

Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild — Durmus et al. (2026) (arXiv:2604.07354, 2026)

What this evaluates

Evaluates speech-to-text systems on their ability to correctly recognize domain-specific custom vocabulary (e.g., company names, products) in real-world earnings call audio. It probes how well models leverage provided keyword contexts (local vs. global/noisy) to improve keyword recognition without introducing transcription artifacts.

Datasets

• Contextual Earnings-22 — total ?; splits: val (-1)

Metrics

• keyword F-score (primary) — range: [0, 1]
  • F-score computed over a predefined set of custom keywords: F = 2 * (precision * recall) / (precision + recall), where precision and recall measure the fraction of correctly recognized keywords in the predicted transcript relative to the ground truth and the keyword list, respectively.
• WER — range: [0, 1]
  • Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the ground truth transcript, normalized by the number of words in the ground truth.

Input / output format

Input: Audio clips from earnings calls paired with a context list of custom keywords (either concise/local or noisy/global with distractors).

Output: Transcribed text output from the STT system.

Scoring recipe

def score(predictions, golds, keywords):
    wer = compute_wer(predictions, golds)
    tp = fp = fn = 0
    for pred, gold in zip(predictions, golds):
        for kw in keywords:
            in_pred = kw in pred
            in_gold = kw in gold
            if in_pred and in_gold: tp += 1
            elif in_pred and not in_gold: fp += 1
            elif not in_pred and in_gold: fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'wer': wer, 'keyword_precision': precision, 'keyword_recall': recall, 'keyword_f1': f1}

Common pitfalls

• WER alone is insufficient for this benchmark because context biasing can improve keyword F-score while simultaneously increasing WER due to hallucinations or false positives.
• Global context evaluation specifically tests robustness to distractor-induced false positives, which heavily penalizes precision and differs fundamentally from local context evaluation.
• Hyperparameters for keyword boosting methods are calibrated on the validation split, which may not generalize to out-of-distribution context formats.

Evidence (verbatim from paper)

We evaluate six STT systems under no, local, and global context, reporting WER and keyword F-score (precision/recall). ... Results show both approaches achieve significantly improved accuracy when scaled, revealing that WER alone fails to capture real-world usability, and highlighting the critical role of context biasing in deployment performance.

Citation

@misc{durmus2026contextual,
  title={Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild},
  author={Durmus et al. (2026)},
  year={2026},
  note={arXiv:2604.07354}
}

• arXiv: 2604.07354