Sentence Stress Detection Eval

sentence-stress-detection-eval

WHISTRESS: Enriching Transcriptions with Sentence Stress Detection — Yosha et al. (2025) (arXiv:2505.19103, 2025)

What this evaluates

Probes a model's ability to detect sentence-level prosodic stress at the word/token level using only audio input. It evaluates zero-shot generalization and alignment-free stress identification across diverse speech styles and synthetic/human datasets.

Datasets

• TinyStress-15K — total 15000; splits: train (-1), val (-1)
• Aix-MARSEC — total 2400; splits: train (1680), test (720)
• Expresso — total ?; splits: test (-1)
• EmphAssess — total 3652; splits: train (1500), test (2152)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard binary classification F1 score: 2 * (Precision * Recall) / (Precision + Recall). Precision and recall are computed at the word level, where a word is labeled stressed if at least one of its tokens is predicted as stressed.

Input / output format

Input: Raw audio recordings. Transcriptions are not required during inference.

Output: Binary stress label per token, aggregated to word-level if at least one token is marked stressed.

Scoring recipe

def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• Confusing token-level predictions with word-level labels: the paper explicitly aggregates token predictions to word-level by marking a word stressed if at least one of its tokens is predicted as stressed.
• Assuming forced alignment or timestamps are required for inference: the model is explicitly alignment-free; baselines use MFA or ground-truth timestamps, but WhiStress does not.
• Mixing up zero-shot vs in-domain evaluation splits: zero-shot uses TinyStress-15K for training and evaluates on Expresso/EmphAssess; in-domain training on EmphAssess uses speakers ex03/ex04 for training and ex01/ex02 for testing.

Evidence (verbatim from paper)

We report performance using standard classification metrics: precision, recall and F$1$ score. In our settings, a word is considered stressed by the WhiStress model if at least one of its tokens is marked as stressed.

Citation

@misc{yosha2025whistress,
  title={WHISTRESS: Enriching Transcriptions with Sentence Stress Detection},
  author={Yosha et al. (2025)},
  year={2025},
  note={arXiv:2505.19103}
}

• arXiv: 2505.19103