Seniortalk Eval

seniortalk-eval

SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniors — Yang Chen et al. (2025) (arXiv:2503.16578, 2025)

What this evaluates

Evaluates speech processing models on authentic, real-world conversations among super-aged Chinese speakers (75+). It probes capabilities in speaker verification, diarization, automatic speech recognition, and speech editing under conditions of age-related vocal degradation, dialectal variation, and presbyphonia.

Datasets

• SeniorTalk — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• EER (primary) — range: percent
  • Equal Error Rate: the threshold τ where the miss probability equals the false alarm probability. The system accepts speakers as the same if similarity > τ, rejects if < τ.
• minDCF — range: percent
  • Minimum Detection Cost: a cost-sensitive metric evaluating speaker verification systems under application-specific conditions.
• DER — range: percent
  • Diarization Error Rate: DER = (FA + MD + Conf) / T, where FA is false alarm segments, MD is missed detection segments, Conf is confusion segments, and T is total reference speech segments.
• CER — range: percent
  • Character Error Rate: CER = (S + D + I) / N, where S, D, I are substitutions, deletions, and insertions, and N is the total number of characters in the reference text.

Input / output format

Input: Raw audio recordings of natural conversations among super-aged speakers, accompanied by reference transcriptions and speaker annotations.

Output: For verification/diarization: speaker identity labels or segment boundaries. For ASR: character-level transcriptions. For editing: modified audio waveforms.

Scoring recipe

def compute_cer(predictions, references):
    total_errors = 0
    total_chars = 0
    for pred, ref in zip(predictions, references):
        # Compute edit distance (substitutions, deletions, insertions)
        edits = levenshtein_distance(pred, ref)
        total_errors += edits
        total_chars += len(ref)
    return (total_errors / total_chars) * 100 if total_chars > 0 else 0.0

Common pitfalls

• Age-related vocal degradation (pitch instability, articulatory imprecision) significantly impacts model performance compared to standard adult speech benchmarks.
• Dialectal variation and regional accents cause substantial CER increases, with heavy accents pushing errors above 80% for some models.
• Gender imbalance (1:3 female-to-male ratio) in the dataset reduces the saliency of secondary sexual voice characteristics, complicating speaker separation.

Evidence (verbatim from paper)

We adopt two scoring approaches: probabilistic linear discriminant analysis (PLDA) and cosine similarity, with evaluation based on two metrics: (1) Equal Error Rate (EER): We define a threshold τ where the miss probability equals the false alarm probability. Specifically, if the similarity score is above this threshold, the system accepts that the speakers are the same person; if it is below this threshold, the system rejects the claim. This threshold is selected when the false acceptance rate equals the false rejection rate. (2) Minimum Detection Cost (minDCF): A cost-sensitive metric for evaluating speaker verification systems under application-specific conditions.

Citation

@misc{chen2025seniortalk,
  title={SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniors},
  author={Yang Chen et al. (2025)},
  year={2025},
  note={arXiv:2503.16578}
}

• arXiv: 2503.16578