qhjqhj00 libricss-eval
Localization Based Sequential Grouping for Continuous Speech Separation — Wang et al. (2021) (arXiv:2107.06853, 2021)
What this evaluates
Evaluates continuous speech separation and speaker diarization in reverberant, multi-microphone environments with varying speaker overlap ratios. It probes the system's ability to separate overlapping speech, estimate speaker locations (DOA), cluster them across time blocks, and produce accurate diarization and speech recognition outputs.
Datasets
- LibriCSS — total ?; splits: test (-1)
Metrics
DER(primary) — range: percent- Diarization Error Rate: (Miss + False Alarm + Confusion) / Total Speech Time, reported as a percentage. Measures the proportion of speech time incorrectly assigned to the wrong speaker, missed, or falsely detected.
cpWER— range: percent- Concatenated minimum-permutation Word Error Rate: utterances per speaker are concatenated in hypothesis and reference, all speaker pairs are scored, and the permutation yielding the lowest WER is selected. Mismatched speaker counts (A vs B) are handled by aligning min(A,B) pairs and treating unaligned references as deletions when A < B.
Input / output format
Input: Multi-channel audio recordings (7 microphones) of conversational speech with controlled overlap ratios (0% to 40%), recorded in reverberant rooms. Evaluated either session-wise (entire 10-minute mini-session) or segment-wise (pre-segmented 5- to 120-second chunks).
Output: Separated speaker audio streams, speaker diarization labels (speaker identity per time segment), and ASR transcriptions for each separated stream.
Scoring recipe
def compute_der(ref_labels, hyp_labels, total_speech_time):
miss = count_missed_speech(ref_labels, hyp_labels)
fa = count_false_alarms(ref_labels, hyp_labels)
conf = count_confused_speech(ref_labels, hyp_labels)
return (miss + fa + conf) / total_speech_time * 100
def compute_cpwer(hyp_streams, ref_streams):
hyp_concat = {spk: concat_utterances(hyp_streams[spk]) for spk in hyp_streams}
ref_concat = {spk: concat_utterances(ref_streams[spk]) for spk in ref_streams}
best_wer = float('inf')
for perm in permutations(hyp_concat.keys(), len(ref_concat)):
wer = average_wer([hyp_concat[h] for h in perm], ref_concat.values())
best_wer = min(best_wer, wer)
if len(hyp_concat) < len(ref_concat):
best_wer += deletion_penalty_for_unaligned_refs
return best_wer * 100
Common pitfalls
- Mismatched speaker counts (A vs B) in cpWER: unaligned reference utterages are treated as deletion errors when A < B, while extra hypotheses (A > B) are simply ignored rather than penalized.
- Evaluation granularity: Results differ significantly between session-wise (entire 10-min mini-session) and segment-wise (5-120s pre-segmented chunks) protocols, so metrics must not be mixed.
- Overlap ratio conditions: 0S (short silence) and 0L (long silence) behave differently from actual overlaps (10-40%), affecting both DER and cpWER performance.
Evidence (verbatim from paper)
Diarization performance is measured using diarization error rates (DER). We report ASR performance using concatenated minimum-permutation word error rates (cpWER) [8]. It is computed by concatenating all the utterances of each speaker in the hypothesis and reference, scoring all speaker pairs, and finding the permutation that produces the best WER. Note that the estimated number of speakers, A, could be different from the actual number of speakers, B. When A < B, we align the A hypotheses to the references and consider all the rest B - A references to produce deletion errors. When A > B, we can only align B hypotheses with the references and compute the WER. We do not score the rest A - B hypotheses.
Citation
@misc{wang2021localization,
title={Localization Based Sequential Grouping for Continuous Speech Separation},
author={Wang et al. (2021)},
year={2021},
note={arXiv:2107.06853}
}
- arXiv: 2107.06853