Easycom Eval

easycom-eval

EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments — Donley et al. (2021) (arXiv:2107.04174, 2021)

What this evaluates

Evaluates real-time, dynamic audio-visual speech enhancement and beamforming systems in noisy, egocentric augmented reality settings. It probes the model's ability to isolate a target speaker's voice from competing talkers and background noise while preserving speech quality and intelligibility across diverse user movements.

Datasets

• EasyCom — total ?; splits: test (-1); repo https://github.com/facebookresearch/EasyComDataset

Metrics

• SNR (primary) — range: other
  • Ratio of the desired target source signal to all other sounds (treated as noise). Computed over the full signal.
• SegSNR — range: other
  • Segmental SNR; computes the mean SNR only over segments where the target source signal is active.
• SDR — range: other
  • Signal to Distortion Ratio; similar to SNR but computed using a specific implementation that accounts for distortion components.
• SI-SDR — range: other
  • Scale-Invariant SDR; a variant of SDR designed to address scale assumptions in the standard SDR implementation.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility; computed on short-time segments and highly correlated with time-frequency weighted noisy speech.
• ESTOI — range: [0, 1]
  • Extended STOI; a modern variant that accurately predicts intelligibility even when highly modulated noise is present.
• HASPI — range: [0, 1]
  • Hearing-Aid Speech Perception Index v2; estimates intelligibility using an auditory periphery model, valid for normal-hearing and hearing-impaired listeners.
• SIIB — range: other
  • Speech Intelligibility In Bits; estimates the amount of information shared between talker and listener in bits, showing higher correlation to intelligibility than STOI/ESTOI/HASPI v1.
• PESQ — range: other
  • Perceptual Evaluation of Speech Quality; widely used for speech quality evaluations, originally designed for telephony applications.
• HASQI — range: other
  • Hearing-Aid Speech Quality Index v2; evaluates quality using an auditory periphery model that accounts for hearing loss effects.
• ViSQOL — range: other
  • Virtual Speech Quality Objective Listener v3; shows higher correlation to speech quality than PESQ on several datasets.

Input / output format

Input: Synchronized 8–10 channel audio, wide-field RGB video, head/face bounding boxes, and source-level annotations captured by egocentric AR glasses. For metric computation, the enhanced audio signal is compared against a close-microphone reference signal.

Output: Enhanced audio signal (beamformed output) for each target speaker instance.

Scoring recipe

def evaluate(predictions, references):
    scores = {}
    for metric in ['SNR', 'SegSNR', 'SDR', 'SI-SDR', 'STOI', 'ESTOI', 'HASPI', 'SIIB', 'PESQ', 'HASQI', 'ViSQOL']:
        scores[metric] = compute_metric(metric, predictions, references)
    # Filter: ignore frames where target participant is actively talking
    # Split into 'Noise' and 'Noise + Interferer' test cases using VA labels
    avg_scores = {}
    for test_case in ['Noise', 'Noise + Interferer']:
        avg_scores[test_case] = mean(scores[test_case])
    return avg_scores

Common pitfalls

• The clean reference used is a close-microphone signal, not the true isolated clean speech component, so absolute metric values may not be directly comparable to standard datasets.
• Evaluations must explicitly exclude frames where the target participant is actively talking, and results are reported separately for 'Noise' and 'Noise + Interferer' conditions.
• PESQ may not improve over the reference microphone in 'Noise + Interferer' cases due to interfering speech leaking into the reference signal provided to the metric.

Evidence (verbatim from paper)

We use 11 intrusive instrumental objective metrics as measures of the performance. Three metrics are related to speech quality, four metrics are related to speech intelligibility and four are related to SNR. We use SNR as one of the metrics [20], which is well established in the literature. We define the SNR as the ratio of the desired target source signal to all other sounds... The metric scores are obtained on the EasyCom dataset for all possible target cases... Two subsets of signals are evaluated as test cases for all metrics, namely 'Noise' and 'Noise + Interferer'... We ignore cases where the participant wearing the AR glasses is actively talking.

Citation

@misc{donley2021easycom,
  title={EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments},
  author={Donley et al. (2021)},
  year={2021},
  note={arXiv:2107.04174}
}

• arXiv: 2107.04174