Llase G1 Eval

llase-g1-eval

LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement — Kang et al. (2025) (arXiv:2503.00493, 2025)

What this evaluates

Evaluates a LLaMA-based generative speech enhancement model's ability to perform multiple audio restoration tasks (noise suppression, packet loss concealment, target speaker extraction, acoustic echo cancellation, and speech separation) in a task-agnostic manner. It probes the model's capacity to preserve acoustic fidelity and semantic content while generalizing across different acoustic conditions and device types.

Datasets

• DNS Challenge blind test set (Interspeech 2020) — total ?; splits: test (-1)
• ICASSP 2022 PLC-challenge blind test set — total ?; splits: test (-1)
• ICASSP 2023 DNS blind test set — total ?; splits: test_headset (-1), test_speakerphone (-1)
• Libri2mix test set — total ?; splits: test (-1)
• WSJ0_2mix test set — total ?; splits: test (-1)

Metrics

• DNSMOS OVRL (primary) — range: [1, 5]
  • Overall quality score from the DNSMOS predictor, ranging from 1 to 5. Higher is better.
• DNSMOS SIG/BAK — range: [1, 5]
  • Speech quality (SIG) and background noise quality (BAK) scores from the DNSMOS predictor, each ranging from 1 to 5.
• PLCMOS — range: [1, 5]
  • Predicted MOS score for packet loss concealment quality, ranging from 1 to 5.
• AECMOS EMOS/DMOS — range: [1, 5]
  • Echo annoyance (EMOS) and degradation (DMOS) scores from the AECMOS predictor, ranging from 1 to 5.
• SpeechBERTScore (SBS) — range: [0, 1]
  • Semantic similarity between enhanced and reference audio, extracted using HuBERT-base.
• SimWB — range: [0, 1]
  • Speaker similarity score computed using WavLM-base-sv embeddings.
• Subjective MOS — range: [1, 5]
  • Human-rated Mean Opinion Score on a 5-point scale (1=poor, 5=excellent).

Input / output format

Input: Degraded audio waveform (typically 4-8 seconds). For AEC, TSE, and SS tasks, reference or enrollment audio is also provided as a second channel.

Output: Enhanced audio waveform generated autoregressively via X-Codec2 tokens and decoded by the X-Codec2 decoder.

Scoring recipe

def compute_metrics(predictions, golds):
    ovrl = dns_mos.predict(golds, predictions).ovrl
    sig = dns_mos.predict(golds, predictions).sig
    bak = dns_mos.predict(golds, predictions).bak
    plcmos = plc_mos.predict(predictions)
    emos, dmos = aec_mos.predict(refs, predictions)
    sbs = speechbertscore(golds, predictions)
    simwb = wavlm_sv_similarity(golds, predictions)
    return {
        'OVRL': np.mean(ovrl),
        'SIG': np.mean(sig),
        'BAK': np.mean(bak),
        'PLCMOS': np.mean(plcmos),
        'EMOS': np.mean(emos),
        'DMOS': np.mean(dmos),
        'SBS': np.mean(sbs),
        'SimWB': np.mean(simwb)
    }

Common pitfalls

• Confusing single-inference results with multi-inference (test-time scaling) results, which can differ significantly (e.g., +0.25 OVRL for PLC).
• Using informed PLC labels during evaluation, whereas the paper evaluates a blind PLC setting without lossy labels.
• Ignoring the 'With Reverb' vs 'No Reverb' splits in the DNS test set, which yield substantially different OVRL scores.

Evidence (verbatim from paper)

We use objective metrics to evaluate the performance of the baseline systems and our model. DNSMOS include speech quality (SIG), background noise quality (BAK), and overall quality (OVRL) of the audio. AECMOS consists of echo annoyance MOS (EMOS) and other degradation MOS (DMOS). PLCMOS is used to assess the quality of audio processed by PLC algorithms. All MOS scores range from 1 to 5, representing audio quality from low to high. SpeechBERTScore (SBS) is also employed to evaluate the semantic similarity between the enhanced audio and the reference audio. ... For acoustic similarity, we calculate speaker similarity SimWB based on the WavLM-base-sv model ... Subjective evaluations are also conducted using the Mean Opinion Score (MOS) as the primary metric to assess the model’s performance.

Citation

@misc{kang2025llaseg1,
  title={LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2503.00493}
}

• arXiv: 2503.00493