Speechglue Eval

speechglue-eval

SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge? — Ashihara et al. (2023) (arXiv:2306.08374, 2023)

What this evaluates

Evaluates whether self-supervised speech models capture linguistic knowledge by probing them on a speech-adapted version of the GLUE benchmark. Tasks include grammaticality judgment, sentence similarity, paraphrase identification, and natural language inference, all converted from text to speech via TTS.

Datasets

• SpeechGLUE — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/ashi-ta/speechGLUE

Metrics

• Accuracy (Acc) (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels. Computed independently for each classification task.
• Matthews Correlation Coefficient (MCC) — range: [-1, 1]
  • Correlation coefficient between observed and predicted binary classifications, robust to class imbalance.
• Pearson Correlation Coefficient (PCC) — range: [-1, 1]
  • Linear correlation between predicted and gold continuous scores for regression tasks (e.g., STS-B).
• Spearman Correlation Coefficient (SCC) — range: [-1, 1]
  • Rank-based correlation between predicted and gold continuous scores for regression tasks.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, reported as macro-average for multi-class or binary tasks.

Input / output format

Input: 16 kHz raw audio waveform generated from text prompts using VITS TTS, with text normalization applied (symbols removed, abbreviations expanded).

Output: Predicted class label (e.g., 0/1 for binary, 0/1/2 for MNLI) or continuous regression score (0-5 for STS-B).

Scoring recipe

def compute_metrics(preds, golds, task_type):
    if task_type == 'regression':
        pcc = pearsonr(golds, preds)
        scc = spearmanr(golds, preds)
        return {'PCC': pcc, 'SCC': scc}
    else:
        acc = accuracy_score(golds, preds)
        f1 = f1_score(golds, preds, average='macro')
        mcc = matthews_corrcoef(golds, preds)
        return {'Acc': acc, 'F1': f1, 'MCC': mcc}

Common pitfalls

• Evaluation is strictly performed on the development set, not the official GLUE private test server.
• Text normalization during TTS synthesis alters original GLUE samples (e.g., removing punctuation, expanding abbreviations), which may impact performance compared to text baselines.
• For fair comparison with speech models, BERT's segment embeddings are disabled, potentially lowering its upper-bound performance.

Evidence (verbatim from paper)

Acc, MCC, PCC and SCC denote accuracy, Matthews, Pearson and Spearman correlation coefficients, respectively. The entire system was evaluated on the development set of low-resource (high-resource) tasks for every 1k (12.5k) steps, and only the highest performances are reported here.

Citation

@misc{ashihara2023speechglue,
  title={SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?},
  author={Ashihara et al. (2023)},
  year={2023},
  note={arXiv:2306.08374}
}

• arXiv: 2306.08374