Babyslm Eval

babyslm-eval

BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models — Lavechin et al. (2023) (arXiv:2306.01506, 2023)

What this evaluates

Evaluates the lexical and syntactic competence of self-supervised spoken language models using child-centered, developmentally plausible speech data. It probes whether models can acquire language-like representations from ecologically valid, in-the-wild audio recordings compared to clean audiobooks or text-based inputs.

Datasets

• BabySLM — total ?; splits: test (-1); repo https://github.com/MarvinLvn/BabySLM

Metrics

• lexical accuracy (primary) — range: percent
  • Percentage of correct predictions on lexical probing tasks. Calculated as the number of correct predictions divided by the total number of predictions, multiplied by 100.
• syntactic accuracy — range: percent
  • Percentage of correct predictions on syntactic probing tasks. Calculated as the number of correct predictions divided by the total number of predictions, multiplied by 100.

Input / output format

Input: Raw audio recordings, phoneme sequences, or orthographic words (BPE) from child-centered or clean speech corpora.

Output: Predicted lexical items or syntactic categories/tokens for probing tasks.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• Domain mismatch between training data (in-the-wild, long-form recordings) and test data (synthesized, well-articulated stimuli) causes chance-level performance for speech models.
• Speech-based models require significantly more data than text-based models to emerge above chance, following a logarithmic scaling trend rather than linear.

Evidence (verbatim from paper)

Results indicate no evidence of lexical and syntactic knowledge for STELA trained on 1,024 hours of speech from SEEDLingS. This contrasts, in appearance, with what has been found in the ZeroSpeech challenge [2], but this is due to the large variability of speech found in long-forms as we will see in Section 3.3. Results are no different for STELA trained on 128 hours of speech extracted from Providence whose lexical and syntactic accuracies remain close to chance level.

Citation

@misc{lavechin2023babyslm,
  title={BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models},
  author={Lavechin et al. (2023)},
  year={2023},
  note={arXiv:2306.01506}
}

• arXiv: 2306.01506