Discophon Eval

discophon-eval

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units — Poli et al. (2026) (arXiv:2603.18612, 2026)

What this evaluates

This benchmark evaluates the ability of discrete speech models to unsupervisedly discover phoneme inventories and capture phonemic contrasts. It measures how well predicted units align with gold phonemes through phonetic similarity, recognition error, and temporal segmentation accuracy across multiple languages.

Datasets

• discoPhon — total ?; splits: dev (-1), test (-1)

Metrics

• PNMI (primary) — range: [0, 1]
  • Normalized Mutual Information: I(p;u)/H(p) = [sum_{i,j} P(i,j) log(P(i,j)/(P_p(i)P_u(j)))] / [sum_i P_p(i) log P_p(i)]. Measures the fraction of phone entropy explained by discrete units.
• PER — range: percent
  • Phone Error Rate computed via edit distance between the mapped predicted phone sequence and the gold phone sequence, abstracting away from temporal alignment.
• R-value — range: percent
  • Segmentation metric comparing predicted and gold boundary positions. Penalizes over-segmentation more than F1. Uses a ±20ms tolerance window around gold boundaries.
• F1 — range: percent
  • Segmentation metric comparing predicted and gold boundary positions with a ±20ms tolerance window. Overlapping windows are split at their midpoint.

Input / output format

Input: Sequence of discrete units u = (u_1,...,u_T) and corresponding gold phoneme sequence p = (p_1,...,p_T) over T time steps.

Output: Mapped phoneme sequence a = (a_1,...,a_T) where each a_t is assigned to unit u_t via many-to-one (most frequent phoneme) or one-to-one (bijection via linear assignment) mapping.

Scoring recipe

# Compute joint distribution P(i, j) from units and phones
P = joint_distribution(units, phones)
# Many-to-one mapping
mapping = {u: argmax_i P[i, u] for u in units}
predicted_phones = [mapping[u] for u in units]
# PNMI
pnmi = mutual_information(phones, units) / entropy(phones)
# PER
per = phone_error_rate(predicted_phones, phones)
# Segmentation (F1, R-value) with ±20ms tolerance
f1, r_val = segmentation_metrics(predicted_boundaries, gold_boundaries, tolerance_ms=20)

Common pitfalls

• Failing to fix the vocabulary size (e.g., to 256 units) allows trivial perfect mappings by increasing unit count, invalidating fair comparison.
• PNMI is sensitive to both unit quality and temporal alignment; relying solely on it obscures whether errors stem from poor units or misalignment.
• Segmentation evaluation requires a strict ±20ms tolerance window around gold boundaries; ignoring this tolerance leads to incorrect F1/R-value scores.

Evidence (verbatim from paper)

We first assess unit quality with PNMI [30]. We then derive a mapping from units to phonemes using the gold annotations, as illustrated in Figure 1. This produces a phonetic transcription, which we evaluate for recognition with Phone Error Rate (PER) and segmentation with $R$ -value and $F_{1}$ .

Citation

@misc{poli2026discophon,
  title={DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units},
  author={Poli et al. (2026)},
  year={2026},
  note={arXiv:2603.18612}
}

• arXiv: 2603.18612