L2 Arctic Mispronunciation Eval

l2-arctic-mispronunciation-eval

Pronunciation Deviation Analysis Through Voice Cloning and Acoustic Comparison — Valdivia et al. (2025) (arXiv:2507.10985, 2025)

What this evaluates

Probes the ability to detect phoneme-level mispronunciations in second-language (L2) speech by comparing acoustic features of learner speech against a voice-cloned native reference using dynamic time warping on MFCC envelopes.

Datasets

• L2-Arctic — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances (both correct and mispronounced) out of the total number of instances.
• precision — range: [0, 1]
  • Ratio of true positive mispronunciations detected to the total number of instances predicted as mispronunciations.
• recall — range: [0, 1]
  • Ratio of true positive mispronunciations detected to the total number of actual mispronunciations in the ground truth.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Per-instance speech audio of a second-language learner reading a target text, processed into MFCC envelopes and compared against a voice-cloned native reference via dynamic time warping to yield an acoustic deviation distance.

Output: Binary classification label (0: correct, 1: mispronunciation) per instance, derived by comparing the acoustic deviation distance against a speaker-specific calibrated threshold.

Scoring recipe

tp = sum((pred == 1) & (gold == 1))
fp = sum((pred == 1) & (gold == 0))
fn = sum((pred == 0) & (gold == 1))
tn = sum((pred == 0) & (gold == 0))
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
accuracy = (tp + tn) / (tp + tn + fp + fn)

Common pitfalls

• The evaluation uses speaker-specific acoustic distance thresholds calibrated per individual, so applying a fixed threshold across different speakers or datasets will yield invalid results.
• Class support is uneven across individuals (e.g., EBVS has 110 vs 133 samples), meaning precision and recall are highly sensitive to the chosen decision boundary and may not reflect balanced performance.
• The binary labels (0/1) are derived from continuous DTW distance scores, so the metric does not evaluate raw distance values but rather thresholded classification decisions.

Evidence (verbatim from paper)

From the L2-Arctic dataset, our model was applied to four individuals: EBVS, ERMS, MBMPS, and NJS. Classification performance was measured using precision, recall, F1-score, and accuracy.

Citation

@misc{valdivia2025pronunciation,
  title={Pronunciation Deviation Analysis Through Voice Cloning and Acoustic Comparison},
  author={Valdivia et al. (2025)},
  year={2025},
  note={arXiv:2507.10985}
}

• arXiv: 2507.10985