qhjqhj00 sentencebench-eval
Fast, Not Fancy: Rethinking G2P with Rich Data and Rule-Based Models — Mahta Fetrat Qharabagh, Zahra Dehghanian, and Hamid R. Rabiee (2025) (arXiv:2505.12973, 2025)
What this evaluates
This benchmark evaluates Persian grapheme-to-phoneme (G2P) systems on sentence-level text, specifically probing their ability to correctly map characters to phonemes and disambiguate homographs using contextual information. It measures both phonetic accuracy and contextual word-sense resolution capabilities.
Datasets
- SentenceBench — total ?; splits: test (-1); repo https://github.com/MahtaFetrat/Persian-G2P-Tools-Benchmark
Metrics
PER (%)— range: percent- Phoneme Error Rate, calculated as the normalized edit distance between predicted and reference phoneme sequences.
Homograph Acc. (%)(primary) — range: percent- Percentage of homograph words correctly assigned the contextually appropriate phoneme sequence.
Input / output format
Input: Persian text (words or sentences).
Output: Sequence of phonemes corresponding to the input text.
Scoring recipe
def compute_metrics(predictions, references):
per = sum(edit_distance(p, r) / max(len(p), len(r)) for p, r in zip(predictions, references)) / len(predictions)
hom_acc = sum(1 for p, r in zip(predictions, references) if p == r) / len(predictions)
return {"PER (%)": per * 100, "Homograph Acc. (%)": hom_acc * 100}
Common pitfalls
- Evaluating on word-level data instead of sentence-level data, which removes the contextual information necessary for homograph disambiguation.
- Reporting only point estimates without standard deviations across multiple random seeds or runs, as the protocol requires mean ± std over 5 independent runs.
- Comparing accuracy metrics without accounting for inference latency, which is critical for real-time accessibility applications like screen readers.
Evidence (verbatim from paper)
Table[3] presents the performance of previously available G2P tools on the SentenceBench benchmark. As shown, the only two models that perform well in terms of PER are the neural GE2PE model ([2024]) and the rule-based eSpeak tool ([2024]). However, even these models perform worse than random when it comes to homograph disambiguation.
Citation
@misc{fetratqharabagh2025fast,
title={Fast, Not Fancy: Rethinking G2P with Rich Data and Rule-Based Models},
author={Mahta Fetrat Qharabagh, Zahra Dehghanian, and Hamid R. Rabiee (2025)},
year={2025},
note={arXiv:2505.12973}
}
- arXiv: 2505.12973