S2l Eval

s2l-eval

Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences — Korzh et al. (2025) (arXiv:2508.03542, 2025)

What this evaluates

Evaluates models' ability to transcribe spoken mathematical equations and sentences into correct LaTeX syntax. It probes audio-to-text conversion, handling of mathematical symbols, and robustness to syntactic variations in LaTeX formatting.

Datasets

• S2L-equations — total ?; splits: train (-1), test (-1)
• S2L-sentences — total ?; splits: train (-1), test (-1)

Metrics

• CER (primary) — range: percent
  • Character Error Rate: the ratio of edit operations (insertions, deletions, substitutions) to the length of the ground truth string. Evaluated on normalized, lowercase text to mitigate syntactic differences.
• TeXBLEU — range: [0, 1]
  • BLEU score computed on LaTeX token sequences after normalization. Specifically designed to compare LaTeX code while accounting for structural equivalence and formatting variations.

Input / output format

Input: 16kHz audio waveform of spoken mathematical expressions or sentences. For ASR baselines, the input is the raw ASR transcription; for multimodal models, it is the raw audio concatenated with a textual prompt.

Output: LaTeX code representing the spoken equation or sentence. For sentences, inline formulas must be correctly embedded within the surrounding text.

Scoring recipe

def score(predictions, ground_truths):
    cer_scores = []
    texbleu_scores = []
    for pred, gt in zip(predictions, ground_truths):
        # Apply equation normalization to handle spacing/capitalization/font differences
        pred_norm = normalize_latex(pred.lower())
        gt_norm = normalize_latex(gt.lower())
        # Compute CER
        cer = edit_distance(pred_norm, gt_norm) / len(gt_norm)
        cer_scores.append(cer)
        # Compute TeXBLEU (on non-lowercased tokens)
        texbleu = compute_bleu_on_latex_tokens(pred, gt)
        texbleu_scores.append(texbleu)
    return {'CER': sum(cer_scores)/len(cer_scores), 'TeXBLEU': sum(texbleu_scores)/len(texbleu_scores)}

Common pitfalls

• Raw CER heavily penalizes syntactically equivalent LaTeX (e.g., \int_a^b vs \int_{a}^{b}) despite identical mathematical meaning.
• Capitalization and font style variations (e.g., \phi vs \Phi, \mathcal{R} vs r) distort character-level metrics if normalization is skipped.
• For S2L-sentences, inline formulas must be extracted and evaluated separately from surrounding text; evaluating the full string as one unit mixes text and math errors.

Evidence (verbatim from paper)

The primary reported metrics are character error rate ($\operatorname{CER}$), and $\operatorname{TeXBLEU}$ (Jung et al. [2025]) metric, recently specifically proposed for LaTeX comparison. For S2L-equations, predictions and ground truth are compared in LaTeX form, as illustrated in Table[5]. For S2L-sentences, which contain inline formulas within English text, we separately evaluate both equation and text components.

Citation

@misc{korzh2025speechlatex,
  title={Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences},
  author={Korzh et al. (2025)},
  year={2025},
  note={arXiv:2508.03542}
}

• arXiv: 2508.03542