# S2l Eval

> Evaluates models' ability to transcribe spoken mathematical equations and sentences into correct LaTeX syntax. It probes audio-to-text conversion, handling of mathematical symbols, and robustness to syntactic variations in LaTeX formatting. Use when the user wants to benchmark on S2L-equations, S2L-sentences, or asks about evaluating this task. Reports CER.

- Skill: `qhjqhj00/s2l-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/s2l-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/s2l-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/s2l-eval

---


# s2l-eval

> Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences — Korzh et al. (2025) (arXiv:2508.03542, 2025)

## What this evaluates

Evaluates models' ability to transcribe spoken mathematical equations and sentences into correct LaTeX syntax. It probes audio-to-text conversion, handling of mathematical symbols, and robustness to syntactic variations in LaTeX formatting.

## Datasets

- **S2L-equations** — total ?; splits: train (-1), test (-1)
- **S2L-sentences** — total ?; splits: train (-1), test (-1)

## Metrics

- `CER` **(primary)** — range: percent
  - Character Error Rate: the ratio of edit operations (insertions, deletions, substitutions) to the length of the ground truth string. Evaluated on normalized, lowercase text to mitigate syntactic differences.
- `TeXBLEU` — range: [0, 1]
  - BLEU score computed on LaTeX token sequences after normalization. Specifically designed to compare LaTeX code while accounting for structural equivalence and formatting variations.

## Input / output format

**Input**: 16kHz audio waveform of spoken mathematical expressions or sentences. For ASR baselines, the input is the raw ASR transcription; for multimodal models, it is the raw audio concatenated with a textual prompt.

**Output**: LaTeX code representing the spoken equation or sentence. For sentences, inline formulas must be correctly embedded within the surrounding text.

## Scoring recipe

```python
def score(predictions, ground_truths):
    cer_scores = []
    texbleu_scores = []
    for pred, gt in zip(predictions, ground_truths):
        # Apply equation normalization to handle spacing/capitalization/font differences
        pred_norm = normalize_latex(pred.lower())
        gt_norm = normalize_latex(gt.lower())
        # Compute CER
        cer = edit_distance(pred_norm, gt_norm) / len(gt_norm)
        cer_scores.append(cer)
        # Compute TeXBLEU (on non-lowercased tokens)
        texbleu = compute_bleu_on_latex_tokens(pred, gt)
        texbleu_scores.append(texbleu)
    return {'CER': sum(cer_scores)/len(cer_scores), 'TeXBLEU': sum(texbleu_scores)/len(texbleu_scores)}
```

## Common pitfalls

- Raw CER heavily penalizes syntactically equivalent LaTeX (e.g., \int_a^b vs \int_{a}^{b}) despite identical mathematical meaning.
- Capitalization and font style variations (e.g., \phi vs \Phi, \mathcal{R} vs r) distort character-level metrics if normalization is skipped.
- For S2L-sentences, inline formulas must be extracted and evaluated separately from surrounding text; evaluating the full string as one unit mixes text and math errors.

## Evidence (verbatim from paper)

> The primary reported metrics are character error rate ($\operatorname{CER}$), and $\operatorname{TeXBLEU}$ *(Jung et al. [2025])* metric, recently specifically proposed for LaTeX comparison. For S2L-equations, predictions and ground truth are compared in LaTeX form, as illustrated in Table[5]. For S2L-sentences, which contain inline formulas within English text, we separately evaluate both equation and text components.

## Citation

```bibtex
@misc{korzh2025speechlatex,
  title={Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences},
  author={Korzh et al. (2025)},
  year={2025},
  note={arXiv:2508.03542}
}
```

- arXiv: 2508.03542