Vera Eval

vera-eval

Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap — Lin et al. (2025) (arXiv:2509.26542, 2025)

What this evaluates

Evaluates the reasoning capabilities of voice and multimodal models under real-time streaming constraints, quantifying the performance gap between text and voice modalities on tasks with well-defined ground truth.

Datasets

• VERA — total ?; splits: test (-1); repo https://github.com/linyueqian/VERA

Metrics

• Accuracy (primary) — range: [0, 1]
  • Assessed via an LLM-as-a-judge protocol using GPT-4o. Each prediction is evaluated three times independently, and the final label (Correct, Incorrect, or Not Attempted) is determined by majority vote.
• WER — range: [0, 1]
  • Word Error Rate comparing ASR transcripts against ground truth, after LLM-based normalization of mathematical notation.

Input / output format

Input: Text or audio prompts for reasoning tasks; voice models receive prompts and output speech, which is transcribed via ASR and normalized to canonical notation before evaluation.

Output: Speech output from voice models, transcribed to text, normalized, and classified as Correct, Incorrect, or Not Attempted by the LLM judge.

Scoring recipe

def score_accuracy(predictions, gold):
    correct_count = 0
    for pred, gt in zip(predictions, gold):
        # GPT-4o judge called 3 times on normalized transcript
        labels = [judge_call(pred, gt) for _ in range(3)]
        if majority_vote(labels) == 'Correct':
            correct_count += 1
    return correct_count / len(predictions)

Common pitfalls

• LLM-as-a-judge stochasticity requires multiple independent evaluations and majority voting to stabilize.
• Transcription artifacts from ASR must be distinguished from genuine reasoning errors during failure analysis.
• Mathematical expressions require strict LLM-based normalization before WER or accuracy comparison to avoid unfair penalties.

Evidence (verbatim from paper)

We assess task accuracy using an LLM-as-a-judge protocol*(Zheng et al., [2023]; Liu et al., [2023]). This approach is highly effective for VERA because our benchmark tasks, while challenging, are designed to have well-defined ground truth answers with minimal ambiguity, making them suitable for reliable automated grading. We employ GPT-4o(OpenAI, [2024a])* as the grader, using the normalized ASR transcript for voice model outputs. Each prediction undergoes three independent evaluations to mitigate judgment stochasticity, with the final label (Correct, Incorrect, or Not Attempted) determined by majority vote.

Citation

@misc{lin2025voiceevaluation,
  title={Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced Performance Gap},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2509.26542}
}

• arXiv: 2509.26542