Wearvox Eval

wearvox-eval

WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables — Lin et al. (2025) (arXiv:2601.02391, 2025)

What this evaluates

Evaluates speech-language models on egocentric, multi-channel wearable audio tasks, probing their ability to handle noisy real-world acoustic conditions, reject side-talk, execute tool calls, answer questions with or without context, and translate speech in conversational settings.

Datasets

• WearVox — total 3842; splits: test (-1)

Metrics

• SearchGrouned QA — range: percent
  • LLM-based judge score comparing model output to annotated ground truth responses, normalized to percentage.
• ClosedbookQA — range: percent
  • LLM-based judge score comparing model output to ground truth, normalized to percentage.
• ToolCalling — range: percent
  • Abstract Syntax Tree (AST) evaluation matching predicted tool call structure and content to gold standard, reported as percentage.
• Side TalkRejection — range: percent
  • Binary accuracy measuring correct identification and suppression of non-device-directed speech.
• SpeechTranslation — range: percent
  • Session-based score averaging turn-level LLM judge ratings for speaker diarization and translation quality, with penalties for missing or hallucinated turns.
• Turn-basedMicro-avg (primary) — range: percent
  • Arithmetic mean of the accuracy scores across the four turn-based tasks (SearchGrouned QA, ClosedbookQA, ToolCalling, Side TalkRejection).

Input / output format

Input: Multi-channel egocentric audio recordings (often beamformed to single-channel for baselines) paired with task-specific text prompts (e.g., search context, tool definitions, or translation instructions).

Output: Text responses, structured tool calls, special control tokens (for side-talk rejection), or translated text, depending on the task.

Scoring recipe

def compute_wearvox_score(predictions, golds, task):
    scores = []
    for pred, gold in zip(predictions, golds):
        if task in ['SearchGrouned QA', 'ClosedbookQA']:
            scores.append(llm_judge_accuracy(pred, gold))
        elif task == 'ToolCalling':
            scores.append(ast_match_accuracy(pred, gold))
        elif task == 'Side TalkRejection':
            scores.append(1.0 if pred == gold else 0.0)
        elif task == 'SpeechTranslation':
            scores.append(session_translation_score(pred, gold))
    return sum(scores) / len(scores) * 100

Common pitfalls

• Single-channel baselines require applying beamforming to the multi-channel recordings before inference; skipping this step invalidates the comparison.
• Speech Translation uses session-based evaluation with turn-level averaging and penalties for missing/hallucinated turns, not simple per-turn accuracy.
• Model context limits may truncate audio (e.g., 30s for Gemma 3n), which must be noted when interpreting low scores.

Evidence (verbatim from paper)

For Search Grounded QA and Closed-book QA, we employ an LLM-based judge that references annotated ground truth responses to evaluate answer quality. In the Tool Calling task, we utilize Abstract Syntax Tree (AST) evaluation, following the methodology described in fbcl, to rigorously compare the structure and content of predicted tool calls. For Side Talk Rejection, performance is measured using binary accuracy, indicating whether the model correctly identifies and suppresses non-device-directed speech.

Citation

@misc{lin2025wearvox,
  title={WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2601.02391}
}

• arXiv: 2601.02391