Egoavu Bench Eval

egoavu-bench-eval

EgoAVU: Egocentric Audio-Visual Understanding — Seth et al. (2026) (arXiv:2602.06139, 2026)

What this evaluates

Evaluates multimodal large language models' ability to perform joint audio-visual reasoning on egocentric videos, including action/object/sound recognition, temporal reasoning, hallucination detection, and dense audio-visual narration. It specifically probes whether models can correctly associate environmental sounds with their visual sources and maintain temporal alignment without relying heavily on visual cues.

Datasets

• EgoAVU-Bench — total 3000; splits: test (3000); repo https://github.com/facebookresearch/EgoAVU

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly matched answers for close-ended questions (multiple-choice, yes/no, or binary indicators) after extracting key phrases via regex.
• LLM-as-a-judge score — range: [1, 5]
  • Qwen3-235B-A22B-Instruct-2507 rates open-ended model responses on a 1–5 scale based on an evaluation prompt.
• ROUGE-L — range: [0, 1]
  • Recall of the longest common subsequence between the generated response and the reference caption.
• METEOR — range: [0, 1]
  • Precision and recall metric weighted by synonymy matching and stemming penalties.

Input / output format

Input: Egocentric video clips with synchronized audio and accompanying text prompts or questions.

Output: Text responses (option IDs, yes/no, or open-ended descriptive narrations).

Scoring recipe

def score_close_ended(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        extracted = extract_key_phrases(pred)  # regex for IDs, yes/no, conclusions
        if extracted == gold:
            correct += 1
    return (correct / len(golds)) * 100

def score_open_ended(predictions, references, judge_model):
    judge_scores, rouge_l_scores, meteor_scores = [], [], []
    for pred, ref in zip(predictions, references):
        judge_scores.append(judge_model.rate(pred, ref, scale=(1, 5)))
        rouge_l_scores.append(rouge_l_score(pred, ref))
        meteor_scores.append(meteor_score(pred, ref))
    return {
        'llm_judge_avg': sum(judge_scores) / len(judge_scores),
        'rouge_l_avg': sum(rouge_l_scores) / len(rouge_l_scores),
        'meteor_avg': sum(meteor_scores) / len(meteor_scores)
    }

Common pitfalls

• Models exhibit strong visual bias, frequently hallucinating sound sources that are not present in the video.
• LLM-as-a-judge scoring can be subjective and may favor verbose but factually incorrect responses over concise correct ones.
• Temporal reasoning requires precise alignment of audio and visual events, which is difficult for models trained primarily on exocentric data.

Evidence (verbatim from paper)

For close-ended QAs in EgoAVU-Bench, we follow (Yue et al., [2024]) and use regex-based string matching, where we construct robust regular expressions and design a response-processing module to extract key phrases such as option IDs (A, B, C, D), binary indicators (yes/no), and conclusion phrases from long responses for accurate answer matching. For open-ended QAs, similar to prior work (Plizzari et al., [2025]), we adopt the LLM-as-a-judge approach, employing Qwen3-235B-A22B-Instruct-2507 (Yang et al., [2025]) as an open-source judge for reproducibility. The model rates MLLM-generated responses on a 1–5 scale (see Appendix [9] for the evaluation prompt). We additionally report standard metrics used for dense response evaluation, including ROUGE-L (Lin, [2004]) and METEOR (Banerjee and Lavie, [2005]).

Citation

@misc{seth2026egoavu,
  title={EgoAVU: Egocentric Audio-Visual Understanding},
  author={Seth et al. (2026)},
  year={2026},
  note={arXiv:2602.06139}
}

• arXiv: 2602.06139