Mmar Eval

mmar-eval

Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models — Longhao Li et al. (2026) (arXiv:2604.12527, 2026)

What this evaluates

Evaluates deep audio reasoning capabilities by testing both final answer correctness and the logical quality of intermediate reasoning steps. It covers single-domain (sound, music, speech) and mixed-domain audio tasks to measure how well models avoid spurious correlations and follow verifiable reasoning paths.

Datasets

• MMAR — total ?; splits: test (-1)

Metrics

• Avg (primary) — range: percent
  • Average accuracy over the dataset: Avg = (1/N) * sum(c_i), where c_i is 1 if the final answer is correct and 0 otherwise.
• Rubrics — range: [0, 1]
  • Average reasoning score across all samples: Rubrics = (1/N) * sum(r_i), where r_i is the proportion of satisfied rubric criteria for correct answers, and 0 for incorrect answers.
• CRS — range: [0, 1]
  • Correct Reasoning Score: CRS = sum(r_i) / sum(c_i), representing the average reasoning score conditioned only on correctly answered samples.

Input / output format

Input: Audio sample paired with a natural language question or prompt.

Output: A final answer string and a step-by-step reasoning trace (Chain-of-Thought).

Scoring recipe

def compute_metrics(predictions, golds, rubrics, judge):
    N = len(predictions)
    correct = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    avg_acc = sum(correct) / N
    
    reasoning_scores = []
    for i in range(N):
        if correct[i] == 0:
            reasoning_scores.append(0.0)
        else:
            satisfied = sum(1 for rubric in rubrics[i] if judge(predictions[i], rubric))
            reasoning_scores.append(satisfied / len(rubrics[i]))
            
    rubrics_score = sum(reasoning_scores) / N
    correct_count = sum(correct)
    crs = sum(reasoning_scores) / correct_count if correct_count > 0 else 0.0
    return avg_acc, rubrics_score, crs

Common pitfalls

• Reasoning scores are explicitly set to 0 for incorrect answers, meaning high Rubrics/CRS requires high accuracy first.
• Evaluation relies on an LLM judge (GPT-4o) against auto-generated rubrics, which can introduce judge bias or miss nuanced reasoning flaws.
• Results are not from a single run; the protocol requires five runs and reports the mean of the middle three scores to reduce variance.

Evidence (verbatim from paper)

Specifically, for each sample $i$, let $c_{i}\in{0,1}$ denote the correctness of the answer, where $c_{i}=1$ indicates a correct prediction and $c_{i}=0$ otherwise. The answer's correctness is measured by the average accuracy (Avg) over the dataset: ... Each MMAR sample is associated with an instance-level rubric, automatically generated by Gemini-2.5-Pro from the ground-truth reasoning path. The rubric contains five verifiable criteria that capture the key reasoning steps for that specific example. Given a model's predicted reasoning trace, an LLM judge evaluates whether each criterion is satisfied. Following the official challenge protocol, we use GPT-4o as the LLM judge. For a correctly answered sample, the judge assigns a binary score (0 or 1) to each criterion, and the reasoning score $r_{i}$ is computed as the proportion of satisfied criteria: ... The overall Rubrics Score across the dataset is defined as: ... We further introduce Correct Reasoning Score (CRS) to evaluate reasoning quality on the correct answer only as follows: ...

Citation

@misc{li2026audiocogito,
  title={Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models},
  author={Longhao Li et al. (2026)},
  year={2026},
  note={arXiv:2604.12527}
}

• arXiv: 2604.12527