Cg Bench Eval

cg-bench-eval

AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs — Lu et al. (2025) (arXiv:2506.05328, 2025)

What this evaluates

Evaluates multimodal large language models on clue-grounded audio-visual counting tasks over long videos. It probes the model's ability to integrate audio and visual cues to locate temporal segments and accurately count events, objects, or attributes within those segments.

Datasets

• CG-Bench — total ?; splits: test (-1)

Metrics

• counting_accuracy (primary) — range: [0, 1]
  • Exact-match accuracy: the model outputs a single integer representing the count. A score of 1 is assigned if the predicted integer exactly matches the ground truth count, otherwise 0. The final metric is the average score across all test instances.

Input / output format

Input: A video file (synchronized audio and visual streams) and a natural language question specifying the counting target, modality type (A2V, V2A, AV, A, or V), and relevant time segment boundaries.

Output: For black-box evaluation: a single integer. For white-box evaluation: a JSON object enclosed in tags containing either event timestamps, object bounding boxes, or attribute clusters with bounding boxes.

Scoring recipe

def score_counting(predictions, gold_counts):
    correct = 0
    for pred, gold in zip(predictions, gold_counts):
        pred_int = int(re.search(r'\d+', str(pred)).group())
        if pred_int == gold:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Models may fail to correctly isolate the relevant temporal segment when audio and visual cues conflict or are sparse.
• Object counting often suffers from double-counting due to camera motion or repeated appearances of the same entity across frames.
• Attribute counting requires semantic clustering rather than instance enumeration, which can confuse models trained only on standard object detection.

Evidence (verbatim from paper)

Watch the video and answer the question ‘{QuestionHere}’ with a number. Just output the number itself, don’t output anything else. The evaluation protocol relies on counting_accuracy to measure exact-match performance across all modality settings.

Citation

@misc{lu2025avreasoner,
  title={AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2506.05328}
}

• arXiv: 2506.05328