Rodent Bench Eval

rodent-bench-eval

Rodent-Bench — Heap et al. (arXiv:2602.18540, 2026)

What this evaluates

Evaluates multimodal large language models on temporal segmentation and fine-grained behavioral annotation of rodent videos. It probes capabilities in long-video processing, distinguishing subtle or rare behaviors, and handling diverse experimental paradigms and camera angles.

Datasets

• Rodent-Bench-Long — total ?; splits: test (-1)
• Rodent-Bench-Short — total ?; splits: test (-1)

Metrics

• Weighted Matthew’s Correlation Coefficient (MCC) (primary) — range: [-1, 1]
  • A correlation coefficient between predicted and observed classifications, weighted by class frequency. Calculated as (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)), where TP, TN, FP, FN are true/false positives/negatives. Ranges from -1 to 1.

Input / output format

Input: Video clips (short or long duration) accompanied by task-specific prompts instructing the model to perform temporal segmentation and behavioral annotation.

Output: JSON-formatted output containing segment keys (e.g., 'end_time') and corresponding timestamps or labels for each behavioral segment in the video.

Scoring recipe

def compute_weighted_mcc(predictions, ground_truth):
    # 1. Align predicted and ground truth temporal segments
    # 2. Map behavioral labels to discrete classes
    # 3. Compute confusion matrix weighted by class prevalence
    # 4. Apply MCC formula: (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))
    # 5. Return scalar MCC value
    return mcc_value

Common pitfalls

• Models often produce malformed JSON (e.g., truncated outputs or incorrect keys like 'end_long_time' instead of 'end_time'), breaking automated parsing.
• Performance drops significantly on videos with non-standard camera angles, visual filters requiring color recognition, or behaviors shorter than one second where the distinction between 'freezing' and stillness is subtle.

Evidence (verbatim from paper)

Figure 3: Weighted Matthew’s Correlation Coefficient (MCC) performance across models. (a) Rodent-Bench-Long: Gemini-2.5-Pro achieves the highest performance with lower variance compared to Gemini-2.5-Flash.

Citation

@misc{heap2026rodentbench,
  title={Rodent-Bench},
  author={Heap et al.},
  year={2026},
  note={arXiv:2602.18540}
}

• arXiv: 2602.18540