Ost Bench Eval

ost-bench-eval

OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding — Jingli Lin et al. (arXiv:2507.07984, 2025)

What this evaluates

Evaluates multimodal large language models' ability to perform online spatio-temporal scene understanding and dynamic, agent-centric reasoning. It tests how well models update spatial and temporal knowledge as they incrementally explore environments, retrieve long-term memory, and infer object relationships across sequential observations.

Datasets

• OST-Bench — total ?; splits: test (-1); repo https://github.com/InternRobotics/OST-Bench

Metrics

• Overall average score (primary) — range: [0, 1]
  • Arithmetic mean of accuracy scores across all question subtypes and main task categories (Agent State, Agent Visible Info, Agent-Object Spatial Relationship).
• Judgement accuracy — range: [0, 1]
  • Exact match: 1 if the model's selected option equals the ground truth, else 0.
• Counting accuracy — range: [0, 1]
  • Exact match: 1 if the model's predicted number equals the ground truth, else 0.
• Temporal-Localization accuracy — range: [0, 1]
  • Exact match: 1 if the model's predicted turn index equals the ground truth, else 0.
• Mean Relative Accuracy (MRA) — range: [0, 1]
  • Measures similarity between the model's floating-point estimate and the ground truth. Calculated as the average relative accuracy across all estimation questions.

Input / output format

Input: Multi-turn dialogue format providing sequential visual observations (frames) and agent state/info. Questions probe agent position/orientation, object existence/quantity/order/direction, and spatial relationships (distance/temporal) across exploration turns.

Output: Final answer: categorical option for Judgement, integer for Counting/Temporal-Localization, or floating-point number for Estimation. Models are also prompted to output reasoning steps.

Scoring recipe

def score(predictions, ground_truths, question_types):
    scores = []
    for pred, gt, qtype in zip(predictions, ground_truths, question_types):
        if qtype in ['Judgement', 'Counting', 'Temporal-Localization']:
            scores.append(1.0 if pred == gt else 0.0)
        elif qtype == 'Estimation':
            scores.append(1.0 - abs(pred - gt) / gt)  # MRA
    return sum(scores) / len(scores)

Common pitfalls

• Chance-level baselines differ by question type: random choice for categorical/counting/temporal questions vs. fixed mean output for estimation questions.
• Performance is highly sensitive to exploration horizon; accuracy drops sharply after 2-4 turns due to increasing memory load and reasoning complexity.
• Models often fail by taking 'spatio-temporal reasoning shortcuts' rather than retrieving long-term memory anchors, leading to overestimation of their true reasoning capabilities.

Evidence (verbatim from paper)

As for the evaluation metrics, Judgement questions are considered correct if the model selects the same option as the ground truth. For Counting and Temporal Localization questions, the model’s output—whether a number or a turn index—must exactly match the ground truth to be deemed correct. For Estimation questions, we adopt the Mean Relative Accuracy (MRA) metric from VSI to score the similarity between the model’s floating-point output and the ground truth.

Citation

@misc{lin2025ostbench,
  title={OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding},
  author={Jingli Lin et al.},
  year={2025},
  note={arXiv:2507.07984}
}

• arXiv: 2507.07984