Exevr Bench Eval

exevr-bench-eval

Video-Based Reward Modeling for Computer-Use Agents — Linxin Song et al. (2026) (arXiv:2603.10178, 2026)

What this evaluates

This benchmark evaluates computer-use agents' ability to correctly judge whether a GUI interaction trajectory succeeds or fails, and to precisely localize the temporal window where the first error occurs. It probes spatiotemporal reasoning, visual redundancy handling, and fine-grained temporal attribution in long video trajectories.

Datasets

• ExeVR-Bench — total 789; splits: test (789); repo https://github.com/limenlp/ExeVRM

Metrics

• accuracy (primary) — range: [0, 1]
  • TP / (TP + FP + FN + TN). Measures the proportion of correctly classified trajectories (positive or negative) out of the total.
• precision — range: [0, 1]
  • TP / (TP + FP). Measures the proportion of predicted positive trajectories that are actually positive.
• recall — range: [0, 1]
  • TP / (TP + FN). Measures the proportion of actual positive trajectories that are correctly identified.
• tIoU — range: [0, 1]
  • tIoU(Î, I) = |Î ∩ I| / |Î ∪ I| = max(0, min(ê, e) - max(ŝ, s)) / max(ê, e) - min(ŝ, s). Measures the overlap between the predicted error interval and the ground-truth error interval.

Input / output format

Input: A user instruction paired with a video trajectory of the agent's GUI interaction. Videos are rendered at 720p and sampled at 1 FPS, capped at 100 frames.

Output: A binary label (correct/incorrect) and, for attribution tasks, a predicted time range [t_s, t_e] indicating the first error occurrence.

Scoring recipe

# Binary classification metrics
tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
accuracy = tp / (tp + fp + fn)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0

# Temporal IoU for attribution
def compute_tIoU(pred_int, gold_int):
    inter = max(0, min(pred_int[1], gold_int[1]) - max(pred_int[0], gold_int[0]))
    union = max(pred_int[1], gold_int[1]) - min(pred_int[0], gold_int[0])
    return inter / union if union > 0 else 0.0

Common pitfalls

• Video input is strictly capped at 100 frames sampled at 1 FPS; models processing full-resolution or full-length videos will receive different inputs than evaluated.
• The temporal attribution task requires predicting a continuous time range [t_s, t_e] for the first error, not just a single timestamp or the final failure point.
• The dataset is approximately balanced (50/50 positive/negative), so accuracy alone can mask poor recall on negative trajectories.

Evidence (verbatim from paper)

We report standard classification metrics, including accuracy, precision, and recall, to measure how well the reward model distinguishes positive from negative trajectories. In addition, to evaluate temporal grounding quality, i.e., whether the model localizes the critical time span responsible for failure, we compute a temporal intersection-over-union (tIoU) between the model-predicted interval and the ground-truth interval:

Citation

@misc{song2026videoreward,
  title={Video-Based Reward Modeling for Computer-Use Agents},
  author={Linxin Song et al. (2026)},
  year={2026},
  note={arXiv:2603.10178}
}

• arXiv: 2603.10178