qhjqhj00 computer-use-agent-eval
VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos — Lu et al. (2025) (arXiv:2510.19488, 2025)
What this evaluates
Evaluates computer-use agents on desktop task completion in online and offline settings, and measures the precision of a video-to-action module in detecting GUI events and extracting interaction parameters from screen recordings.
Datasets
- OSWorld-Verified — total 369; splits: test (369)
- AgentNetBench — total 100; splits: test (100)
- Video2Action Held-out Test Set — total 20282; splits: test (20282)
Metrics
task success rate(primary) — range: percent- Percentage of tasks completed successfully out of the total number of tasks in the benchmark.
step success rate(primary) — range: percent- Percentage of individual interaction steps executed correctly out of the total steps in the benchmark.
F1— range: [0, 1]- Harmonic mean of Precision and Recall for action event detection. A prediction counts as a hit if its type matches and its temporal interval has any overlap with a ground-truth event.
Accuracy— range: [0, 1]- Proportion of manually validated action parameter predictions that correctly explain the observed on-screen transition between pre/post frames.
Input / output format
Input: For agent benchmarks: OS state/screenshots and task instructions. For Video2Action: Screen-capture video frames.
Output: For agent benchmarks: Sequence of GUI actions (click, drag, press, scroll, type) with parameters (coordinates, text). For Video2Action: Action event tuples (type, start_time, end_time) and interaction parameters.
Scoring recipe
def compute_success_rate(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return correct / len(gold) * 100
def compute_f1(preds, gold):
hits = 0
for p in preds:
if any(p['type'] == g['type'] and intervals_overlap(p['t_s'], p['t_e'], g['t_s'], g['t_e']) for g in gold):
hits += 1
precision = hits / len(preds) if preds else 0
recall = hits / len(gold) if gold else 0
return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Confusing online (OSWorld-Verified) vs offline (AgentNetBench) evaluation settings, which use different environments and action budgets.
- Assuming temporal overlap requires exact boundary matching; the protocol uses any overlap for event detection hits.
- Parameter accuracy is manually assessed due to missing target-element boxes, not automatically computed.
Evidence (verbatim from paper)
On OSWorld-Verified, our complete approach achieves a task success rate of 14.13%, demonstrating a 4.83 percentage point improvement (+52% relative) over SFT-only training (9.3%) and more than tripling the performance of the base model (4.5%). On AgentNetBench, incorporating VideoAgentTrek pretraining achieves a step success rate of 69.3%, representing a 5.2 percentage point improvement over the SFT-only baseline (64.1%) and a substantial 30.8 percentage point gain over the base model (38.5%). A prediction counts as a hit iff its type matches and its interval has any temporal overlap with a ground-truth event; unmatched predictions are false positives and unmatched ground truths are false negatives. We report per-type Precision/Recall/F1 and micro/macro aggregates.
Citation
@misc{lu2025videoagenttrek,
title={VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos},
author={Lu et al. (2025)},
year={2025},
note={arXiv:2510.19488}
}
- arXiv: 2510.19488