qhjqhj00 real-world-cross-app-eval
AgentStudio: A Toolkit for Building General Virtual Agents — Longtao Zheng et al. (2024) (arXiv:2403.17918, 2024)
What this evaluates
Assesses multi-step agent capabilities including tool use, GUI grounding, compositional generalization, and long-horizon planning across real-world desktop and web applications. It evaluates whether agents can execute complex, cross-application workflows and self-evaluate their trajectories.
Datasets
- Real-World Cross-Application Benchmark Suite — total 77; splits: test (77)
Metrics
Success(primary) — range: percent- Percentage of tasks where the agent's trajectory successfully achieves the intended outcome as verified by an environment evaluation function $f_{\mathcal{E}}$.
Critic Accuracy— range: percent- Accuracy of the model's self-evaluation in correctly judging whether its own trajectory succeeded or failed.
Input / output format
Input: Natural language instruction $g$, optionally an environment reset function $f_{\mathcal{R}}$, and an environment evaluation function $f_{\mathcal{E}}$.
Output: A sequence of actions (API calls, GUI interactions, tool uses) forming a trajectory.
Scoring recipe
successes = 0
critic_correct = 0
for task in dataset:
trajectory = model.execute(task.instruction)
is_success = task.evaluate(trajectory)
model_judgment = model.self_evaluate(trajectory)
if is_success: successes += 1
if model_judgment == is_success: critic_correct += 1
return (successes / len(dataset)) * 100, (critic_correct / len(dataset)) * 100
Common pitfalls
- Static single-step datasets cannot evaluate multi-step tasks that allow multiple valid solutions.
- Level 3 tasks require human-in-the-loop evaluation because automatic rule-based evaluators are insufficient for complex cross-application workflows.
Evidence (verbatim from paper)
A task is formalized as a tuple: $T=(g,f_{\mathcal{R}},f_{\mathcal{E}})$, where $g$ is a natural language description of the task instruction, $f_{\mathcal{R}}$ optionally resets the environment, and $f_{\mathcal{E}}$ optionally evaluates the outcome trajectory. ... Each task is evaluated based on two metrics: the percentage of tasks the model successfully completes (Success), and the accuracy in evaluating the success of trajectories (Critic Accuracy).
Citation
@misc{zheng2024agentstudio,
title={AgentStudio: A Toolkit for Building General Virtual Agents},
author={Longtao Zheng et al. (2024)},
year={2024},
note={arXiv:2403.17918}
}
- arXiv: 2403.17918