qhjqhj00 webarena-human-trajectory-eval
AI Planning Framework for LLM-Based Web Agents — Shahnovsky et al. (2026) (arXiv:2603.12710, 2026)
What this evaluates
Evaluates the planning and execution quality of LLM-based web agents by comparing their action trajectories against human-demonstrated gold standards. It measures recovery from deviations, action repetitiveness, step fulfillment, partial task completion, and alignment between planned and executed actions.
Datasets
- WebArena Human Trajectory Dataset — total 794; splits: full (794); repo https://github.com/shahnovsky/WebArena-Human-Trajectory-Dataset
Metrics
Recovery Rate(primary) — range: [0, 1]- Average ratio of recoveries to deviation incidents across tasks: (1/#tasks) * sum(# recoveries in task t / # deviation incidents in task t). A recovery occurs when an agent action fulfills a subsequent human gold step after a deviation.
Repetitiveness Rate— range: [0, 1]- Inverted proportion of consecutive duplicate actions: 1 - (1/#tasks) * sum(# repetitive actions in task t / # actions in task t). Higher values indicate fewer redundant steps.
Step Success Rate— range: [0, 1]- Average proportion of human gold steps matched by the agent: (1/#tasks) * sum(# matched human gold steps in task t / # human gold steps in task t). Matches are determined via LLM semantic equivalence.
Partial Success Rate— range: [0, 1]- Average proportion of completed requirements for multi-output tasks: (1/#req_tasks) * sum(# completed requirements in task t / # requirements in task t). Inapplicable to single-output tasks.
Element Accuracy Rate— range: [0, 1]- Average proportion of planned actions that match the actually executed actions: (1/#tasks) * sum(# matching steps predicted and actual / # agent steps).
Input / output format
Input: Agent execution trajectory (sequence of actions), human gold trajectory (reference sequence), task instructions, and task-specific requirements (for multi-output tasks).
Output: Numerical scores between 0 and 1 for each of the five trajectory quality metrics, computed via rule-based counting and LLM-as-a-judge semantic matching.
Scoring recipe
def compute_metrics(agent_traj, gold_traj, task_reqs):
scores = {}
for t in tasks:
devs = count_deviations(agent_traj[t], gold_traj[t])
recs = count_recoveries(agent_traj[t], gold_traj[t], lookahead=5)
scores['recovery_rate'] = recs / max(devs, 1)
reps = count_consecutive_duplicates(agent_traj[t])
scores['repetitiveness_rate'] = 1 - (reps / len(agent_traj[t]))
matches = llm_semantic_match(agent_traj[t], gold_traj[t])
scores['step_success_rate'] = matches / len(gold_traj[t])
if len(task_reqs[t]) > 1:
completed = count_completed_requirements(agent_traj[t], task_reqs[t])
scores['partial_success_rate'] = completed / len(task_reqs[t])
planned = agent_traj[t].planned
actual = agent_traj[t].actual
scores['element_accuracy_rate'] = count_matching(planned, actual) / len(actual)
return {k: np.mean(v) for k, v in scores.items()}
Common pitfalls
- Relies on an LLM-as-a-judge for semantic equivalence, which may introduce variability or bias compared to exact string matching.
- Recovery Rate depends on a user-defined lookahead parameter to check future human steps, significantly altering scores if set too high or low.
- Partial Success Rate is explicitly inapplicable to tasks with only a single required output; applying it to such tasks yields undefined results.
Evidence (verbatim from paper)
Definition 4.1 (Recovery Rate). Measures how well an agent can return to the expected human-demonstrated sequence of actions after deviating from it. A deviation incident is recorded when an agent step does not directly fulfill the current human gold step. A recovery occurs when the agent performs an action that successfully fulfills a subsequent human gold step.
Citation
@misc{shahnovsky2026aiplanning,
title={AI Planning Framework for LLM-Based Web Agents},
author={Shahnovsky et al. (2026)},
year={2026},
note={arXiv:2603.12710}
}
- arXiv: 2603.12710