Odysseys Eval

odysseys-eval

Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks — Jang et al. (2026) (arXiv:2604.24964, 2026)

What this evaluates

Probes an agent's ability to perform realistic, long-horizon web navigation tasks that require sustained cross-site reasoning, context maintenance across multiple tabs, and efficient action execution. It evaluates whether models can complete complex, multi-step user journeys derived from real browsing behavior within strict step budgets.

Datasets

• Odysseys — total 200; splits: test (200)

Metrics

• Perfect Rubrics (%) (primary) — range: percent
  • Binary pass/fail per task. A task is scored as passing only if every one of its predefined success rubrics are satisfied during the agent's trajectory.
• Trajectory Efficiency (%) — range: percent
  • Average per-task ratio of rubric score to step count: (1/N) * sum(s_i / n_i), where s_i is the averaged rubric score for task i and n_i is the number of agent steps (LLM calls) taken.

Input / output format

Input: Task description, starting URL (if provided), and full access to a Google Chrome window within a virtual Ubuntu environment.

Output: Action trajectories consisting of browser interactions, system commands, or Python scripts, executed sequentially up to a maximum of 100 steps.

Scoring recipe

def score_perfect_rubrics(task_rubrics_satisfied):
    return 1.0 if all(task_rubrics_satisfied) else 0.0

def score_trajectory_efficiency(rubric_scores, step_counts):
    return sum(s / n for s, n in zip(rubric_scores, step_counts)) / len(rubric_scores)

Common pitfalls

• Step count refers to LLM round-trips/calls, not individual UI actions; models like GPT-5.4 can batch multiple actions per call, artificially lowering step counts.
• The benchmark enforces a hard 100-step limit per task, which heavily penalizes inefficient exploration or research-heavy strategies that would succeed with more steps.
• Rubric-based scoring provides partial credit for partial completion, unlike holistic trajectory judges that give a binary pass/fail, making direct comparison with older web navigation benchmarks misleading.

Evidence (verbatim from paper)

To quantify this, we report Trajectory Efficiency, defined as the per-task ratio of rubric score to step count, averaged over all tasks: where $s_{i}$ is the averaged rubric score for task $i$ and $n_{i}$ is the number of agent steps taken on that task.

Citation

@misc{jang2026odysseys,
  title={Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2604.24964}
}

• arXiv: 2604.24964