Molmoweb Eval

molmoweb-eval

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web — Gupta et al. (2026) (arXiv:2604.08516, 2026)

What this evaluates

Evaluates the capability of vision-language web agents to navigate live websites and complete complex, multi-step tasks using only screenshot inputs. It probes GUI perception, action grounding, and long-horizon planning under real-world web constraints.

Datasets

• WebVoyager — total ?; splits: test (-1)
• Online-Mind2Web — total ?; splits: test (-1)
• DeepShop — total ?; splits: test (-1)
• WebTailBench — total ?; splits: test (-1)
• ScreenSpot — total ?; splits: test (-1)
• ScreenSpot v2 — total ?; splits: test (-1)

Metrics

• pass@k accuracy (primary) — range: [0, 1]
  • Estimates the probability that at least one of k independent rollouts succeeds. Computed as 1 - binom(m-c, k) / binom(m, k), where m is the number of collected rollouts per task and c is the number of successful rollouts.
• task success rate — range: [0, 1]
  • Binary success indicator per task (1 if completed within max steps and retry budget, 0 otherwise). Reported as average across 3-5 runs.

Input / output format

Input: Screenshot of the current webpage state and the natural language task instruction.

Output: Sequential actions (e.g., click coordinates, scroll commands, text input) or final answer text.

Scoring recipe

def compute_pass_k(successes, k, m=5):
    c = sum(successes)
    if c == m:
        return 1.0
    if m - c >= k:
        return 0.0
    from math import comb
    return 1.0 - comb(m - c, k) / comb(m, k)

# Per task
successes = [judge.evaluate(rollout) for rollout in rollouts]
pass_k = compute_pass_k(successes, k)
# Aggregate
mean_pass_k = sum(pass_k for task in tasks) / len(tasks)

Common pitfalls

• Time-sensitive tasks require date patching to avoid failures; unpatched dates cause immediate task failure.
• LLM-as-a-judge variability affects cross-benchmark comparability, especially when judges differ across benchmarks.
• Step limits (30 vs 100) and retry budgets (10) significantly impact reported scores and must be strictly enforced.

Evidence (verbatim from paper)

Because some tasks are time-sensitive, we change dates in outdated requests (e.g., find a flight on August 5, 2025) to be meaningful for the task across all benchmarks. For each benchmark and model, we run 3-5 evaluations up to 100 steps and report the average score across runs. If a model does not complete the task by the maximum number of steps, it is considered a failure. As environment errors occasionally occur, we allow up to 10 retries per trajectory; tasks that do not complete within this budget are also marked as failures. ... To get an unbiased, low variance estimate of pass@k, we collect m>k rollouts per task and compute the estimate: pass@k = 1 - binom(m-c, k)/binom(m, k), where c is the number of successful rollouts among the m attempts.

Citation

@misc{gupta2026molmoweb,
  title={MolmoWeb: Open Visual Web Agent and Open Data for the Open Web},
  author={Gupta et al. (2026)},
  year={2026},
  note={arXiv:2604.08516}
}

• arXiv: 2604.08516