Miniwob Wge Eval

miniwob-wge-eval

Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration — Liu et al. (2018) (arXiv:1802.08802, 2018)

What this evaluates

Evaluates an agent's ability to navigate and interact with semi-structured web interfaces to complete goal-directed tasks. It probes relational reasoning over DOM trees, handling of natural language instructions, and sample efficiency in sparse-reward reinforcement learning settings.

Datasets

• MiniWoB — total 40; splits: test (-1)
• MiniWoB++ — total ?; splits: test (-1)
• Alaska — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • The percentage of test episodes that achieve a reward of +1. Partial rewards are explicitly disabled, making this metric a linear scaling of the average reward.

Input / output format

Input: A visual state (160×210px or 375×667px), DOM tree access via Selenium, and a text goal (structured or natural language).

Output: A sequence of environment actions (click DOM elements, type strings) until task completion or step limit.

Scoring recipe

def compute_success_rate(predictions, gold):
    successes = 0
    for pred, goal in zip(predictions, gold):
        reward = env.step_sequence(pred)
        if reward == 1.0:
            successes += 1
    return (successes / len(predictions)) * 100

Common pitfalls

• The benchmark explicitly filters out tasks requiring specialized reasoning (e.g., computing angles, algebra), so results do not generalize to all web tasks.
• Partial rewards are disabled for consistency, which fundamentally changes the reward landscape compared to the original MiniWoB and Alaska benchmarks.
• The Alaska benchmark uses a surrogate JavaScript backend and clamped dates, meaning performance may not transfer to live production websites.

Evidence (verbatim from paper)

Evaluation metric. We report success rate: the percentage of test episodes with reward +1 . Since we have removed partial rewards, success rate is a linear scaling of the average reward, and is equivalent to the definition of success rate in Shi et al. (2017).

Citation

@misc{liu2018wge,
  title={Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration},
  author={Liu et al. (2018)},
  year={2018},
  note={arXiv:1802.08802}
}

• arXiv: 1802.08802