Mind2web Eval

mind2web-eval

Mind2Web: Towards a Generalist Agent for the Web — Deng et al. (2023) (arXiv:2306.06070, 2023)

What this evaluates

Evaluates a model's ability to act as a generalist web agent by completing multi-step tasks across diverse, unseen websites and domains. It probes out-of-distribution generalization, web element grounding, and sequential action planning in real-world browser environments.

Datasets

• Mind2Web — total 2350; splits: train (1009), TestCross-Task (252), TestCross-Website (177), TestCross-Domain (912); repo https://github.com/OSU-NLP-Group/Mind2Web

Metrics

• Element Accuracy — range: percent
  • Compares the model's selected element against all acceptable ground-truth elements. Returns 1 if the prediction matches any acceptable element, 0 otherwise.
• Operation F1 — range: percent
  • Calculates token-level F1 score for the predicted operation. Functions as exact accuracy for Click actions, but accounts for input value correctness for Type and Select Option actions.
• Step Success Rate (primary) — range: percent
  • Returns 1 only if both the selected element matches an acceptable element AND the predicted operation exactly matches the ground-truth operation; otherwise returns 0. Evaluated independently per step.
• Task Success Rate — range: percent
  • Returns 1 only if all steps in the entire task sequence achieve Step Success Rate of 1; otherwise returns 0. Represents a stringent end-to-end success metric.

Input / output format

Input: Cleaned HTML elements (visible, semantically meaningful), task instruction, and ground-truth action history (provided during evaluation).

Output: A selected web element identifier and a predicted operation (e.g., Click, Type, Select Option) with associated parameters.

Scoring recipe

def evaluate_step(pred_elem, pred_op, gold_elems, gold_op):
    elem_acc = 1.0 if pred_elem in gold_elems else 0.0
    op_f1 = token_level_f1(pred_op, gold_op)  # exact match for Click
    step_success = 1.0 if (elem_acc == 1.0 and op_f1 == 1.0) else 0.0
    return elem_acc, op_f1, step_success

def evaluate_task(steps):
    return 1.0 if all(s.step_success for s in steps) else 0.0

Common pitfalls

• Step-wise evaluation assumes ground-truth action history is provided, which does not reflect fully autonomous agent deployment.
• Task Success Rate requires every single step to be correct, making it extremely stringent and often yielding near-zero scores for minor deviations.
• Models frequently output a 'None' option claiming the task cannot be completed on the current page, complicating element/operation matching.

Evidence (verbatim from paper)

For evaluation, we first calculate Element Accuracy that compares the selected element with all acceptable elements, and Operation F1 that calculates token-level F1 score for the predicted operation. This is the same as accuracy for Click, but considers the correctness of the input value for Type and Select Option. Each step of the task is evaluated independently with the ground truth action history provided. We then define Step Success Rate and Success Rate (for the whole task). A step is regarded as successful only if both the selected element and the predicted operation are correct. A task is regarded successful only if all steps have succeeded.

Citation

@misc{deng2023mind2web,
  title={Mind2Web: Towards a Generalist Agent for the Web},
  author={Deng et al. (2023)},
  year={2023},
  note={arXiv:2306.06070}
}

• arXiv: 2306.06070