State Control Eval

state-control-eval

See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles — Wu et al. (2025) (arXiv:2509.13615, 2025)

What this evaluates

Evaluates multimodal agents' ability to perceive current GUI states from screenshots, interpret natural language toggle instructions, and execute precise click actions. It specifically probes state-aware reasoning by measuring accuracy on both positive and negative toggle instructions, as well as grounding precision and false positive/negative rates.

Datasets

• state control benchmark — total ?; splits: test (-1); repo https://github.com/ZrW00/StaR
• dynamic evaluation benchmark — total 20; splits: test (20); repo https://github.com/ZrW00/StaR

Metrics

• O-AMR (primary) — range: percent
  • Overall Action Match Rate: proportion of test samples where the predicted action matches the ground truth in both type and parameters (e.g., click coordinates, text content, app names).
• N-AMR — range: percent
  • Negative Action Match Rate: proportion of samples with negative instructions where the predicted action correctly matches the ground truth in type and parameters.
• TMR — range: percent
  • Type Match Rate: proportion of test samples where the predicted action type matches the ground truth.
• Task Success Rate — range: [0, 1]
  • Proportion of successful task trajectories. For dynamic tasks with multiple subtasks, success is partial: if half succeed and half fail, the rate is 0.5.

Input / output format

Input: Current GUI screenshot(s), natural language instruction (specifying desired toggle state), and optionally prior action history or low-level step instructions.

Output: A reasoning process (Thought) followed by an action decision (Action) specifying action type and parameters (e.g., normalized click coordinates [0,1000], text content, or app name).

Scoring recipe

def compute_amr(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred['type'] == gold['type'] and pred['params'] == gold['params']:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Confusing positive-instruction metrics (P-AMR, P-FNR) with negative-instruction metrics (N-AMR, N-FPTR, N-FPR), which require opposite state-change logic.
• Forgetting that click coordinates must be normalized to [0, 1000] during evaluation, as specified in the implementation details.
• Misinterpreting partial success in dynamic environments: a task with multiple subtasks scores 0.5 if exactly half succeed, not 0 or 1.

Evidence (verbatim from paper)

Type Match Rate (TMR)↑: Proportion of test samples where the predicted action type matches the ground truth. Action Match Rate (AMR)↑: Proportion of test samples where the predicted action matches the ground truth in both type and parameters (e.g., click coordinates, text content, app names).

Citation

@misc{wu2025star,
  title={See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.13615}
}

• arXiv: 2509.13615