Gui Agent Kv Eval

gui-agent-kv-eval

GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness — Huang et al. (2025) (arXiv:2510.00536, 2025)

What this evaluates

Evaluates the accuracy and efficiency of GUI agents under varying KV cache compression budgets across visual grounding, offline action prediction, and online task completion benchmarks.

Datasets

• ScreenSpotV2 — total ?; splits: test (-1)
• ScreenSpot-Pro — total ?; splits: test (-1)
• AndroidControl — total ?; splits: test (-1)
• Multimodal-Mind2Web — total ?; splits: test (-1)
• AgentNetBench — total ?; splits: test (-1)
• OSWorld-Verified — total ?; splits: test (-1)

Metrics

• click accuracy — range: [0, 1]
  • Proportion of test samples where the model's predicted coordinate falls within the ground truth bounding box.
• success rate — range: [0, 1]
  • Proportion of tasks where a series of operations are successfully executed and specific milestones are achieved.
• step accuracy (primary) — range: [0, 1]
  • Proportion of test samples where a single predicted step contains the correct operation (e.g., click, write) and arguments (e.g., coordinate or textual content).

Input / output format

Input: GUI screenshots (current and recent history frames) paired with natural language instructions or task descriptions.

Output: Predicted UI element coordinates (bounding boxes) or discrete action steps with arguments (e.g., click, write, coordinates).

Scoring recipe

def compute_metric(predictions, golds, metric_type):
    if metric_type == 'click_accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p_in_box(p, g)) / len(golds)
    elif metric_type == 'success_rate':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif metric_type == 'step_accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p.op == g.op and p.args == g.args) / len(golds)

Common pitfalls

• Metrics are benchmark-specific (click accuracy vs success rate vs step accuracy), so direct numerical comparison across different datasets is invalid.
• KV cache budgets are evaluated at discrete, non-linear levels (1%, 3%, 5%, 10%, 15%, 20%, 40%, 80%, 100%), and performance does not always scale monotonically with budget.
• Efficiency metrics (MFLOPs per decoded token) are only reported for AgentNetBench, making cross-dataset efficiency comparisons impossible.

Evidence (verbatim from paper)

For ScreenSpot-V2 and ScreenSpot-Pro, we measure click accuracy, defined as the proportion of test samples where the model’s predicted coordinate falls within the ground truth bounding box. For OSWorld-Verified, we use success rate, determined by whether a series of operations are successfully executed and specific milestones are achieved. For the remaining benchmarks, we adopt step accuracy, which assesses whether a single predicted step contains the correct operation (e.g., click, write) and arguments (e.g., coordinate or textual content).

Citation

@misc{huang2025guikv,
  title={GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2510.00536}
}

• arXiv: 2510.00536