Pira Bench Eval

pira-bench-eval

PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents — Chai et al. (2026) (arXiv:2603.08013, 2026)

What this evaluates

This benchmark evaluates multimodal large language models on proactive intent recommendation from continuous, noisy GUI visual streams. It probes the model's ability to track long-horizon user behavior, distinguish true latent goals from background noise, and exercise operational restraint by remaining silent when no action is required.

Datasets

• PIRA-Bench — total 100; splits: test (100)

Metrics

• Precision — range: percent
  • True positives divided by all positive predictions. Measures the accuracy of intent suggestions.
• Recall — range: percent
  • True positives divided by all actual positive intents. Measures the model's sensitivity to user goals.
• F1_avg — range: percent
  • Harmonic mean of Precision and Recall. Balances accuracy and sensitivity for intent detection.
• FPS_norm — range: percent
  • Normalized False Positive Score measuring noise robustness. Penalizes hallucinations during idle or noisy frames.
• S_final (primary) — range: percent
  • Final composite score calculated as the product of F1_avg and FPS_norm. Rewards both intent detection capability and operational restraint.

Input / output format

Input: A sequence of N=10 resized GUI screenshots per turn, accompanied by the user's profile/context. The model receives these frames incrementally over a trajectory.

Output: The model must output a prediction per turn: either a specific actionable intent, or a 'no action' (noise) designation. In the PIRF setting, it also updates a memory state (CREATE, RESUME, UPDATE, or IDLE).

Scoring recipe

# predictions: list of model outputs per frame/turn
# gold: list of ground truth labels per frame/turn
tp = sum(1 for p, g in zip(predictions, gold) if p == g and p != "noise")
fp = sum(1 for p, g in zip(predictions, gold) if p != g and p != "noise")
fn = sum(1 for p, g in zip(predictions, gold) if p == "noise" and g != "noise")
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1_avg = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
# FPS_norm approximates normalized false positive rate on noise frames
fps_norm = 1.0 - (fp / max(1, total_noise_frames))
s_final = f1_avg * fps_norm
return precision, recall, f1_avg, fps_norm, s_final

Common pitfalls

• High recall alone is misleading; models with high recall but low precision suffer from 'over-proactivity' and receive heavily penalized final scores due to hallucinations during idle moments.
• Visual noise drastically collapses precision across all models, even when recall remains stable or slightly increases, indicating a failure to exercise operational restraint in cluttered streams.
• Human baseline performance is significantly higher but requires 15-20x more inference time, meaning the benchmark metrics do not account for latency or real-time responsiveness.

Evidence (verbatim from paper)

The performance of GPT-5.2 in the Naive setting serves as a stark illustration of the "over-proactivity" trap. While it achieves a remarkable recall of 83.37%, which is the highest among all naive baselines, this sensitivity proves deceptive. The accompanying precision is critically low (31.95%), and its noise robustness score (FPS_norm) is the worst in the table at 31.31%.

Citation

@misc{chai2026pirabench,
  title={PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents},
  author={Chai et al. (2026)},
  year={2026},
  note={arXiv:2603.08013}
}

• arXiv: 2603.08013