# Gui Test Time Scaling > Improve GUI agent planning and action grounding through test-time scaling and reinforcement learning. Sample and evaluate multiple action candidates, then use RL to precisely target visual interface elements. - Skill: `adu2021/gui-test-time-scaling` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/gui-test-time-scaling` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/gui-test-time-scaling/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/gui-test-time-scaling --- # GTA1: GUI Test-Time Scaling and Grounding for Autonomous Agents GUI agents that autonomously complete desktop and web tasks face two fundamental challenges: planning through vast action spaces (where multiple valid action sequences exist) and precisely grounding actions to visual elements in complex, high-resolution interfaces. GTA1 addresses both through test-time compute scaling and reinforcement learning. At each step, multiple action proposals are sampled and evaluated by a judge model, trading computation for decision quality. Simultaneously, RL is applied to ensure actions are accurately grounded to their target elements on screen. The insight is that GUI tasks involve both discrete planning (which action sequence?) and precise grounding (which pixel?). These challenges require different solutions: planning benefits from exploration and judgment; grounding benefits from RL's natural alignment with click success. ## Core Concept GTA1 operates on two key innovations: **Test-Time Scaling for Planning**: Rather than commit to a single action proposal, sample multiple candidates and use a judge model to select the best one. This trades computation (multiple forward passes) for decision quality, shifting the efficiency frontier at inference time. **Reinforcement Learning for Grounding**: Train a grounding module with RL such that rewards directly track "did the click land on the intended element?" This natural objective alignment makes RL highly effective for this problem compared to supervised learning. ## Architecture Overview - **Action proposal module**: Generates multiple candidate action sequences - **Judge model**: Evaluates candidates and selects best proposal - **Visual element detector**: Identifies clickable UI elements from screenshots - **Grounding module**: Maps high-level actions to precise pixel coordinates - **RL environment**: Web or desktop environment providing click success rewards - **Curriculum design**: Starts with simple tasks, progressively increases complexity ## Implementation Set up the action proposal and judge system for test-time scaling: ```python import torch import torch.nn as nn from gui_agent.models import ActionProposer, JudgeModel from gui_agent.env import GUIEnvironment proposer = ActionProposer(model="gpt-4-vision") judge = JudgeModel(model="gpt-4-vision") def plan_with_test_time_scaling(screenshot, task_goal, num_candidates=5): """Plan actions by sampling and evaluating multiple candidates.""" # Sample multiple action proposals candidates = [] for i in range(num_candidates): proposal = proposer.propose_action( screenshot=screenshot, task_goal=task_goal, seed=i # Different seed yields different proposal ) candidates.append(proposal) # Examples of diverse proposals for "click the login button": # Candidate 1: "Click on element at coordinates (450, 200)" # Candidate 2: "Click on the blue button labeled 'Login'" # Candidate 3: "Click in the top-right corner" # Candidate 4: "Submit the form by pressing Enter" # Candidate 5: "Click the element with ID 'login-btn'" # Judge evaluates each candidate scores = [] for candidate in candidates: score = judge.evaluate( screenshot=screenshot, task_goal=task_goal, action_proposal=candidate, context="Are we making progress toward the goal?" ) scores.append(score) # Select highest-scoring proposal best_idx = torch.argmax(torch.tensor(scores)) best_action = candidates[best_idx] return best_action, scores ``` Build the visual element detector to identify targets for grounding: ```python from gui_agent.detection import ElementDetector, GroundingModule detector = ElementDetector(model="yolo-v8-gui") grounding = GroundingModule() def detect_and_ground_action(screenshot, action_description): """Detect visual elements and ground action to precise coordinates.""" # Detect all interactive elements (buttons, links, text fields, etc.) elements = detector.detect(screenshot) # Each element has visual features, text, clickability, etc. # Example detection output: # [ # {"type": "button", "text": "Login", "bbox": [400, 180, 500, 220], "confidence": 0.95}, # {"type": "text_field", "placeholder": "Email", "bbox": [350, 100, 550, 130], "confidence": 0.88}, # {"type": "link", "text": "Forgot password?", "bbox": [450, 250, 550, 270], "confidence": 0.92}, # ] # Ground the high-level action to a specific element # Grounding module must handle: # - Ambiguous descriptions ("the button" when multiple buttons exist) # - Implicit references ("click here" when "here" is contextual) # - Non-matching descriptions (action mentions "red button" but no red button visible) grounded_action = grounding.ground( action_description=action_description, detected_elements=elements, screenshot=screenshot ) # Output: {"element_id": 0, "coordinates": [450, 200], "confidence": 0.87} return grounded_action, elements ``` Train grounding module with reinforcement learning: ```python import torch.optim as optim from gui_agent.rl import GroundingRLTrainer class GroundingRL(nn.Module): def __init__(self): super().__init__() self.grounding = GroundingModule() self.optimizer = optim.Adam(self.grounding.parameters(), lr=1e-4) def train_on_rollout(self, screenshots, actions, success_labels): """Train grounding via RL where reward = click success.""" total_loss = 0 for screenshot, action, success in zip(screenshots, actions, success_labels): # Detect elements elements = detector.detect(screenshot) # Ground the action grounded = self.grounding.ground(action, elements, screenshot) # Simulate the click click_success = simulate_click( screenshot=screenshot, coordinates=grounded["coordinates"], intended_element=find_intended_element(action, elements) ) # Reward is click success (binary or soft) reward = float(click_success) # REINFORCE or policy gradient update # Maximize reward: log P(grounding | action) * reward log_prob = self.grounding.log_probability( grounded["coordinates"], action, elements ) loss = -log_prob * reward # Maximize probability of successful grasps self.optimizer.zero_grad() loss.backward() self.optimizer.step() total_loss += loss.item() return total_loss / len(screenshots) # Initialize trainer grounding_trainer = GroundingRL() # Collect rollouts from environment env = GUIEnvironment(tasks="web_automation_tasks.json") for episode in range(100): task = env.sample_task() rollout = { "screenshots": [], "actions": [], "successes": [] } state = env.reset(task) for step in range(50): # Plan action with test-time scaling action, scores = plan_with_test_time_scaling( state["screenshot"], task["goal"], num_candidates=5 ) # Ground action to visual elements grounded, elements = detect_and_ground_action( state["screenshot"], action ) # Execute in environment success = env.execute_click(grounded["coordinates"]) # Collect for RL training rollout["screenshots"].append(state["screenshot"]) rollout["actions"].append(action) rollout["successes"].append(success) # Transition to next state state = env.step(grounded["coordinates"]) # Train grounding module on this episode's rollout loss = grounding_trainer.train_on_rollout( rollout["screenshots"], rollout["actions"], rollout["successes"] ) print(f"Episode {episode} training loss: {loss:.3f}") ``` ## Practical Guidance ### When to Use GTA1 Use this approach for: - Automating web browsing and form filling tasks - Desktop GUI automation across diverse applications - Tasks where multiple action sequences are valid - Scenarios where action grounding precision is critical - Environments supporting rapid online evaluation (click success is observable) ### When NOT to Use Avoid GTA1 for: - Tasks with unambiguous single-best action sequences - Domains lacking clear feedback signal (click success/failure) - Real-time systems where test-time scaling (multiple forward passes) is too slow - Environments where actions have irreversible consequences - Visual scenes with poor or inconsistent element detection ### Test-Time Scaling Trade-offs | Num Candidates | Planning Quality | Compute Cost | Latency | |----------------|------------------|--------------|---------| | 1 (baseline) | Lower | 1x | Low | | 3 | Improved | 3x | Medium | | 5 | Good | 5x | Medium | | 10+ | Marginal gains | 10x+ | High | Typically 3-5 candidates offer good quality-to-cost ratio. ### Grounding RL Rewards | Feedback Type | Pros | Cons | |---------------|------|------| | Binary (click succeeded) | Clear, objective | Sparse; hard to learn from | | Soft (distance to target) | Dense reward signal | May reward wrong clicks | | Multi-step (progress toward goal) | Long-horizon credit | Requires task model | Start with binary rewards; switch to soft if learning plateaus. ### Element Detection Quality Impact | Detection Recall | Grounding Success | Notes | |------------------|-------------------|-------| | >95% | High | Grounding can focus on found elements | | 85-95% | Medium | Missing elements create target confusion | | <85% | Low | Grounding module must guess unmapped elements | Ensure element detection is robust (>90% recall) before training grounding RL. ### Key Hyperparameters | Parameter | Typical Range | Guidance | |-----------|---------------|----------| | Test-time candidates | 3-5 | 5 good default; increase if budget allows | | Judge temperature | 0.7-1.0 | Lower temp = more confident judge decisions | | Grounding LR | 1e-4 to 5e-4 | Standard RL rates work | | RL discount factor | 0.99 | Standard; long-horizon tasks may need 0.95 | | Curriculum difficulty | Gradual increase | Start easy (single button clicks), end complex (multi-step workflows) | ### Common Pitfalls 1. **Oversimplifying action space**: Real GUI agents need diverse action types (click, type, scroll, drag). Handle each appropriately. 2. **Ignoring element detection failures**: When detector misses elements, grounding becomes impossible. Validate detection first. 3. **Using binary rewards too long**: Pure binary rewards (click success/failure) provide weak signal. Add soft rewards (distance to target, progress metrics). 4. **Insufficient curriculum**: Jumping to complex tasks causes RL to fail. Use curriculum: simple clicks → simple forms → complex workflows. 5. **Forgetting state representation**: GUI state is high-dimensional (screenshots). Use vision encoders to compress effectively. ### Evaluation Metrics - **Planning success**: Does judge-selected action make progress toward goal? - **Grounding accuracy**: Does click land on intended element? (Measure by visual inspection or ground truth annotations) - **Task completion rate**: What % of tasks are fully completed? (Multi-step metric) - **Efficiency**: Steps per task, wall-clock time, compute cost ### Curriculum Learning Strategy ```python # Phase 1: Simple single-element tasks tasks_easy = [ "Click the login button", "Click the search box", "Click 'Agree' on popup" ] # Phase 2: Multi-element single-step tasks tasks_medium = [ "Fill email field then click submit", "Click dropdown and select option", "Type in search and press enter" ] # Phase 3: Multi-step workflows tasks_hard = [ "Book a flight: search, filter, select, checkout", "Complete account registration across multiple screens", "Fill form, fix errors, resubmit" ] # Train in phases, graduating to harder tasks as RL improves for phase, tasks in enumerate([tasks_easy, tasks_medium, tasks_hard]): for epoch in range(100): # Train on current difficulty loss = grounding_trainer.train_on_rollouts(tasks) ``` ## Reference "GTA1: GUI Test-time Scaling Agent" - [arXiv:2507.05791](https://arxiv.org/abs/2507.05791)