Unreasonable Scaling Computer Use Agents

Unreasonable Effectiveness of Scaling Computer-Use Agents with Behavior Judgment

This work identifies a fundamental bottleneck in scaling computer-use agents: "wide scaling exposes a fundamental bottleneck: evaluation." The solution converts dense agent trajectories into compact behavior narratives capturing task-relevant changes, enabling intelligent best-of-N selection via comparative evaluation.

Core Architecture

• Parallel trajectory execution: Generate multiple independent agent rollouts per task
• Behavior narrative generation: Compress trajectories into task-relevant summaries (10-20% original length)
• Comparative evaluation: Select best trajectory through structured pairwise comparison
• Cross-platform validation: Works on OSWorld (desktop), WindowsAgentArena (Windows), AndroidWorld (mobile)

Implementation Steps

Setup parallel agent executor with behavior narrative evaluation:

# Initialize parallel computer-use agent system
from bjudge import ParallelAgentExecutor, BehaviorNarrativeGenerator

executor = ParallelAgentExecutor(
    base_agent=your_computer_agent,
    num_rollouts=4,  # generate 4 parallel attempts per task
    executor="isolated_vms",  # each rollout in isolated environment
    snapshot_capability=True  # required for rollback
)

narrative_generator = BehaviorNarrativeGenerator(
    llm=your_evaluator_model,
    max_narrative_length=1000,  # tokens
    focus="task_relevant_changes"
)

Execute parallel agent execution with trajectory selection:

# Task execution with parallel rollouts
task = "Download Q1 financial reports from company website and extract revenue"

# Stage 1: Generate multiple rollouts in parallel
trajectories = executor.execute_parallel(
    task=task,
    num_rollouts=4,
    max_steps_per_rollout=50,
    environment_snapshots=True
)

# Stage 2: Convert trajectories to behavior narratives
narratives = []
for trajectory in trajectories:
    narrative = narrative_generator.generate(
        trajectory=trajectory,
        task_description=task,
        include_action_sequence=True,
        include_state_changes=True,
        summarization_ratio=0.1  # compress to 10% of original
    )
    narratives.append(narrative)

# Stage 3: Comparative evaluation to select best trajectory
best_trajectory_idx = executor.select_best_trajectory(
    narratives=narratives,
    task=task,
    evaluation_criteria=[
        "task_completion",
        "action_efficiency",
        "error_recovery",
        "state_consistency"
    ]
)

# Stage 4: Return best result
best_result = trajectories[best_trajectory_idx]
print(f"Task success rate: {best_result.success}, Steps: {best_result.num_steps}")

Practical Guidance

When to use Behavior Judgment:

• Desktop automation with complex, multi-step workflows
• Web scraping and data extraction tasks with high variance
• Software testing and interaction workflows
• Systems where user trust depends on interpretable agent decisions

When NOT to use:

• Real-time systems (parallel rollouts add latency overhead)
• Tasks where reproducibility is essential (multiple attempts may produce different results)
• Systems without snapshot/rollback capability (required for isolated rollouts)
• Single-step deterministic tasks (parallelization overhead not justified)

Performance characteristics:

• OSWorld (desktop): 72.6% success rate (up from 30% baseline)
• WindowsAgentArena: 56.6% success rate
• AndroidWorld: 71.6% success rate
• Human performance on OSWorld: ~75% (agent approaching human level)

Hyperparameters:

• Num rollouts (4): Sweet spot between quality and latency. Test 3-5; 2 insufficient, 6+ diminishing returns
• Max steps (50): Adjust based on task complexity; increase to 80 for very complex workflows
• Narrative length (1000 tokens): Must fit in evaluator context; reduce to 500 for larger models
• Summarization ratio (0.1): 10% of original trajectory length; adjust to 0.05-0.15 based on task specificity

Computational requirements:

• Parallel VM setup: 4 isolated environments for baseline (4 simultaneous rollouts)
• Total latency: 4x single-agent time (sequential execution) or near-equal (parallel VMs)
• Evaluation overhead: 10-15% additional compute for trajectory selection

Trajectory Compression Strategy

Behavior narratives preserve critical information while eliminating noise:

• Keep: State changes, tool results, error messages, decision points
• Discard: Intermediate steps, UI details, navigation noise
• Compress: Action sequences into high-level summaries (e.g., "searched for Q1 report PDF")

Cross-Domain Results

The approach generalizes across distinct platforms:

• OSWorld: Complex desktop web interactions
• WindowsAgentArena: Windows-specific applications
• AndroidWorld: Mobile app workflows
• Performance scales reasonably across domains (56-72% success)

Key Limitation

Approach requires isolated execution environments capable of snapshots. Standard cloud VMs support this; real user systems may not, limiting real-world deployment in some contexts.

References

Builds on work in agent evaluation, trajectory-based learning, and multi-rollout selection strategies.