Mai UI Agents

Overview

MAI-UI addresses critical limitations in existing GUI agents through pragmatic design choices: extended actions enable richer interactions, device-cloud collaboration preserves privacy, and online RL at scale improves agentic reasoning.

Core Technique

Extended Action Space: Beyond pure UI operations, agents can request clarification and invoke tools.

class ExtendedActionSpace:
    # Actions: click, type, scroll, + new ones
    user_ask = "ask_user"      # Request clarification
    mcp_call = "mcp_call"      # Use external tools

Device-Cloud Collaboration: Local agent monitors alignment; cloud only handles complex cases.

class HybridAgent:
    def should_handoff_to_cloud(self, trajectory, instruction):
        deviation = self.alignment_monitor.evaluate(trajectory)
        if deviation > threshold:
            return True  # Handoff to cloud
        return False  # Continue locally

Key Performance

• 73.5% grounding (ScreenSpot-Pro)
• 76.7% mobile navigation (AndroidWorld)
• 41.7% real-world tasks (MobileWorld)
• 500+ parallel environments

References

• Extended action space design
• Device-cloud collaboration architecture
• Online RL with asynchronous parallelism