Openclaw

Agent-S - Autonomous GUI Agent

Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. It combines vision and action understanding to interact with any GUI element.

What It Does

Agent-S can:

• Navigate and interact with desktop applications
• Fill forms, click buttons, and manipulate GUI elements
• Complete multi-step workflows across different applications
• Take screenshots and understand visual interfaces
• Execute complex GUI automation tasks autonomously

When to Use

Use Agent-S when you need to:

• Automate GUI-based tasks that don't have CLI alternatives
• Interact with desktop applications programmatically
• Complete workflows that require visual understanding
• Perform actions across multiple applications
• Test GUI interfaces

How to Invoke

Call the Agent-S wrapper via bash from the OpenClaw skills directory:

./agent_s_task "task description"

Or if installed in the default OpenClaw skills location:

~/.openclaw/workspace/skills/agent-s/agent_s_task "task description"

Note: Agent-S tasks can take 2-5 minutes to complete (up to 15 steps by default). The wrapper will wait for completion.

Parameters

• task (required): Natural language description of the GUI task to complete
• max_steps (optional): Maximum steps the agent can take (default: 15)
• enable_reflection (optional): Enable self-reflection for better performance (default: true)

Examples

# Basic navigation
agent_s_task(task="Open Finder and create a new folder called 'Reports'")

# Form filling
agent_s_task(task="Open TextEdit, create a new document, and type 'Hello World'")

# Multi-step workflows
agent_s_task(task="Open Chrome, search for 'Python tutorials', and bookmark the first result")

# Application interaction
agent_s_task(task="Open System Preferences and check the current display resolution")

Technical Details

Agent-S uses:

• Main Model: Claude Sonnet 4.5 for reasoning and planning
• Grounding Model: UI-TARS-1.5-7B for visual grounding and coordinate extraction
• Screen Resolution: Automatically scaled to 2400px max dimension
• Platform Support: macOS, Linux, Windows

Safety

• Agent-S has full GUI control - only use for trusted tasks
• The agent will pause on Ctrl+C and can be resumed with Esc
• Each action is logged to ~/workspace/Agent-S/logs/
• Tasks timeout after 15 steps by default

Configuration

Agent-S requires configuration via environment variables:

Required:

• ANTHROPIC_API_KEY: API key for Claude model
• AGENT_S_GROUND_URL: Grounding model endpoint URL
• AGENT_S_GROUND_MODEL: Grounding model name (default: ui-tars-1.5-7b)
• AGENT_S_GROUNDING_WIDTH: Output width (default: 1920)
• AGENT_S_GROUNDING_HEIGHT: Output height (default: 1080)

Optional:

• AGENT_S_GROUND_API_KEY: API key for grounding endpoint

See the README.md in this directory for detailed setup instructions.

Limitations

• Cannot interact with system-level dialogs requiring admin approval
• Performance depends on screen resolution and GUI complexity
• Some applications may have accessibility restrictions
• Voice/audio commands are not supported

Source

Agent-S GitHub: https://github.com/simular-ai/Agent-S Installation: pip install gui-agents