# Agent S

> Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. It combines vision and action understanding to interact with any GUI element. Use when this capability is needed.

- Skill: `tomevault-io/agent-s` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/agent-s`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/agent-s/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/agent-s

---

# Agent-S - Autonomous GUI Agent

Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. It combines vision and action understanding to interact with any GUI element.

## What It Does

Agent-S can:
- Navigate and interact with desktop applications
- Fill forms, click buttons, and manipulate GUI elements
- Complete multi-step workflows across different applications
- Take screenshots and understand visual interfaces
- Execute complex GUI automation tasks autonomously

## When to Use

Use Agent-S when you need to:
- Automate GUI-based tasks that don't have CLI alternatives
- Interact with desktop applications programmatically
- Complete workflows that require visual understanding
- Perform actions across multiple applications
- Test GUI interfaces

## How to Invoke

Call the Agent-S wrapper via bash from the OpenClaw skills directory:

```bash
./agent_s_task "task description"
```

Or if installed in the default OpenClaw skills location:

```bash
~/.openclaw/workspace/skills/agent-s/agent_s_task "task description"
```

**Note**: Agent-S tasks can take 2-5 minutes to complete (up to 15 steps by default). The wrapper will wait for completion.

## Parameters

- `task` (required): Natural language description of the GUI task to complete
- `max_steps` (optional): Maximum steps the agent can take (default: 15)
- `enable_reflection` (optional): Enable self-reflection for better performance (default: true)

## Examples

```python
# Basic navigation
agent_s_task(task="Open Finder and create a new folder called 'Reports'")

# Form filling
agent_s_task(task="Open TextEdit, create a new document, and type 'Hello World'")

# Multi-step workflows
agent_s_task(task="Open Chrome, search for 'Python tutorials', and bookmark the first result")

# Application interaction
agent_s_task(task="Open System Preferences and check the current display resolution")
```

## Technical Details

Agent-S uses:
- **Main Model**: Claude Sonnet 4.5 for reasoning and planning
- **Grounding Model**: UI-TARS-1.5-7B for visual grounding and coordinate extraction
- **Screen Resolution**: Automatically scaled to 2400px max dimension
- **Platform Support**: macOS, Linux, Windows

## Safety

- Agent-S has full GUI control - only use for trusted tasks
- The agent will pause on Ctrl+C and can be resumed with Esc
- Each action is logged to `~/workspace/Agent-S/logs/`
- Tasks timeout after 15 steps by default

## Configuration

Agent-S requires configuration via environment variables:

**Required:**
- `ANTHROPIC_API_KEY`: API key for Claude model
- `AGENT_S_GROUND_URL`: Grounding model endpoint URL
- `AGENT_S_GROUND_MODEL`: Grounding model name (default: ui-tars-1.5-7b)
- `AGENT_S_GROUNDING_WIDTH`: Output width (default: 1920)
- `AGENT_S_GROUNDING_HEIGHT`: Output height (default: 1080)

**Optional:**
- `AGENT_S_GROUND_API_KEY`: API key for grounding endpoint

See the README.md in this directory for detailed setup instructions.

## Limitations

- Cannot interact with system-level dialogs requiring admin approval
- Performance depends on screen resolution and GUI complexity
- Some applications may have accessibility restrictions
- Voice/audio commands are not supported

## Source

Agent-S GitHub: https://github.com/simular-ai/Agent-S
Installation: `pip install gui-agents`

---
> Source: [simular-ai/Agent-S](https://github.com/simular-ai/Agent-S) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-06-24 -->