Agent S operator
Operate Agent S3 from the gui-agents package as a supervised tool, never as an unattended one. This
skill was audited against commit 0fff0e634869dab72027435b4586e62300185d41 (2026-10-21) and the
published gui-agents 0.3.2 wheel; audited facts, hazards and the pin are in
references/upstream.md.
When to use this skill
- Fit-check, install, configure, preflight, run, or troubleshoot Agent S3 and its
agent_s CLI.
- Choose providers, set up a grounding model (UI-TARS), or explain
--ground_* and --grounding_* flags.
- Decide whether
--enable_local_env is acceptable (it runs arbitrary Python and Bash).
- Not for Playwright or Selenium test automation, tasks with a CLI or API (
browser-harness, stagehand), or Agent S4's AWS-VM evolution loop.
Instructions
1. Freeze the surface
Record one mode (orient, install, configure, run, troubleshoot), one concrete task, and the
machine it will run on. Prefer a disposable VM, a separate OS user, or a cleared desktop. The model
sees the whole screen on every step.
2. Run the offline gate first
The helper reads host facts and, with --repo, a checkout. It never starts Agent S, pip, a model
call or a screenshot, opens no sockets, and writes nothing:
python3 .agent-skills/agent-s/scripts/preflight.py --json
python3 .agent-skills/agent-s/scripts/preflight.py --repo /path/to/Agent-S --expect-commit 0fff0e634869dab72027435b4586e62300185d41 --json
An installed skill uses python3 "$SKILLS_ROOT/agent-s/scripts/preflight.py". Read the python
check carefully: pip refuses gui-agents 0.3.2 on Python 3.12.1 and later, so the environment needs
3.9 to 3.11 (or exactly 3.12.0). audit:* lines report what the checkout's cli_app.py actually does.
3. Gate every side effect
Require separate explicit approval for each:
pip install gui-agents (or pip install -e .), which pulls heavy dependencies such as paddleocr and paddlepaddle, plus brew install tesseract.
- Creating or using a grounding endpoint (UI-TARS on Hugging Face Inference Endpoints or another provider; paid, and it receives every screenshot).
- Exporting provider keys; never paste them into chat or logs.
- The first run: the agent clicks and types on the real desktop and executes model-generated code.
--enable_local_env: arbitrary Python and Bash with the user's permissions. Allow it only inside a disposable VM or container.
- Any task that sends messages, spends money, deletes data, or touches credentials: require a human confirmation step the tool itself does not provide.
4. Follow the matching reference branch
- Install, Python range, dependencies → Install in
references/upstream.md.
- Models, grounding, flags, coordinates → Models and grounding.
- What the run loop does, logs, pause keys → Run loop.
- Safety and the unused permission dialog → Safety findings.
- Agent S4, OSWorld, OpenClaw wrapper → Other upstream surfaces.
5. Verify narrowly
Start with a harmless task such as opening a calculator or a blank text editor, with --task "<one sentence>"
and a person watching. Ctrl+C pauses the run, Esc resumes, a second Ctrl+C quits. Report what the agent
did from the screen and the run's logs/ files, not from the model's claim of success.
Examples
Fit check before installing
Run the preflight, report the Python verdict and the tesseract result, list the model and grounding
endpoints the user must supply, and state that screenshots go to them. Do not install.
"Let it clean my downloads folder"
Decline an unattended run on the personal machine. Offer a VM, a one-folder task with
--task, no --enable_local_env, and a human watching with Ctrl+C ready.
Best practices
- Treat the generated code as untrusted input: this version executes it directly with
exec().
- Do not assume a confirmation dialog exists:
show_permission_dialog is defined in cli_app.py but never called.
- Match
--grounding_width and --grounding_height to the grounding model's output space (1920x1080 for UI-TARS-1.5-7B, 1000x1000 for UI-TARS-72B).
- Close password managers, chats and mail before a run; the model reads whatever is on screen.
- Keep the helper offline and honest: a tool on PATH is not a working model endpoint or screen permission.
References
1---2name: agent-s3description: >4 Operate simular-ai/Agent-S, the Apache-2.0 open-source computer-use agent framework whose5 Agent S3 (`gui-agents` 0.3.2, `agent_s` CLI) screenshots the desktop, asks a multimodal model6 plus a grounding model what to do, and executes generated pyautogui code to click, type and7 scroll on macOS, Windows or Linux. Use when the user names Agent S, Agent-S, Simular,8 `gui-agents`, `agent_s`, UI-TARS grounding, or an OSWorld-style GUI agent and needs a fit9 check, install, model and grounding setup, an offline preflight, a supervised run, or10 troubleshooting. Do not use for ordinary browser automation, tasks with a CLI or API,11 unsupervised runs on a personal machine, or Agent S4's AWS benchmark evolution. Triggers on:12 Agent S, Agent-S, gui-agents, agent_s, computer use agent, 컴퓨터 사용 에이전트, 화면 보고 PC 조작하는 AI.13allowed-tools: Bash Read Write Edit Glob Grep14compatibility: >15 Agent S3 needs Python 3.9 to 3.12.0 for pip (the published Requires-Python is '>=3.9, <=3.12',16 which excludes 3.12.1 and later), the tesseract binary, a single monitor, a multimodal model17 API key, and a reachable grounding model endpoint. Every screenshot leaves the machine for the18 configured endpoints. Use a disposable desktop for anything sensitive.19license: Apache-2.020metadata:21 version: "1.0.0"22 source: https://skillmd.com/skills/akillness/agent-s23 upstream: https://github.com/simular-ai/Agent-S24 upstream_commit: 0fff0e634869dab72027435b4586e62300185d4125---2627# Agent S operator2829Operate Agent S3 from the `gui-agents` package as a supervised tool, never as an unattended one. This30skill was audited against commit `0fff0e634869dab72027435b4586e62300185d41` (2026-10-21) and the31published `gui-agents` 0.3.2 wheel; audited facts, hazards and the pin are in32[`references/upstream.md`](references/upstream.md).3334## When to use this skill3536- Fit-check, install, configure, preflight, run, or troubleshoot Agent S3 and its `agent_s` CLI.37- Choose providers, set up a grounding model (UI-TARS), or explain `--ground_*` and `--grounding_*` flags.38- Decide whether `--enable_local_env` is acceptable (it runs arbitrary Python and Bash).39- Not for Playwright or Selenium test automation, tasks with a CLI or API (`browser-harness`, `stagehand`), or Agent S4's AWS-VM evolution loop.4041## Instructions4243### 1. Freeze the surface4445Record one mode (`orient`, `install`, `configure`, `run`, `troubleshoot`), one concrete task, and the46machine it will run on. Prefer a disposable VM, a separate OS user, or a cleared desktop. The model47sees the whole screen on every step.4849### 2. Run the offline gate first5051The helper reads host facts and, with `--repo`, a checkout. It never starts Agent S, pip, a model52call or a screenshot, opens no sockets, and writes nothing:5354```bash55python3 .agent-skills/agent-s/scripts/preflight.py --json56python3 .agent-skills/agent-s/scripts/preflight.py --repo /path/to/Agent-S --expect-commit 0fff0e634869dab72027435b4586e62300185d41 --json57```5859An installed skill uses `python3 "$SKILLS_ROOT/agent-s/scripts/preflight.py"`. Read the `python`60check carefully: pip refuses `gui-agents` 0.3.2 on Python 3.12.1 and later, so the environment needs613.9 to 3.11 (or exactly 3.12.0). `audit:*` lines report what the checkout's `cli_app.py` actually does.6263### 3. Gate every side effect6465Require separate explicit approval for each:66671. `pip install gui-agents` (or `pip install -e .`), which pulls heavy dependencies such as `paddleocr` and `paddlepaddle`, plus `brew install tesseract`.682. Creating or using a grounding endpoint (UI-TARS on Hugging Face Inference Endpoints or another provider; paid, and it receives every screenshot).693. Exporting provider keys; never paste them into chat or logs.704. The first run: the agent clicks and types on the real desktop and executes model-generated code.715. `--enable_local_env`: arbitrary Python and Bash with the user's permissions. Allow it only inside a disposable VM or container.726. Any task that sends messages, spends money, deletes data, or touches credentials: require a human confirmation step the tool itself does not provide.7374### 4. Follow the matching reference branch7576- Install, Python range, dependencies → **Install** in `references/upstream.md`.77- Models, grounding, flags, coordinates → **Models and grounding**.78- What the run loop does, logs, pause keys → **Run loop**.79- Safety and the unused permission dialog → **Safety findings**.80- Agent S4, OSWorld, OpenClaw wrapper → **Other upstream surfaces**.8182### 5. Verify narrowly8384Start with a harmless task such as opening a calculator or a blank text editor, with `--task "<one sentence>"`85and a person watching. Ctrl+C pauses the run, Esc resumes, a second Ctrl+C quits. Report what the agent86did from the screen and the run's `logs/` files, not from the model's claim of success.8788## Examples8990### Fit check before installing9192Run the preflight, report the Python verdict and the `tesseract` result, list the model and grounding93endpoints the user must supply, and state that screenshots go to them. Do not install.9495### "Let it clean my downloads folder"9697Decline an unattended run on the personal machine. Offer a VM, a one-folder task with98`--task`, no `--enable_local_env`, and a human watching with Ctrl+C ready.99100## Best practices101102- Treat the generated code as untrusted input: this version executes it directly with `exec()`.103- Do not assume a confirmation dialog exists: `show_permission_dialog` is defined in `cli_app.py` but never called.104- Match `--grounding_width` and `--grounding_height` to the grounding model's output space (1920x1080 for UI-TARS-1.5-7B, 1000x1000 for UI-TARS-72B).105- Close password managers, chats and mail before a run; the model reads whatever is on screen.106- Keep the helper offline and honest: a tool on PATH is not a working model endpoint or screen permission.107108## References109110- [Audited Agent S facts, hazards and pin](references/upstream.md)111- [Pinned source manifest](references/source.json)112- [Agent S3 README at the audited commit](https://github.com/simular-ai/Agent-S/blob/0fff0e634869dab72027435b4586e62300185d41/gui_agents/s3/README.md)113- [Supported model APIs](https://github.com/simular-ai/Agent-S/blob/0fff0e634869dab72027435b4586e62300185d41/models.md)114- [Apache-2.0 license at the pin](https://github.com/simular-ai/Agent-S/blob/0fff0e634869dab72027435b4586e62300185d41/LICENSE)115- [Offline helper](scripts/preflight.py)