Computer Use Agents
Build or review agents that interact with computers through screenshots, vision reasoning, and mouse or keyboard actions. Keep the entrypoint focused on architecture, safety boundaries, and the decision points that determine which deep implementation pattern to load.
When to Use
Designing a perception-reasoning-action loop for browser, desktop, or VM control.
Adding safety controls for autonomous clicking, typing, scrolling, or keyboard use.
Comparing Anthropic Computer Use, OpenAI CUA/Operator-style flows, browser-use, or Playwright-backed approaches.
Auditing an existing computer-use agent for sandboxing, step limits, cost limits, and action verification.
When Not to Use
A deterministic DOM or API automation path can solve the task with less risk.
The request requires controlling a real user machine without an explicit sandbox and rollback plan.
The task is just visual QA or browser testing and does not need an autonomous action loop.
Core Workflow
Clarify the target surface, user goal, allowed actions, credential boundaries, and stop conditions.
Choose the narrowest control layer: DOM/API first, browser automation second, full screen control only when required.
Implement the observe-plan-act loop with an explicit action schema, max steps, timeouts, and result verification after every important action.
Run inside a sandboxed environment with network, filesystem, user, memory, and process limits appropriate to the task.
Log screenshots, model decisions, actions, errors, retries, and costs so failures can be replayed and audited.
Test in an isolated environment before allowing real accounts, payments, production systems, or destructive actions.
Reference Map
Read references/full-guidance.md when the task needs the complete examples and edge cases. It includes:
Perception-reasoning-action loop implementation and action schema examples.
Sandboxed environment, Anthropic Computer Use, browser-use, confirmation, and action-logging patterns.
Sharp edges around prompt injection, unreliable UI controls, context/cost growth, and running agents on real computers.
Validation checks for missing sandboxing, absent step limits, root containers, missing seccomp, and no cost tracking.
Safety and Quality Rules
Prefer read-only or reversible actions until the agent proves it can observe and verify accurately.
Require user confirmation for purchases, account changes, destructive operations, credential entry, and external communication.
Never rely on visual reasoning alone for high-stakes state; verify through DOM, logs, API responses, or screenshots after action.
Progressive Loading
Start with this entrypoint for routing and planning. Load references/full-guidance.md only after the task clearly requires deep implementation detail, code examples, validation checklists, or troubleshooting guidance.
1---2name: computer-use-agents3description: Use when building, reviewing, or hardening AI agents that observe screens and control mouse, keyboard, browser, or desktop environments.4license: MIT5---67# Computer Use Agents89Build or review agents that interact with computers through screenshots, vision reasoning, and mouse or keyboard actions. Keep the entrypoint focused on architecture, safety boundaries, and the decision points that determine which deep implementation pattern to load.1011## When to Use1213- Designing a perception-reasoning-action loop for browser, desktop, or VM control.1415- Adding safety controls for autonomous clicking, typing, scrolling, or keyboard use.1617- Comparing Anthropic Computer Use, OpenAI CUA/Operator-style flows, browser-use, or Playwright-backed approaches.1819- Auditing an existing computer-use agent for sandboxing, step limits, cost limits, and action verification.2021## When Not to Use2223- A deterministic DOM or API automation path can solve the task with less risk.2425- The request requires controlling a real user machine without an explicit sandbox and rollback plan.2627- The task is just visual QA or browser testing and does not need an autonomous action loop.2829## Core Workflow30311. Clarify the target surface, user goal, allowed actions, credential boundaries, and stop conditions.32332. Choose the narrowest control layer: DOM/API first, browser automation second, full screen control only when required.34353. Implement the observe-plan-act loop with an explicit action schema, max steps, timeouts, and result verification after every important action.36374. Run inside a sandboxed environment with network, filesystem, user, memory, and process limits appropriate to the task.38395. Log screenshots, model decisions, actions, errors, retries, and costs so failures can be replayed and audited.40416. Test in an isolated environment before allowing real accounts, payments, production systems, or destructive actions.4243## Reference Map4445Read `references/full-guidance.md` when the task needs the complete examples and edge cases. It includes:4647- Perception-reasoning-action loop implementation and action schema examples.4849- Sandboxed environment, Anthropic Computer Use, browser-use, confirmation, and action-logging patterns.5051- Sharp edges around prompt injection, unreliable UI controls, context/cost growth, and running agents on real computers.5253- Validation checks for missing sandboxing, absent step limits, root containers, missing seccomp, and no cost tracking.5455## Safety and Quality Rules5657- Prefer read-only or reversible actions until the agent proves it can observe and verify accurately.5859- Require user confirmation for purchases, account changes, destructive operations, credential entry, and external communication.6061- Never rely on visual reasoning alone for high-stakes state; verify through DOM, logs, API responses, or screenshots after action.6263## Progressive Loading6465Start with this entrypoint for routing and planning. Load `references/full-guidance.md` only after the task clearly requires deep implementation detail, code examples, validation checklists, or troubleshooting guidance.
newmindsgroup