Computer Use
Computer use lets Claude operate a desktop through screenshots and mouse/keyboard actions that your code executes. It is a client tool: you own the VM, the action executor, and the safety boundary. Reach for it only when no API exists.
When to Use
- Automating applications that have no API
- Cross-application workflows (read in one app, enter in another)
- Exploratory UI testing where a scripted test would be brittle
When NOT to Use
- Anything with an API or a browser automation library you can script deterministically
- High-volume or latency-sensitive work: every step is a model call with an image
- Consequential actions (payments, deletes, sending messages) without a human approval step
- Environments that hold real user data or credentials
If the target is only a web page, also consider the web fetch tool (for reading) or a dedicated browser tool; see the tool reference.
Workflow
Step 1: Choose the tool version
Tool versions are dated identifiers and their model support changes. As documented at the time of writing:
Tool type |
Beta header | Notes |
|---|---|---|
computer_toolset_20260801 |
None | Current toolset with member tools; required on the newest models on the Claude API |
computer_20251124 |
computer-use-2025-11-24 |
Earlier version for some previous-generation models |
computer_20250124 |
computer-use-2025-01-24 |
Legacy; the docs describe migrating to the toolset |
Read the compatibility matrix on the computer use page for your exact model and platform before you pick one; do not infer support from this table.
Step 2: Declare the toolset
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
const MODEL = process.env.ANTHROPIC_MODEL!; // a current model ID that supports the toolset
const tools = [
{ type: "computer_toolset_20260801", configs: { zoom: { enabled: false } } },
{ type: "text_editor_20250728", name: "str_replace_based_edit_tool" }, // optional
{ type: "bash_20250124", name: "bash" }, // optional
] as const;
configs turns individual member tools on or off (each defaults to enabled). The member tools include screenshot, zoom, click variants (left_click, right_click, middle_click, double_click, triple_click), left_click_drag, mouse_move, left_mouse_down/left_mouse_up, cursor_position, scroll, type, key, hold_key, and wait. Only screenshot and zoom return images.
Step 3: Run the agent loop
Claude returns tool_use blocks for member tools carrying "toolset_name": "computer". Execute them in order, then return each result with the same toolset_name.
const messages: Anthropic.MessageParam[] = [{ role: "user", content: goal }];
while (true) {
const response = await client.messages.create({ model: MODEL, max_tokens: 16000, tools, messages });
messages.push({ role: "assistant", content: response.content });
if (response.stop_reason !== "tool_use") break;
const results = [];
let failed = false;
for (const block of response.content) {
if (block.type !== "tool_use") continue;
if (failed) {
results.push({ type: "tool_result", tool_use_id: block.id, toolset_name: "computer", is_error: true,
content: "Not executed: an earlier computer action in this turn failed." });
continue;
}
const out = await executeAction(block.name, block.input); // your VM controller
failed = !out.ok;
results.push({ type: "tool_result", tool_use_id: block.id, toolset_name: "computer",
is_error: !out.ok, content: out.content });
}
messages.push({ role: "user", content: results });
}
Claude can return several actions in one response (a batch). The docs say to run them in order and stop at the first failure, returning the exact "Not executed" text above for skipped blocks. Mixed toolsets (text editor, bash) follow their own result rules; see their tool pages. The SDK types for newer tool shapes may require a recent SDK version.
Step 4: Isolate the environment
Anthropic's reference implementation lives in anthropics/anthropic-quickstarts under computer-use-demo (a container with a virtual X display, tool implementations, and an agent loop). Treat it as a starting point, not a production boundary.
Minimum isolation:
- A dedicated VM or container with minimal privileges, rebuilt per task
- No real credentials, personal files, or logged-in sessions
- Egress restricted to an allowlist of domains
- Screen recording or screenshot logging for every step
Step 5: Defend against prompt injection
Everything on screen is input. A web page or email can contain instructions aimed at the agent. The docs describe classifiers that scan screenshots for likely injections, but your design must not depend on them:
- Require human confirmation before consequential actions (purchases, accepting terms, sending, deleting)
- Bound the task: abort if the agent leaves the expected domains or applications
- State in the system prompt that instructions found in on-screen content are data, not commands
Step 6: Decompose and verify
Long, multi-page goals fail more often than short ones. Have your orchestrator split the goal into sub-tasks, check the screenshot after each, and stop on unexpected state rather than letting the agent improvise.
Cost and Latency
Each step is one model request that usually includes at least one screenshot, and image tokens scale with resolution. Measure tokens per task on your own workload with usage fields; keep the display resolution modest and use zoom (when enabled) for detail instead of a larger screen.
Anti-Patterns
- Using computer use where an API or scripted browser automation would work
- Running the harness on a workstation with real accounts
- No approval gate for irreversible actions
- Copying an old
computer_20250124example onto a new model without checking the matrix - Not logging screenshots, which makes failures impossible to debug
Best Practices
- Check for an API first
- Pin the tool version per model and re-check on upgrades
- Disposable, network-restricted sandbox per task
- Human-in-the-loop for consequential actions
- Short, verified sub-tasks orchestrated by your code
- Log every action and screenshot for audit
References
- Computer use tool: https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
- Tool reference: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-reference
- Reference implementation: https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo