Computer Use Agents V2

Computer Use Agents

Overview

This public intake copy packages plugins/antigravity-awesome-skills/skills/computer-use-agents from https://github.com/sickn33/antigravity-awesome-skills into the native Omni Skills editorial shape without hiding its origin.

Use it when the operator needs the upstream workflow, support files, and repository context to stay intact while the public validator and private enhancer continue their normal downstream flow.

This intake keeps the copied upstream files intact and uses the external_source block in metadata.json plus ORIGIN.md as the provenance anchor for review.

Computer Use Agents Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control.

Imported source sections that did not map cleanly to the public headings are still preserved below or in the support files. Notable imported sections: Patterns, Sharp Edges, Defense in depth - no single solution works, Add human-like variance to actions, Rotate user agents and fingerprints, Fall back to DOM access for web.

When to Use This Skill

Use this section as the trigger filter. It should make the activation boundary explicit before the operator loads files, runs commands, or opens a pull request.

• User mentions or implies: computer use
• User mentions or implies: desktop automation agent
• User mentions or implies: screen control AI
• User mentions or implies: vision-based agent
• User mentions or implies: GUI automation
• User mentions or implies: Claude computer

Operating Table

Workflow

This workflow is intentionally editorial and operational at the same time. It keeps the imported source useful to the operator while still satisfying the public intake standards that feed the downstream enhancer flow.

1. python # Instead of drag-and-drop async def reliable_drag(page, source, target): # Step 1: Click and hold await page.mouse.move(source["x"], source["y"]) await page.mouse.down() await asyncio.sleep(0.2) # Step 2: Move in steps steps = 10 for i in range(steps): x = source["x"] + (target["x"] - source["x"]) i / steps y = source["y"] + (target["y"] - source["y"]) i / steps await page.mouse.move(x, y) await asyncio.sleep(0.05) # Step 3: Release await page.mouse.move(target["x"], target["y"]) await asyncio.sleep(0.1) await page.mouse.up()
2. Confirm the user goal, the scope of the imported workflow, and whether this skill is still the right router for the task.
3. Read the overview and provenance files before loading any copied upstream support files.
4. Load only the references, examples, prompts, or scripts that materially change the outcome for the current request.
5. Execute the upstream workflow while keeping provenance and source boundaries explicit in the working notes.
6. Validate the result against the upstream expectations and the evidence you can point to in the copied files.
7. Escalate or hand off to a related skill when the work moves out of this imported workflow's center of gravity.

Imported Workflow Notes

Imported: Break complex actions into steps

# Instead of drag-and-drop
async def reliable_drag(page, source, target):
    # Step 1: Click and hold
    await page.mouse.move(source["x"], source["y"])
    await page.mouse.down()
    await asyncio.sleep(0.2)

    # Step 2: Move in steps
    steps = 10
    for i in range(steps):
        x = source["x"] + (target["x"] - source["x"]) * i / steps
        y = source["y"] + (target["y"] - source["y"]) * i / steps
        await page.mouse.move(x, y)
        await asyncio.sleep(0.05)

    # Step 3: Release
    await page.mouse.move(target["x"], target["y"])
    await asyncio.sleep(0.1)
    await page.mouse.up()

Imported: Patterns

Perception-Reasoning-Action Loop

The fundamental architecture of computer use agents: observe screen, reason about next action, execute action, repeat. This loop integrates vision models with action execution through an iterative pipeline.

Key components:

1. PERCEPTION: Screenshot captures current screen state
2. REASONING: Vision-language model analyzes and plans
3. ACTION: Execute mouse/keyboard operations
4. FEEDBACK: Observe result, continue or correct

Critical insight: Vision agents are completely still during "thinking" phase (1-5 seconds), creating a detectable pause pattern.

When to use: Building any computer use agent from scratch,Integrating vision models with desktop control,Understanding agent behavior patterns

from anthropic import Anthropic from PIL import Image import base64 import pyautogui import time

class ComputerUseAgent: """ Perception-Reasoning-Action loop implementation. Based on Anthropic Computer Use patterns. """

def __init__(self, client: Anthropic, model: str = "claude-sonnet-4-20250514"):
    self.client = client
    self.model = model
    self.max_steps = 50  # Prevent runaway loops
    self.action_delay = 0.5  # Seconds between actions

def capture_screenshot(self) -> str:
    """Capture screen and return base64 encoded image."""
    screenshot = pyautogui.screenshot()
    # Resize for token efficiency (1280x800 is good balance)
    screenshot = screenshot.resize((1280, 800), Image.LANCZOS)

    import io
    buffer = io.BytesIO()
    screenshot.save(buffer, format="PNG")
    return base64.b64encode(buffer.getvalue()).decode()

def execute_action(self, action: dict) -> dict:
    """Execute mouse/keyboard action on the computer."""
    action_type = action.get("type")

    if action_type == "click":
        x, y = action["x"], action["y"]
        button = action.get("button", "left")
        pyautogui.click(x, y, button=button)
        return {"success": True, "action": f"clicked at ({x}, {y})"}

    elif action_type == "type":
        text = action["text"]
        pyautogui.typewrite(text, interval=0.02)
        return {"success": True, "action": f"typed {len(text)} chars"}

    elif action_type == "key":
        key = action["key"]
        pyautogui.press(key)
        return {"success": True, "action": f"pressed {key}"}

    elif action_type == "scroll":
        direction = action.get("direction", "down")
        amount = action.get("amount", 3)
        scroll = -amount if direction == "down" else amount
        pyautogui.scroll(scroll)
        return {"success": True, "action": f"scrolled {direction}"}

    elif action_type == "move":
        x, y = action["x"], action["y"]
        pyautogui.moveTo(x, y)
        return {"success": True, "action": f"moved to ({x}, {y})"}

    else:
        return {"success": False, "error": f"Unknown action: {action_type}"}

def run(self, task: str) -> dict:
    """
    Run perception-reasoning-action loop until task complete.

    The loop:
    1. Screenshot current state
    2. Send to vision model with task context
    3. Parse action from response
    4. Execute action
    5. Repeat until done or max steps
    """
    messages = []
    step_count = 0

    system_prompt = """You are a computer use agent. You can see the screen
    and control mouse/keyboard.

    Available actions (respond with JSON):
    - {"type": "click", "x": 100, "y": 200, "button": "left"}
    - {"type": "type", "text": "hello world"}
    - {"type": "key", "key": "enter"}
    - {"type": "scroll", "direction": "down", "amount": 3}
    - {"type": "done", "result": "task completed successfully"}

    Always respond with ONLY a JSON action object.
    Be precise with coordinates - click exactly where needed.
    If you see an error, try to recover.
    """

    while step_count < self.max_steps:
        step_count += 1

        # 1. PERCEPTION: Capture current screen
        screenshot_b64 = self.capture_screenshot()

        # 2. REASONING: Send to vision model
        user_content = [
            {"type": "text", "text": f"Task: {task}\n\nStep {step_count}. What action should I take?"},
            {"type": "image", "source": {
                "type": "base64",
                "media_type": "image/png",
                "data": screenshot_b64
            }}
        ]

        messages.append({"role": "user", "content": user_content})

        response = self.client.messages.create(
            model=self.model,
            max_tokens=1024,
            system=system_prompt,
            messages=messages
        )

        assistant_message = response.content[0].text
        messages.append({"role": "assistant", "content": assistant_message})

        # 3. Parse action from response
        import json
        try:
            action = json.loads(assistant_message)
        except json.JSONDecodeError:
            # Try to extract JSON from response
            import re
            match = re.search(r'\{[^}]+\}', assistant_message)
            if match:
                action = json.loads(match.group())
            else:
                continue

        # Check if done
        if action.get("type") == "done":
            return {
                "success": True,
                "result": action.get("result"),
                "steps": step_count
            }

        # 4. ACTION: Execute
        result = self.execute_action(action)

        # Small delay for UI to update
        time.sleep(self.action_delay)

    return {
        "success": False,
        "error": "Max steps reached",
        "steps": step_count
    }

Usage

agent = ComputerUseAgent(Anthropic()) result = agent.run("Open Chrome and search for 'weather today'")

Anti_patterns

• Running without step limits (infinite loops)
• No delay between actions (UI can't keep up)
• Screenshots at full resolution (token explosion)
• Ignoring action failures (no recovery)

Sandboxed Environment Pattern

Computer use agents MUST run in isolated, sandboxed environments. Never give agents direct access to your main system - the security risks are too high. Use Docker containers with virtual desktops.

Key isolation requirements:

1. NETWORK: Restrict to necessary endpoints only
2. FILESYSTEM: Read-only or scoped to temp directories
3. CREDENTIALS: No access to host credentials
4. SYSCALLS: Filter dangerous system calls
5. RESOURCES: Limit CPU, memory, time

The goal is "blast radius minimization" - if the agent goes wrong, damage is contained to the sandbox.

When to use: Deploying any computer use agent,Testing agent behavior safely,Running untrusted automation tasks

Dockerfile for sandboxed computer use environment

Based on Anthropic's reference implementation pattern

FROM ubuntu:22.04

Install desktop environment

RUN apt-get update && apt-get install -y
xvfb
x11vnc
fluxbox
xterm
firefox
python3
python3-pip
supervisor

Security: Create non-root user

RUN useradd -m -s /bin/bash agent &&
mkdir -p /home/agent/.vnc

Install Python dependencies

COPY requirements.txt /tmp/ RUN pip3 install -r /tmp/requirements.txt

Security: Drop capabilities

RUN apt-get install -y --no-install-recommends libcap2-bin &&
setcap -r /usr/bin/python3 || true

Copy agent code

COPY --chown=agent:agent . /app WORKDIR /app

Supervisor config for virtual display + VNC

COPY supervisord.conf /etc/supervisor/conf.d/

Expose VNC port only (not desktop directly)

EXPOSE 5900

Run as non-root

USER agent

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]

docker-compose.yml with security constraints

version: '3.8'

services: computer-use-agent: build: . ports: - "5900:5900" # VNC for observation - "8080:8080" # API for control

# Security constraints
security_opt:
  - no-new-privileges:true
  - seccomp:seccomp-profile.json

# Resource limits
deploy:
  resources:
    limits:
      cpus: '2'
      memory: 4G
    reservations:
      cpus: '0.5'
      memory: 1G

# Network isolation
networks:
  - agent-network

# No access to host filesystem
volumes:
  - agent-tmp:/tmp

# Read-only root filesystem
read_only: true
tmpfs:
  - /run
  - /var/run

# Environment
environment:
  - DISPLAY=:99
  - NO_PROXY=localhost

networks: agent-network: driver: bridge internal: true # No internet by default

volumes: agent-tmp:

Python wrapper with additional runtime sandboxing

import subprocess import os from dataclasses import dataclass from typing import Optional

@dataclass class SandboxConfig: """Configuration for agent sandbox.""" network_allowed: list[str] = None # Allowed domains max_runtime_seconds: int = 300 max_memory_mb: int = 2048 allow_downloads: bool = False allow_clipboard: bool = False

class SandboxedAgent: """ Run computer use agent in Docker sandbox. """

def __init__(self, config: SandboxConfig):
    self.config = config
    self.container_id: Optional[str] = None

def start(self):
    """Start sandboxed environment."""
    # Build network rules
    network_rules = ""
    if self.config.network_allowed:
        for domain in self.config.network_allowed:
            network_rules += f"--add-host={domain}:$(dig +short {domain}) "
    else:
        network_rules = "--network=none"

    cmd = f"""
    docker run -d \
        --name computer-use-sandbox-$$ \
        --security-opt no-new-privileges \
        --cap-drop ALL \
        --memory {self.config.max_memory_mb}m \
        --cpus 2 \
        --read-only \
        --tmpfs /tmp \
        {network_rules} \
        computer-use-agent:latest
    """

    result = subprocess.run(cmd, shell=True, capture_output=True)
    self.container_id = result.stdout.decode().strip()

    # Set up kill timer
    subprocess.Popen([
        "sh", "-c",
        f"sleep {self.config.max_runtime_seconds} && docker kill {self.container_id}"
    ])

    return self.container_id

def execute_task(self, task: str) -> dict:
    """Execute task in sandbox."""
    if not self.container_id:
        self.start()

    # Send task to agent via API
    import requests
    response = requests.post(
        f"http://localhost:8080/task",
        json={"task": task},
        timeout=self.config.max_runtime_seconds
    )

    return response.json()

def stop(self):
    """Stop and remove sandbox."""
    if self.container_id:
        subprocess.run(f"docker rm -f {self.container_id}", shell=True)
        self.container_id = None

Anti_patterns

• Running agents on host system directly
• Giving sandbox full network access
• Running as root in container
• No resource limits (denial of service)
• Persistent storage (data can leak between runs)

Anthropic Computer Use Implementation

Official implementation pattern using Claude's computer use capability. Claude 3.5 Sonnet was the first frontier model to offer computer use. Claude Opus 4.5 is now the "best model in the world for computer use."

Key capabilities:

• screenshot: Capture current screen state
• mouse: Click, move, drag operations
• keyboard: Type text, press keys
• bash: Run shell commands
• text_editor: View and edit files

Tool versions:

• computer_20251124 (Opus 4.5): Adds zoom action for detailed inspection
• computer_20250124 (All other models): Standard capabilities

Critical limitation: "Some UI elements (like dropdowns and scrollbars) might be tricky for Claude to manipulate" - Anthropic docs

When to use: Building production computer use agents,Need highest quality vision understanding,Full desktop control (not just browser)

from anthropic import Anthropic from anthropic.types.beta import ( BetaToolComputerUse20241022, BetaToolBash20241022, BetaToolTextEditor20241022, ) import subprocess import base64 from PIL import Image import io

class AnthropicComputerUse: """ Official Anthropic Computer Use implementation.

Requires:
- Docker container with virtual display
- VNC for viewing agent actions
- Proper tool implementations
"""

def __init__(self):
    self.client = Anthropic()
    self.model = "claude-sonnet-4-20250514"  # Best for computer use
    self.screen_size = (1280, 800)

def get_tools(self) -> list:
    """Define computer use tools."""
    return [
        BetaToolComputerUse20241022(
            type="computer_20241022",
            name="computer",
            display_width_px=self.screen_size[0],
            display_height_px=self.screen_size[1],
        ),
        BetaToolBash20241022(
            type="bash_20241022",
            name="bash",
        ),
        BetaToolTextEditor20241022(
            type="text_editor_20241022",
            name="str_replace_editor",
        ),
    ]

def execute_tool(self, name: str, input: dict) -> dict:
    """Execute a tool and return result."""

    if name == "computer":
        return self._handle_computer_action(input)
    elif name == "bash":
        return self._handle_bash(input)
    elif name == "str_replace_editor":
        return self._handle_editor(input)
    else:
        return {"error": f"Unknown tool: {name}"}

def _handle_computer_action(self, input: dict) -> dict:
    """Handle computer control actions."""
    action = input.get("action")

    if action == "screenshot":
        # Capture via xdotool/scrot
        subprocess.run(["scrot", "/tmp/screenshot.png"])

        with open("/tmp/screenshot.png", "rb") as f:
            img_data = f.read()

        # Resize for efficiency
        img = Image.open(io.BytesIO(img_data))
        img = img.resize(self.screen_size, Image.LANCZOS)

        buffer = io.BytesIO()
        img.save(buffer, format="PNG")

        return {
            "type": "image",
            "source": {
                "type": "base64",
                "media_type": "image/png",
                "data": base64.b64encode(buffer.getvalue()).decode()
            }
        }

    elif action == "mouse_move":
        x, y = input.get("coordinate", [0, 0])
        subprocess.run(["xdotool", "mousemove", str(x), str(y)])
        return {"success": True}

    elif action == "left_click":
        subprocess.run(["xdotool", "click", "1"])
        return {"success": True}

    elif action == "right_click":
        subprocess.run(["xdotool", "click", "3"])
        return {"success": True}

    elif action == "double_click":
        subprocess.run(["xdotool", "click", "--repeat", "2", "1"])
        return {"success": True}

    elif action == "type":
        text = input.get("text", "")
        # Use xdotool type with delay for reliability
        subprocess.run(["xdotool", "type", "--delay", "50", text])
        return {"success": True}

    elif action == "key":
        key = input.get("key", "")
        # Map common key names
        key_map = {
            "return": "Return",
            "enter": "Return",
            "tab": "Tab",
            "escape": "Escape",
            "backspace": "BackSpace",
        }
        xdotool_key = key_map.get(key.lower(), key)
        subprocess.run(["xdotool", "key", xdotool_key])
        return {"success": True}

    elif action == "scroll":
        direction = input.get("direction", "down")
        amount = input.get("amount", 3)
        button = "5" if direction == "down" else "4"
        for _ in range(amount):
            subprocess.run(["xdotool", "click", button])
        return {"success": True}

    return {"error": f"Unknown action: {action}"}

def _handle_bash(self, input: dict) -> dict:
    """Execute bash command."""
    command = input.get("command", "")

    # Security: Sanitize and limit commands
    dangerous_patterns = ["rm -rf", "mkfs", "dd if=", "> /dev/"]
    for pattern in dangerous_patterns:
        if pattern in command:
            return {"error": "Dangerous command blocked"}

    try:
        result = subprocess.run(
            command,
            shell=True,
            capture_output=True,
            text=True,
            timeout=30
        )
        return {
            "stdout": result.stdout[:10000],  # Limit output
            "stderr": result.stderr[:1000],
            "returncode": result.returncode
        }
    except subprocess.TimeoutExpired:
        return {"error": "Command timed out"}

def _handle_editor(self, input: dict) -> dict:
    """Handle text editor operations."""
    command = input.get("command")
    path = input.get("path")

    if command == "view":
        try:
            with open(path, "r") as f:
                content = f.read()
            return {"content": content[:50000]}  # Limit size
        except Exception as e:
            return {"error": str(e)}

    elif command == "str_replace":
        old_str = input.get("old_str")
        new_str = input.get("new_str")
        try:
            with open(path, "r") as f:
                content = f.read()
            if old_str not in content:
                return {"error": "old_str not found in file"}
            content = content.replace(old_str, new_str, 1)
            with open(path, "w") as f:
                f.write(content)
            return {"success": True}
        except Exception as e:
            return {"error": str(e)}

    return {"error": f"Unknown editor command: {command}"}

def run_task(self, task: str, max_steps: int = 50) -> dict:
    """Run computer use task with agentic loop."""
    messages = [{"role": "user", "content": task}]
    tools = self.get_tools()

    for step in range(max_steps):
        response = self.client.beta.messages.create(
            model=self.model,
            max_tokens=4096,
            tools=tools,
            messages=messages,
            betas=["computer-use-2024-10-22"]
        )

        # Check for completion
        if response.stop_reason == "end_turn":
            return {
                "success": True,
                "result": response.content[0].text if response.content else "",
                "steps": step + 1
            }

        # Handle tool use
        if response.stop_reason == "tool_use":
            messages.append({"role": "assistant", "content": response.content})

            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    result = self.execute_tool(block.name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result
                    })

            messages.append({"role": "user", "content": tool_results})

    return {"success": False, "error": "Max steps reached"}

Anti_patterns

• Not using betas=['computer-use-2024-10-22'] flag
• Full resolution screenshots (wasteful)
• No command sanitization for bash tool
• Unbounded execution time

Browser-Use Pattern (Playwright-based)

For browser-only automation, using structured DOM access is more efficient than pixel-based computer use. Playwright MCP allows LLMs to control browsers using accessibility snapshots rather than screenshots.

Advantages over vision-based:

• Faster: No image processing required
• Cheaper: Text tokens vs image tokens
• More precise: Direct element targeting
• More reliable: No coordinate drift

When to use vision vs structured:

• Vision: Desktop apps, complex UIs, visual verification
• Structured: Web automation, form filling, data extraction

When to use: Browser-only automation tasks,Form filling and web interactions,When speed and cost matter more than visual understanding

from playwright.async_api import async_playwright from dataclasses import dataclass from typing import Optional import asyncio

@dataclass class BrowserAction: """Structured browser action.""" action: str # click, type, navigate, scroll, extract selector: Optional[str] = None text: Optional[str] = None url: Optional[str] = None

class BrowserUseAgent: """ Browser automation using Playwright with structured commands. More efficient than pixel-based for web tasks. """

def __init__(self):
    self.browser = None
    self.page = None

async def start(self, headless: bool = True):
    """Start browser session."""
    self.playwright = await async_playwright().start()
    self.browser = await self.playwright.chromium.launch(headless=headless)
    self.page = await self.browser.new_page()

async def get_page_snapshot(self) -> dict:
    """
    Get structured snapshot of page for LLM.
    Uses accessibility tree for efficiency.
    """
    # Get accessibility tree
    snapshot = await self.page.accessibility.snapshot()

    # Get simplified DOM info
    elements = await self.page.evaluate('''() => {
        const interactable = [];
        const selector = 'a, button, input, select, textarea, [role="button"]';
        document.querySelectorAll(selector).forEach((el, i) => {
            const rect = el.getBoundingClientRect();
            if (rect.width > 0 && rect.height > 0) {
                interactable.push({
                    index: i,
                    tag: el.tagName.toLowerCase(),
                    text: el.textContent?.trim().slice(0, 100),
                    type: el.type,
                    placeholder: el.placeholder,
                    name: el.name,
                    id: el.id,
                    class: el.className
                });
            }
        });
        return interactable;
    }''')

    return {
        "url": self.page.url,
        "title": await self.page.title(),
        "accessibility_tree": snapshot,
        "interactable_elements": elements[:50]  # Limit for token efficiency
    }

async def execute_action(self, action: BrowserAction) -> dict:
    """Execute structured browser action."""

    try:
        if action.action == "navigate":
            await self.page.goto(action.url, wait_until="domcontentloaded")
            return {"success": True, "url": self.page.url}

        elif action.action == "click":
            await self.page.click(action.selector, timeout=5000)
            await self.page.wait_for_load_state("networkidle", timeout=5000)
            return {"success": True}

        elif action.action == "type":
            await self.page.fill(action.selector, action.text)
            return {"success": True}

        elif action.action == "scroll":
            direction = action.text or "down"
            distance = 500 if direction == "down" else -500
            await self.page.evaluate(f"window.scrollBy(0, {distance})")
            return {"success": True}

        elif action.action == "extract":
            # Extract text content
            if action.selector:
                text = await self.page.text_content(action.selector)
            else:
                text = await self.page.text_content("body")
            return {"success": True, "text": text[:5000]}

        elif action.action == "screenshot":
            # Fall back to vision when needed
            screenshot = await self.page.screenshot(type="png")
            import base64
            return {
                "success": True,
                "image": base64.b64encode(screenshot).decode()
            }

    except Exception as e:
        return {"success": False, "error": str(e)}

    return {"success": False, "error": f"Unknown action: {action.action}"}

async def run_with_llm(self, task: str, llm_client, max_steps: int = 20):
    """
    Run browser task with LLM decision making.
    Uses structured DOM instead of screenshots.
    """

    system_prompt = """You are a browser automation agent. You receive
    page snapshots with interactable elements and decide actions.

    Respond with JSON action:
    - {"action": "navigate", "url": "https://..."}
    - {"action": "click", "selector": "button.submit"}
    - {"action": "type", "selector": "input[name='email']", "text": "..."}
    - {"action": "scroll", "text": "down"}
    - {"action": "extract", "selector": ".results"}
    - {"action": "done", "result": "task completed"}

    Use CSS selectors based on the element info provided.
    Prefer id > name > class > text content for selectors.
    """

    messages = []

    for step in range(max_steps):
        # Get current page state
        snapshot = await self.get_page_snapshot()

        user_message = f"""Task: {task}

        Current page:
        URL: {snapshot['url']}
        Title: {snapshot['title']}

        Interactable elements:
        {snapshot['interactable_elements']}

        What action should I take?"""

        messages.append({"role": "user", "content": user_message})

        # Get LLM decision
        response = llm_client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1024,
            system=system_prompt,
            messages=messages
        )

        assistant_text = response.content[0].text
        messages.append({"role": "assistant", "content": assistant_text})

        # Parse and execute
        import json
        action_dict = json.loads(assistant_text)

        if action_dict.get("action") == "done":
            return {"success": True, "result": action_dict.get("result")}

        action = BrowserAction(**action_dict)
        result = await self.execute_action(action)

        if not result.get("success"):
            messages.append({
                "role": "user",
                "content": f"Action failed: {result.get('error')}"
            })

        await asyncio.sleep(0.5)  # Rate limit

    return {"success": False, "error": "Max steps reached"}

async def close(self):
    """Clean up browser."""
    if self.browser:
        await self.browser.close()
    if hasattr(self, 'playwright'):
        await self.playwright.stop()

Usage

async def main(): agent = BrowserUseAgent() await agent.start(headless=False)

from anthropic import Anthropic
result = await agent.run_with_llm(
    "Go to weather.com and find the weather for New York",
    Anthropic()
)

print(result)
await agent.close()

asyncio.run(main())

Anti_patterns

• Using screenshots when DOM access works
• Not waiting for page loads
• Hardcoded selectors that break
• No error recovery for stale elements

User Confirmation Pattern

For sensitive actions, agents should pause and ask for human confirmation. "ChatGPT agent also pauses and asks for confirmation prior to taking sensitive steps such as completing a purchase."

Sensitivity levels:

1. LOW: Navigation, reading (auto-approve)
2. MEDIUM: Form filling, clicking (log, maybe confirm)
3. HIGH: Purchases, authentication, file operations (always confirm)
4. CRITICAL: Credential entry, financial transactions (confirm + review)

When to use: Actions with real-world consequences,Financial transactions,Authentication flows,File modifications

from enum import Enum from dataclasses import dataclass from typing import Callable, Optional import asyncio

class ActionSeverity(Enum): LOW = "low" # Auto-approve MEDIUM = "medium" # Log, optional confirm HIGH = "high" # Always confirm CRITICAL = "critical" # Confirm + review details

@dataclass class SensitiveAction: """Action that may need user confirmation.""" action_type: str description: str severity: ActionSeverity details: dict

class ConfirmationGate: """ Gate sensitive actions through user confirmation. """

# Action type -> severity mapping
ACTION_SEVERITY = {
    # LOW - auto-approve
    "navigate": ActionSeverity.LOW,
    "scroll": ActionSeverity.LOW,
    "read": ActionSeverity.LOW,
    "screenshot": ActionSeverity.LOW,

    # MEDIUM - log and maybe confirm
    "click": ActionSeverity.MEDIUM,
    "type": ActionSeverity.MEDIUM,
    "search": ActionSeverity.MEDIUM,

    # HIGH - always confirm
    "download": ActionSeverity.HIGH,
    "submit_form": ActionSeverity.HIGH,
    "login": ActionSeverity.HIGH,
    "file_write": ActionSeverity.HIGH,

    # CRITICAL - confirm with full review
    "purchase": ActionSeverity.CRITICAL,
    "enter_password": ActionSeverity.CRITICAL,
    "enter_credit_card": ActionSeverity.CRITICAL,
    "send_money": ActionSeverity.CRITICAL,
    "delete": ActionSeverity.CRITICAL,
}

def __init__(
    self,
    confirm_callback: Callable[[SensitiveAction], bool] = None,
    auto_confirm_low: bool = True,
    auto_confirm_medium: bool = False
):
    self.confirm_callback = confirm_callback or self._default_confirm
    self.auto_confirm_low = auto_confirm_low
    self.auto_confirm_medium = auto_confirm_medium
    self.action_log = []

def _default_confirm(self, action: SensitiveAction) -> bool:
    """Default confirmation via CLI prompt."""
    print(f"\n{'='*60}")
    print(f"ACTION CONFIRMATION REQUIRED")
    print(f"{'='*60}")
    print(f"Type: {action.action_type}")
    print(f"Severity: {action.severity.value.upper()}")
    print(f"Description: {action.description}")
    print(f"Details: {action.details}")
    print(f"{'='*60}")

    while True:
        response = input("Allow this action? [y/n]: ").lower().strip()
        if response in ['y', 'yes']:
            return True
        elif response in ['n', 'no']:
            return False

def classify_action(self, action_type: str, context: dict) -> ActionSeverity:
    """Classify action severity, considering context."""
    base_severity = self.ACTION_SEVERITY.get(action_type, ActionSeverity.MEDIUM)

    # Escalate based on context
    if context.get("involves_credentials"):
        return ActionSeverity.CRITICAL
    if context.get("involves_money"):
        return ActionSeverity.CRITICAL
    if context.get("irreversible"):
        return max(base_severity, ActionSeverity.HIGH, key=lambda x: x.value)

    return base_severity

def check_action(
    self,
    action_type: str,
    description: str,
    details: dict = None
) -> tuple[bool, str]:
    """
    Check if action should proceed.
    Returns (approved, reason).
    """
    details = details or {}
    severity = self.classify_action(action_type, details)

    action = SensitiveAction(
        action_type=action_type,
        description=description,
        severity=severity,
        details=details
    )

    # Log all actions
    self.action_log.append({
        "action": action,
        "timestamp": __import__('datetime').datetime.now().isoformat()
    })

    # Auto-approve low severity
    if severity == ActionSeverity.LOW and self.auto_confirm_low:
        return True, "auto-approved (low severity)"

    # Maybe auto-approve medium
    if severity == ActionSeverity.MEDIUM and self.auto_confirm_medium:
        return True, "auto-approved (medium severity)"

    # Request confirmation
    approved = self.confirm_callback(action)

    if approved:
        return True, "user approved"
    else:
        return False, "user rejected"

class ConfirmedComputerUseAgent: """ Computer use agent with confirmation gates. """

def __init__(self, base_agent, confirmation_gate: ConfirmationGate):
    self.agent = base_agent
    self.gate = confirmation_gate

def execute_action(self, action: dict) -> dict:
    """Execute action with confirmation check."""
    action_type = action.get("type", "unknown")

    # Build description
    if action_type == "click":
        desc = f"Click at ({action.get('x')}, {action.get('y')})"
    elif action_type == "type":
        text = action.get('text', '')
        # Mask if looks like password
        if self._looks_sensitive(text):
            desc = f"Type sensitive text ({len(text)} chars)"
        else:
            desc = f"Type: {text[:50]}..."
    else:
        desc = f"Execute: {action_type}"

    # Context for severity classification
    context = {
        "involves_credentials": self._looks_sensitive(action.get("text", "")),
        "involves_money": self._mentions_money(action),
    }

    # Check with gate
    approved, reason = self.gate.check_action(
        action_type, desc, context
    )

    if not approved:
        return {
            "success": False,
            "error": f"Action blocked: {reason}",
            "action": action_type
        }

    # Execute if approved
    return self.agent.execute_action(action)

def _looks_sensitive(self, text: str) -> bool:
    """Check if text looks like sensitive data."""
    if not text:
        return False
    # Common patterns
    patterns = [
        r'\b\d{16}\b',  # Credit card
        r'\b\d{3,4}\b.*\b\d{3,4}\b',  # CVV-like
        r'password',
        r'secret',
        r'api.?key',
        r'token'
    ]
    import re
    return any(re.search(p, text.lower()) for p in patterns)

def _mentions_money(self, action: dict) -> bool:
    """Check if action involves money."""
    text = str(action)

…(truncated)