Open Computer Use Automation

open-computer-use-automation

Skill by ara.so — Codex Skills collection.

Overview

open-computer-use is an open-source Computer Use service wrapped as MCP (Model Context Protocol). It enables AI agents to interact with desktop applications through accessibility APIs on macOS, Linux, and Windows. Inspired by OpenAI's Codex Computer Use, it provides non-intrusive automation using native accessibility features.

The project exposes tools to:

• List running applications
• Get application UI state and elements
• Click elements, buttons, and UI components
• Type text and press keys
• Take screenshots
• Manage app focus and windows

Installation

Global Installation

npm i -g open-computer-use

macOS Permissions

On first run, grant Accessibility and Screen Recording permissions:

open-computer-use

Follow the system prompts to enable permissions in System Settings.

MCP Integration

Install into your AI agent:

# Codex
open-computer-use install-codex-mcp

# Claude Code
open-computer-use install-claude-mcp

# Gemini CLI (project scope)
open-computer-use install-gemini-mcp

# Gemini CLI (user scope)
open-computer-use install-gemini-mcp --scope user

# opencode
open-computer-use install-opencode-mcp

Manual MCP Configuration

Add to your MCP client config (e.g., ~/.codex/config.toml, ~/.claude.json):

{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"]
    }
  }
}

Skill Installation

# Install for Codex
npx skills add iFurySt/open-codex-computer-use -g -a codex --skill open-computer-use -y

# Install for Claude Code
npx skills add iFurySt/open-codex-computer-use -g -a claude-code --skill open-computer-use -y

# Update existing skill
npx skills update open-computer-use -g -y

# List installed skills
npx skills ls -g -a codex | rg 'open-computer-use'

Core Commands

CLI Usage

# Check permissions and system readiness
open-computer-use doctor

# Call a single tool (returns MCP JSON)
open-computer-use call list_apps

# Call with arguments
open-computer-use call get_app_state --args '{"app":"TextEdit"}'

# Run a sequence of operations (maintains element_index state)
open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

# Run sequence from file with custom sleep between operations
open-computer-use call --calls-file sequence.json --sleep 0.5

# Show help
open-computer-use -h

Codex Plugin Installation

For Codex App (macOS):

open-computer-use install-codex-plugin

MCP Tools Reference

list_apps

List all running applications.

Arguments: None

Returns: Array of app names

{
  "apps": ["Safari", "TextEdit", "Terminal"]
}

get_app_state

Get the UI element tree for an application.

Arguments:

• app (string, required): Application name
• include_screenshot (boolean, optional): Include base64 screenshot

Returns: UI hierarchy with element metadata

{
  "app": "TextEdit",
  "elements": [
    {
      "element_index": 0,
      "role": "AXWindow",
      "title": "Untitled",
      "children": [...]
    }
  ],
  "screenshot": "data:image/png;base64,..."
}

click_element

Click a UI element by index.

Arguments:

• app (string, required): Application name
• element_index (number, required): Element index from get_app_state
• click_type (string, optional): "single" (default) or "double"

Returns: Success confirmation

{
  "success": true,
  "element_index": 5
}

type_text

Type text into the focused field.

Arguments:

• app (string, required): Application name
• text (string, required): Text to type

Returns: Success confirmation

press_key

Press a keyboard key or key combination.

Arguments:

• app (string, required): Application name
• key (string, required): Key name (e.g., "Return", "Tab", "Command+S")

Supported keys: Return, Tab, Space, Delete, Escape, Arrow keys, Command+[key], etc.

take_screenshot

Capture the current screen.

Arguments:

• app (string, optional): Application name to focus

Returns: Base64-encoded PNG

activate_app

Bring an application to the foreground.

Arguments:

• app (string, required): Application name

Usage Patterns

Basic App Interaction

// From an MCP client or AI agent

// 1. List running apps
const apps = await call_tool("list_apps");

// 2. Get app UI state
const state = await call_tool("get_app_state", {
  app: "TextEdit",
  include_screenshot: true
});

// 3. Find and click a button
// (element_index 3 might be a "Save" button from state.elements)
await call_tool("click_element", {
  app: "TextEdit",
  element_index: 3
});

// 4. Type text
await call_tool("type_text", {
  app: "TextEdit",
  text: "Hello, world!"
});

// 5. Save with keyboard shortcut
await call_tool("press_key", {
  app: "TextEdit",
  key: "Command+S"
});

Sequence Execution

Create a JSON sequence file automation.json:

[
  {
    "tool": "activate_app",
    "args": {"app": "TextEdit"}
  },
  {
    "tool": "get_app_state",
    "args": {"app": "TextEdit"}
  },
  {
    "tool": "type_text",
    "args": {
      "app": "TextEdit",
      "text": "This is automated text."
    }
  },
  {
    "tool": "press_key",
    "args": {
      "app": "TextEdit",
      "key": "Return"
    }
  },
  {
    "tool": "take_screenshot",
    "args": {"app": "TextEdit"}
  }
]

Run it:

open-computer-use call --calls-file automation.json --sleep 1

Finding Elements

When you call get_app_state, inspect the returned elements to find the one you need:

{
  "elements": [
    {
      "element_index": 0,
      "role": "AXWindow",
      "title": "Document",
      "children": [
        {
          "element_index": 1,
          "role": "AXButton",
          "title": "Close",
          "enabled": true
        },
        {
          "element_index": 2,
          "role": "AXTextArea",
          "value": "Current text content"
        }
      ]
    }
  ]
}

Use element_index from this tree when calling click_element.

Cross-Platform Considerations

• macOS: Requires Accessibility and Screen Recording permissions
• Linux: Uses AT-SPI (accessibility toolkit)
• Windows: Uses UI Automation API

All platforms use the same MCP interface, but element roles and properties may differ slightly.

Configuration

Environment Variables

No environment variables required for basic operation. Permissions are handled at the OS level.

Custom Sleep Between Operations

Default sleep is 1 second. Customize with --sleep:

open-computer-use call --calls-file seq.json --sleep 0.5

MCP Server Args

When configuring MCP manually, you can pass custom args:

{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"],
      "env": {}
    }
  }
}

Troubleshooting

Permission Denied (macOS)

Symptom: Cannot access UI elements or take screenshots.

Solution:

1. Run open-computer-use doctor to check permissions
2. Grant Accessibility permission in System Settings → Privacy & Security
3. Grant Screen Recording permission
4. Restart the terminal or agent

App Not Found

Symptom: list_apps doesn't show the target application.

Solution:

• Ensure the app is running
• Check exact app name (case-sensitive): open-computer-use call list_apps
• Some apps use different process names (e.g., "Google Chrome" vs "Chrome")

Element Index Invalid

Symptom: click_element fails with invalid index.

Solution:

• Refresh app state with get_app_state before clicking
• Element indices can change when UI updates
• Use sequences to maintain state across operations

MCP Server Not Starting

Symptom: Agent can't connect to open-computer-use.

Solution:

# Verify installation
which open-computer-use

# Test manual MCP mode
open-computer-use mcp

# Reinstall globally
npm i -g open-computer-use

# Check agent config file syntax
cat ~/.codex/config.toml  # or relevant config

Linux: AT-SPI Not Available

Symptom: Tools fail on Linux with accessibility errors.

Solution:

# Install AT-SPI dependencies (Ubuntu/Debian)
sudo apt-get install at-spi2-core

# Enable accessibility
gsettings set org.gnome.desktop.interface toolkit-accessibility true

Advanced Usage

Programmatic Integration (TypeScript)

If building a custom MCP client or agent:

import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

async function automateApp(appName: string) {
  // Get app state
  const { stdout } = await execAsync(
    `open-computer-use call get_app_state --args '{"app":"${appName}"}'`
  );
  
  const state = JSON.parse(stdout);
  
  // Find button with specific title
  const button = state.elements
    .flatMap(e => e.children || [])
    .find(e => e.role === 'AXButton' && e.title === 'Submit');
  
  if (button) {
    // Click it
    await execAsync(
      `open-computer-use call click_element --args '{"app":"${appName}","element_index":${button.element_index}}'`
    );
  }
}

await automateApp('Safari');

Custom Skill Integration

When writing agent prompts or skills that use open-computer-use:

To interact with desktop apps:
1. Always list apps first to verify the target is running
2. Get app state to find element indices
3. Use element_index from state when clicking
4. Add small delays between operations (1s default)
5. Take screenshots to verify results

Example workflow:
- list_apps → verify "Safari" is running
- get_app_state(app="Safari") → find address bar element_index
- click_element(element_index=X) → focus address bar
- type_text(text="https://example.com") → enter URL
- press_key(key="Return") → navigate

Related Tools

• Cursor Motion: Separate macOS app for smooth cursor animations (download from releases page)
• open-browser-use: Companion project for browser-specific automation

Resources

• GitHub Repository
• Releases
• Demo Videos