# Open Computer Use Automation

> AI-powered computer automation using MCP to control desktop apps, click elements, and interact with the UI on macOS, Linux, and Windows

- Skill: `aradotso/open-computer-use-automation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aradotso/open-computer-use-automation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aradotso/open-computer-use-automation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: aradotso (https://skillmd.com/u/aradotso)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aradotso/open-computer-use-automation

---


# open-computer-use-automation

> Skill by [ara.so](https://ara.so) — Codex Skills collection.

## Overview

`open-computer-use` is an open-source Computer Use service wrapped as MCP (Model Context Protocol). It enables AI agents to interact with desktop applications through accessibility APIs on macOS, Linux, and Windows. Inspired by OpenAI's Codex Computer Use, it provides non-intrusive automation using native accessibility features.

The project exposes tools to:
- List running applications
- Get application UI state and elements
- Click elements, buttons, and UI components
- Type text and press keys
- Take screenshots
- Manage app focus and windows

## Installation

### Global Installation

```bash
npm i -g open-computer-use
```

### macOS Permissions

On first run, grant **Accessibility** and **Screen Recording** permissions:

```bash
open-computer-use
```

Follow the system prompts to enable permissions in System Settings.

### MCP Integration

Install into your AI agent:

```bash
# Codex
open-computer-use install-codex-mcp

# Claude Code
open-computer-use install-claude-mcp

# Gemini CLI (project scope)
open-computer-use install-gemini-mcp

# Gemini CLI (user scope)
open-computer-use install-gemini-mcp --scope user

# opencode
open-computer-use install-opencode-mcp
```

### Manual MCP Configuration

Add to your MCP client config (e.g., `~/.codex/config.toml`, `~/.claude.json`):

```json
{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"]
    }
  }
}
```

### Skill Installation

```bash
# Install for Codex
npx skills add iFurySt/open-codex-computer-use -g -a codex --skill open-computer-use -y

# Install for Claude Code
npx skills add iFurySt/open-codex-computer-use -g -a claude-code --skill open-computer-use -y

# Update existing skill
npx skills update open-computer-use -g -y

# List installed skills
npx skills ls -g -a codex | rg 'open-computer-use'
```

## Core Commands

### CLI Usage

```bash
# Check permissions and system readiness
open-computer-use doctor

# Call a single tool (returns MCP JSON)
open-computer-use call list_apps

# Call with arguments
open-computer-use call get_app_state --args '{"app":"TextEdit"}'

# Run a sequence of operations (maintains element_index state)
open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

# Run sequence from file with custom sleep between operations
open-computer-use call --calls-file sequence.json --sleep 0.5

# Show help
open-computer-use -h
```

### Codex Plugin Installation

For Codex App (macOS):

```bash
open-computer-use install-codex-plugin
```

## MCP Tools Reference

### list_apps

List all running applications.

**Arguments:** None

**Returns:** Array of app names

```json
{
  "apps": ["Safari", "TextEdit", "Terminal"]
}
```

### get_app_state

Get the UI element tree for an application.

**Arguments:**
- `app` (string, required): Application name
- `include_screenshot` (boolean, optional): Include base64 screenshot

**Returns:** UI hierarchy with element metadata

```json
{
  "app": "TextEdit",
  "elements": [
    {
      "element_index": 0,
      "role": "AXWindow",
      "title": "Untitled",
      "children": [...]
    }
  ],
  "screenshot": "data:image/png;base64,..."
}
```

### click_element

Click a UI element by index.

**Arguments:**
- `app` (string, required): Application name
- `element_index` (number, required): Element index from `get_app_state`
- `click_type` (string, optional): "single" (default) or "double"

**Returns:** Success confirmation

```json
{
  "success": true,
  "element_index": 5
}
```

### type_text

Type text into the focused field.

**Arguments:**
- `app` (string, required): Application name
- `text` (string, required): Text to type

**Returns:** Success confirmation

### press_key

Press a keyboard key or key combination.

**Arguments:**
- `app` (string, required): Application name
- `key` (string, required): Key name (e.g., "Return", "Tab", "Command+S")

**Supported keys:** Return, Tab, Space, Delete, Escape, Arrow keys, Command+[key], etc.

### take_screenshot

Capture the current screen.

**Arguments:**
- `app` (string, optional): Application name to focus

**Returns:** Base64-encoded PNG

### activate_app

Bring an application to the foreground.

**Arguments:**
- `app` (string, required): Application name

## Usage Patterns

### Basic App Interaction

```typescript
// From an MCP client or AI agent

// 1. List running apps
const apps = await call_tool("list_apps");

// 2. Get app UI state
const state = await call_tool("get_app_state", {
  app: "TextEdit",
  include_screenshot: true
});

// 3. Find and click a button
// (element_index 3 might be a "Save" button from state.elements)
await call_tool("click_element", {
  app: "TextEdit",
  element_index: 3
});

// 4. Type text
await call_tool("type_text", {
  app: "TextEdit",
  text: "Hello, world!"
});

// 5. Save with keyboard shortcut
await call_tool("press_key", {
  app: "TextEdit",
  key: "Command+S"
});
```

### Sequence Execution

Create a JSON sequence file `automation.json`:

```json
[
  {
    "tool": "activate_app",
    "args": {"app": "TextEdit"}
  },
  {
    "tool": "get_app_state",
    "args": {"app": "TextEdit"}
  },
  {
    "tool": "type_text",
    "args": {
      "app": "TextEdit",
      "text": "This is automated text."
    }
  },
  {
    "tool": "press_key",
    "args": {
      "app": "TextEdit",
      "key": "Return"
    }
  },
  {
    "tool": "take_screenshot",
    "args": {"app": "TextEdit"}
  }
]
```

Run it:

```bash
open-computer-use call --calls-file automation.json --sleep 1
```

### Finding Elements

When you call `get_app_state`, inspect the returned elements to find the one you need:

```json
{
  "elements": [
    {
      "element_index": 0,
      "role": "AXWindow",
      "title": "Document",
      "children": [
        {
          "element_index": 1,
          "role": "AXButton",
          "title": "Close",
          "enabled": true
        },
        {
          "element_index": 2,
          "role": "AXTextArea",
          "value": "Current text content"
        }
      ]
    }
  ]
}
```

Use `element_index` from this tree when calling `click_element`.

### Cross-Platform Considerations

- **macOS**: Requires Accessibility and Screen Recording permissions
- **Linux**: Uses AT-SPI (accessibility toolkit)
- **Windows**: Uses UI Automation API

All platforms use the same MCP interface, but element roles and properties may differ slightly.

## Configuration

### Environment Variables

No environment variables required for basic operation. Permissions are handled at the OS level.

### Custom Sleep Between Operations

Default sleep is 1 second. Customize with `--sleep`:

```bash
open-computer-use call --calls-file seq.json --sleep 0.5
```

### MCP Server Args

When configuring MCP manually, you can pass custom args:

```json
{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"],
      "env": {}
    }
  }
}
```

## Troubleshooting

### Permission Denied (macOS)

**Symptom:** Cannot access UI elements or take screenshots.

**Solution:**
1. Run `open-computer-use doctor` to check permissions
2. Grant **Accessibility** permission in System Settings → Privacy & Security
3. Grant **Screen Recording** permission
4. Restart the terminal or agent

### App Not Found

**Symptom:** `list_apps` doesn't show the target application.

**Solution:**
- Ensure the app is running
- Check exact app name (case-sensitive): `open-computer-use call list_apps`
- Some apps use different process names (e.g., "Google Chrome" vs "Chrome")

### Element Index Invalid

**Symptom:** `click_element` fails with invalid index.

**Solution:**
- Refresh app state with `get_app_state` before clicking
- Element indices can change when UI updates
- Use sequences to maintain state across operations

### MCP Server Not Starting

**Symptom:** Agent can't connect to `open-computer-use`.

**Solution:**
```bash
# Verify installation
which open-computer-use

# Test manual MCP mode
open-computer-use mcp

# Reinstall globally
npm i -g open-computer-use

# Check agent config file syntax
cat ~/.codex/config.toml  # or relevant config
```

### Linux: AT-SPI Not Available

**Symptom:** Tools fail on Linux with accessibility errors.

**Solution:**
```bash
# Install AT-SPI dependencies (Ubuntu/Debian)
sudo apt-get install at-spi2-core

# Enable accessibility
gsettings set org.gnome.desktop.interface toolkit-accessibility true
```

## Advanced Usage

### Programmatic Integration (TypeScript)

If building a custom MCP client or agent:

```typescript
import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

async function automateApp(appName: string) {
  // Get app state
  const { stdout } = await execAsync(
    `open-computer-use call get_app_state --args '{"app":"${appName}"}'`
  );
  
  const state = JSON.parse(stdout);
  
  // Find button with specific title
  const button = state.elements
    .flatMap(e => e.children || [])
    .find(e => e.role === 'AXButton' && e.title === 'Submit');
  
  if (button) {
    // Click it
    await execAsync(
      `open-computer-use call click_element --args '{"app":"${appName}","element_index":${button.element_index}}'`
    );
  }
}

await automateApp('Safari');
```

### Custom Skill Integration

When writing agent prompts or skills that use `open-computer-use`:

```markdown
To interact with desktop apps:
1. Always list apps first to verify the target is running
2. Get app state to find element indices
3. Use element_index from state when clicking
4. Add small delays between operations (1s default)
5. Take screenshots to verify results

Example workflow:
- list_apps → verify "Safari" is running
- get_app_state(app="Safari") → find address bar element_index
- click_element(element_index=X) → focus address bar
- type_text(text="https://example.com") → enter URL
- press_key(key="Return") → navigate
```

## Related Tools

- **Cursor Motion**: Separate macOS app for smooth cursor animations (download from releases page)
- **open-browser-use**: Companion project for browser-specific automation

## Resources

- [GitHub Repository](https://github.com/iFurySt/open-codex-computer-use)
- [Releases](https://github.com/iFurySt/open-codex-computer-use/releases)
- [Demo Videos](https://youtu.be/2s6aVpGiwaQ)