OpenInterpreter — Desktop GUI Automation

OpenInterpreter — Desktop GUI Automation

Desktop control for Claude Code via OpenInterpreter (62k stars, AGPL-3.0). Mouse, keyboard, screenshot, and OCR primitives backed by pyautogui + pytesseract.

Mode Selection

Use Library mode by default. Use OS subprocess to delegate self-contained GUI tasks. Use Local agent when offline or to avoid API costs.

Installation

Run once:

~/.claude/skills/open-interpreter/scripts/oi_install.sh

Installs open-interpreter[os] via uv, verifies pyautogui and tesseract, checks macOS permissions.

macOS permissions (one-time, manual):

• System Settings > Privacy & Security > Accessibility > add terminal app (Ghostty/Terminal/iTerm2)
• System Settings > Privacy & Security > Screen Recording > add terminal app

Verify permissions:

python3 ~/.claude/skills/open-interpreter/scripts/oi_permission_check.py

Library Mode: The Screenshot Loop

The core pattern for GUI automation:

1. Take screenshot   →  oi_screenshot.py
2. Read PNG          →  Claude Read tool (native vision)
3. Decide action     →  Claude reasoning
4. Execute action    →  oi_click.py / oi_type.py
5. Verify            →  Take another screenshot
6. Loop until done

Scripts

oi_screenshot.py — Capture screen, return file path with Retina metadata

python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py --region 0,0,800,600
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py --active-window

Output (3 lines):

/tmp/oi_screenshot_1708789200.png
SCALE_FACTOR=2
SCREEN_SIZE=1512x982

oi_click.py — Mouse click by coordinates or OCR text

python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 900 --y 600 --image-coords
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300 --double
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300 --right

• --image-coords: auto-divides by Retina scale factor (use when coordinates come from screenshot image pixels)
• --text: OCR-based — screenshots, finds text via pytesseract, clicks center of match

oi_type.py — Keyboard input

python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --text "hello world"
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --key enter
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --hotkey command space
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --text "search" --method typewrite

• Default text input: clipboard-paste (Cmd+V) for speed and Unicode safety
• --method typewrite: character-by-character (use when clipboard is needed for other purposes)
• --hotkey: AppleScript on macOS for reliable modifier key handling

oi_find_text.py — OCR screen reading

python3 ~/.claude/skills/open-interpreter/scripts/oi_find_text.py --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_find_text.py --text "Price" --screenshot /tmp/ss.png

Returns JSON array: [{"text": "Submit", "x": 450, "y": 300, "w": 80, "h": 24, "confidence": 95}]

oi_computer.py — Unified dispatch for all actions

python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py screenshot
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py click --x 450 --y 300
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py type --text "hello"
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py find --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py scroll --clicks 3
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py mouse-position
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py screen-size

Retina Display Handling

macOS Retina displays render at 2x (or 3x) scaling. Screenshot image pixels differ from screen coordinates:

When estimating click targets from a screenshot image, use --image-coords on oi_click.py to auto-divide by the scale factor. The oi_screenshot.py output includes SCALE_FACTOR metadata.

OS Mode: Delegate Full Tasks

For self-contained GUI tasks, delegate to OI's full agent loop:

python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py "Open Calculator and compute 2+2"
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py --provider anthropic "Change the desktop wallpaper"

OI runs its own screenshot → analyze → act loop using the Claude API. Requires ANTHROPIC_API_KEY.

Local Mode: Offline Computer Use

Run OI with a local vision model via Ollama:

python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py --local "What apps are open?"

Prerequisites:

1. Ollama running: ollama serve
2. Vision model pulled: ollama pull llama3.2-vision

Limitation: Local models use OI's classic code-execution mode, not the screenshot-driven OS Mode (which requires Claude 3.5 Sonnet). Local mode generates and executes code to accomplish GUI tasks rather than using pixel-level screenshot analysis.

Common Recipes

Open an App via Spotlight

python3 scripts/oi_type.py --hotkey command space
sleep 0.5
python3 scripts/oi_type.py --text "Calculator"
sleep 0.3
python3 scripts/oi_type.py --key enter

Read Text from Screen

python3 scripts/oi_screenshot.py > /tmp/ss_meta.txt
python3 scripts/oi_find_text.py --text "Total" --screenshot "$(head -1 /tmp/ss_meta.txt)"

Click a Button by Label

python3 scripts/oi_click.py --text "Save"

Fill a Form Field

python3 scripts/oi_click.py --text "Email"
python3 scripts/oi_type.py --text "user@example.com"
python3 scripts/oi_type.py --key tab
python3 scripts/oi_type.py --text "password123"

Safety

1. Confirm before destructive actions — before clicking Send, Delete, Submit, or Confirm buttons, verify with the user
2. Screenshot before and after every action for verification
3. No unbounded autonomous loops — confirm with user between multi-step GUI workflows
4. pyautogui failsafe — moving mouse to any screen corner raises pyautogui.FailSafeException (enabled by default)
5. Action logging — every script logs actions to stderr: [oi] click at (450, 300) button=left

Troubleshooting

Reference Documentation