Skill: desktop-control
When to Use
Use this skill when the user asks to:
- Click somewhere on the screen
- Move the mouse to a position
- Type text into an application
- Press keyboard shortcuts or hotkeys
- Read what's on the current screen (accessibility tree)
- Get information about the frontmost window
- Automate desktop interactions
- Control the computer (mouse, keyboard, screen)
- Scroll up/down in an application
- Drag and drop elements
IMPORTANT: This skill requires Accessibility permissions for the terminal/IDE. On macOS, go to System Settings > Privacy & Security > Accessibility and enable the running application.
Bundled Scripts
| Script |
Type |
Description |
scripts/mouse.py |
Python |
Mouse movement, clicking, dragging, scrolling |
scripts/keyboard.py |
Python |
Text typing, key presses, hotkeys |
scripts/screen.py |
Python |
Screen info, capture, accessibility tree reading |
All scripts auto-install pyautogui if needed.
Mouse Control
Input Parameters
| Parameter |
Required |
Description |
Example |
action |
Yes |
move, click, doubleclick, rightclick, drag, scroll |
click |
x |
For most |
X coordinate (pixels from left) |
500 |
y |
For most |
Y coordinate (pixels from top) |
300 |
button |
No |
Mouse button: left (default), right, middle |
left |
to_x |
For drag |
Destination X coordinate |
700 |
to_y |
For drag |
Destination Y coordinate |
400 |
amount |
For scroll |
Scroll amount (positive=up, negative=down) |
-3 |
Script Usage
# Move mouse
python3 skills/desktop-control/scripts/mouse.py move --x 500 --y 300
# Click at position
python3 skills/desktop-control/scripts/mouse.py click --x 500 --y 300
# Double click
python3 skills/desktop-control/scripts/mouse.py doubleclick --x 500 --y 300
# Right click
python3 skills/desktop-control/scripts/mouse.py rightclick --x 500 --y 300
# Drag from one position to another
python3 skills/desktop-control/scripts/mouse.py drag --x 100 --y 100 --to-x 500 --to-y 500
# Scroll down 3 clicks
python3 skills/desktop-control/scripts/mouse.py scroll --amount -3
# Scroll up 5 clicks at specific position
python3 skills/desktop-control/scripts/mouse.py scroll --x 500 --y 300 --amount 5
# Get current mouse position
python3 skills/desktop-control/scripts/mouse.py position
Keyboard Control
Input Parameters
| Parameter |
Required |
Description |
Example |
action |
Yes |
type, press, hotkey |
type |
text |
For type |
Text to type |
Hello World |
key |
For press |
Key name to press |
enter |
keys |
For hotkey |
Key combination, plus-separated |
command+c |
interval |
No |
Delay between keystrokes in seconds (default: 0.02) |
0.05 |
Script Usage
# Type text
python3 skills/desktop-control/scripts/keyboard.py type --text "Hello World"
# Type slowly
python3 skills/desktop-control/scripts/keyboard.py type --text "Hello" --interval 0.1
# Press a single key
python3 skills/desktop-control/scripts/keyboard.py press --key enter
python3 skills/desktop-control/scripts/keyboard.py press --key tab
python3 skills/desktop-control/scripts/keyboard.py press --key escape
# Keyboard shortcuts (hotkeys)
python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "command+c"
python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "command+shift+s"
python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "alt+tab"
python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "command+space"
Common Key Names
enter, return, tab, space, backspace, delete, escape, up, down, left, right, home, end, pageup, pagedown, f1-f12, command, ctrl, alt, shift, capslock
Screen Reading
Input Parameters
| Parameter |
Required |
Description |
Example |
action |
Yes |
info, capture, read-ui |
read-ui |
output |
For capture |
Screenshot output path |
/tmp/screen.png |
x, y, width, height |
For capture region |
Region to capture |
|
Script Usage
# Get screen size and mouse position
python3 skills/desktop-control/scripts/screen.py info
# Take a screenshot
python3 skills/desktop-control/scripts/screen.py capture --output /tmp/screen.png
# Capture a specific region
python3 skills/desktop-control/scripts/screen.py capture --x 0 --y 0 --width 800 --height 600 --output /tmp/region.png
# Read the accessibility tree of the frontmost application (MOST USEFUL)
python3 skills/desktop-control/scripts/screen.py read-ui
# Read accessibility tree with depth limit
python3 skills/desktop-control/scripts/screen.py read-ui --depth 3
The read-ui command uses AppleScript to read the accessibility tree of the frontmost application, returning window titles, buttons, text fields, menus, and other UI elements. This is the primary way to understand what's on screen before interacting.
Typical Workflow
- Read the screen to understand what's visible:
python3 skills/desktop-control/scripts/screen.py read-ui
- Identify targets from the accessibility tree output
- Interact using mouse/keyboard:
python3 skills/desktop-control/scripts/mouse.py click --x 500 --y 300
python3 skills/desktop-control/scripts/keyboard.py type --text "search query"
python3 skills/desktop-control/scripts/keyboard.py press --key enter
- Verify by reading the screen again
Example
click on the search bar
type "hello" into the text field
press command+s to save
what's on the screen right now
read the UI elements of the current window
move the mouse to the center of the screen
scroll down in this window
1---2name: desktop-control3description: Control the mouse, keyboard, and read screen content via accessibility. Use this skill when the user asks to click somewhere on screen, type text into an app, move the mouse, press keyboard shortcuts, read what's on screen, get the accessibility tree of the current window, automate desktop interactions, or control the computer.4---56# Skill: desktop-control78## When to Use910Use this skill when the user asks to:1112- Click somewhere on the screen13- Move the mouse to a position14- Type text into an application15- Press keyboard shortcuts or hotkeys16- Read what's on the current screen (accessibility tree)17- Get information about the frontmost window18- Automate desktop interactions19- Control the computer (mouse, keyboard, screen)20- Scroll up/down in an application21- Drag and drop elements2223**IMPORTANT**: This skill requires Accessibility permissions for the terminal/IDE. On macOS, go to System Settings > Privacy & Security > Accessibility and enable the running application.2425## Bundled Scripts2627| Script | Type | Description |28| --------------------- | ------ | ------------------------------------------------ |29| `scripts/mouse.py` | Python | Mouse movement, clicking, dragging, scrolling |30| `scripts/keyboard.py` | Python | Text typing, key presses, hotkeys |31| `scripts/screen.py` | Python | Screen info, capture, accessibility tree reading |3233All scripts auto-install `pyautogui` if needed.3435---3637## Mouse Control3839### Input Parameters4041| Parameter | Required | Description | Example |42| --------- | ---------- | -------------------------------------------------------------- | ------- |43| `action` | Yes | `move`, `click`, `doubleclick`, `rightclick`, `drag`, `scroll` | click |44| `x` | For most | X coordinate (pixels from left) | 500 |45| `y` | For most | Y coordinate (pixels from top) | 300 |46| `button` | No | Mouse button: `left` (default), `right`, `middle` | left |47| `to_x` | For drag | Destination X coordinate | 700 |48| `to_y` | For drag | Destination Y coordinate | 400 |49| `amount` | For scroll | Scroll amount (positive=up, negative=down) | -3 |5051### Script Usage5253```bash54# Move mouse55python3 skills/desktop-control/scripts/mouse.py move --x 500 --y 3005657# Click at position58python3 skills/desktop-control/scripts/mouse.py click --x 500 --y 3005960# Double click61python3 skills/desktop-control/scripts/mouse.py doubleclick --x 500 --y 3006263# Right click64python3 skills/desktop-control/scripts/mouse.py rightclick --x 500 --y 3006566# Drag from one position to another67python3 skills/desktop-control/scripts/mouse.py drag --x 100 --y 100 --to-x 500 --to-y 5006869# Scroll down 3 clicks70python3 skills/desktop-control/scripts/mouse.py scroll --amount -37172# Scroll up 5 clicks at specific position73python3 skills/desktop-control/scripts/mouse.py scroll --x 500 --y 300 --amount 57475# Get current mouse position76python3 skills/desktop-control/scripts/mouse.py position77```7879---8081## Keyboard Control8283### Input Parameters8485| Parameter | Required | Description | Example |86| ---------- | ---------- | --------------------------------------------------- | ----------- |87| `action` | Yes | `type`, `press`, `hotkey` | type |88| `text` | For type | Text to type | Hello World |89| `key` | For press | Key name to press | enter |90| `keys` | For hotkey | Key combination, plus-separated | command+c |91| `interval` | No | Delay between keystrokes in seconds (default: 0.02) | 0.05 |9293### Script Usage9495```bash96# Type text97python3 skills/desktop-control/scripts/keyboard.py type --text "Hello World"9899# Type slowly100python3 skills/desktop-control/scripts/keyboard.py type --text "Hello" --interval 0.1101102# Press a single key103python3 skills/desktop-control/scripts/keyboard.py press --key enter104python3 skills/desktop-control/scripts/keyboard.py press --key tab105python3 skills/desktop-control/scripts/keyboard.py press --key escape106107# Keyboard shortcuts (hotkeys)108python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "command+c"109python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "command+shift+s"110python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "alt+tab"111python3 skills/desktop-control/scripts/keyboard.py hotkey --keys "command+space"112```113114### Common Key Names115116`enter`, `return`, `tab`, `space`, `backspace`, `delete`, `escape`, `up`, `down`, `left`, `right`, `home`, `end`, `pageup`, `pagedown`, `f1`-`f12`, `command`, `ctrl`, `alt`, `shift`, `capslock`117118---119120## Screen Reading121122### Input Parameters123124| Parameter | Required | Description | Example |125| --------------------------- | ------------------ | ---------------------------- | --------------- |126| `action` | Yes | `info`, `capture`, `read-ui` | read-ui |127| `output` | For capture | Screenshot output path | /tmp/screen.png |128| `x`, `y`, `width`, `height` | For capture region | Region to capture | |129130### Script Usage131132```bash133# Get screen size and mouse position134python3 skills/desktop-control/scripts/screen.py info135136# Take a screenshot137python3 skills/desktop-control/scripts/screen.py capture --output /tmp/screen.png138139# Capture a specific region140python3 skills/desktop-control/scripts/screen.py capture --x 0 --y 0 --width 800 --height 600 --output /tmp/region.png141142# Read the accessibility tree of the frontmost application (MOST USEFUL)143python3 skills/desktop-control/scripts/screen.py read-ui144145# Read accessibility tree with depth limit146python3 skills/desktop-control/scripts/screen.py read-ui --depth 3147```148149The `read-ui` command uses AppleScript to read the accessibility tree of the frontmost application, returning window titles, buttons, text fields, menus, and other UI elements. This is the primary way to understand what's on screen before interacting.150151---152153## Typical Workflow1541551. **Read the screen** to understand what's visible:156 ```bash157 python3 skills/desktop-control/scripts/screen.py read-ui158 ```1592. **Identify targets** from the accessibility tree output1603. **Interact** using mouse/keyboard:161 ```bash162 python3 skills/desktop-control/scripts/mouse.py click --x 500 --y 300163 python3 skills/desktop-control/scripts/keyboard.py type --text "search query"164 python3 skills/desktop-control/scripts/keyboard.py press --key enter165 ```1664. **Verify** by reading the screen again167168## Example169170```171click on the search bar172type "hello" into the text field173press command+s to save174what's on the screen right now175read the UI elements of the current window176move the mouse to the center of the screen177scroll down in this window178```
dalehurley