Superset Computer Control
Operate the user's real desktop through an accessibility-first driver: a local
CLI that exposes accessibility snapshots, exact window screenshots, native menu
operations, and targeted input without making the agent guess at stale screen
coordinates. The contract in this skill (preflight, task-scoped session,
snapshot, act, verify, semantic operations first, conservative failure
handling, clean teardown) is driver-agnostic. The command reference below is
for Cua Driver, the default.
Choose the right surface
- Use this skill for native apps, OS UI, and browser windows outside Superset.
- Use
superset:browser for Superset's in-app browser pane.
- Prefer an application API, purpose-built CLI, or direct filesystem operation
when the requested result does not require GUI interaction.
Choose a driver
- If the user names a driver, use that one.
- Otherwise use whichever supported driver is already installed:
Cua Driver (
cua-driver, cross-platform,
the default) or Peekaboo (peekaboo, macOS only). If
both are installed, use Cua Driver and mention the alternative once.
- If neither is installed, ask the user which to install before installing
anything; recommend Cua Driver unless the user is macOS-only and prefers
Peekaboo.
- With a driver other than Cua Driver, apply this skill's contract through that
driver's own command surface; discover it with the driver's help output and
documentation rather than assuming the Cua command shapes below.
Set up Cua Driver (default)
Confirm the installed executable with command -v cua-driver on macOS or
Linux, or Get-Command cua-driver in Windows PowerShell, then inspect
cua-driver --version. Do not silently install or upgrade it. If it is
missing, ask the user to authorize an installer from the Cua Driver
installation guide. On
macOS or Linux, use:
/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"
On Windows, use PowerShell:
irm https://cua.ai/driver/install.ps1 | iex
cua-driver autostart enable
cua-driver autostart kick
Run cua-driver doctor and cua-driver status. On macOS, also run
cua-driver permissions status. If Accessibility or Screen Recording is
missing, ask the user to run cua-driver permissions grant and complete the
OS prompts. Never work around missing OS permissions with AppleScript,
synthetic shell input, or another GUI driver.
Preserve an already-running daemon's permission mode; do not restart it
merely to broaden permissions. If the environment needs explicit service
setup, follow the platform-specific guidance reported by doctor.
Inspect the installed tool surface instead of assuming a version-specific
schema:
cua-driver list-tools
cua-driver describe get_window_state
cua-driver describe click
Invoke tools with cua-driver call <tool> '<json>'. Use describe whenever an
argument is uncertain.
Declare a task session
Create a unique session before the first GUI observation:
cua-driver call start_session '{"session":"superset-computer-<unique>"}'
Pass the same session to every tool that accepts it. It owns this run's agent
cursor and lifecycle cleanup. It does not select the capture mode or grant
access beyond the daemon's fixed permission mode.
Snapshot, act, verify
Follow this loop for every GUI action:
- State the exact postcondition, such as "Settings shows Dark mode selected."
- Discover the target with
get_accessibility_tree or another semantic tool,
then take a fresh get_window_state snapshot for the exact PID and window.
- Prefer the snapshot's
element_token over element_index, and prefer both
over pixel coordinates. Tokens identify the control and fail closed when the
snapshot becomes stale.
- Perform one action. Default to background delivery; use foreground delivery
only when fresh evidence shows the background action did not land.
- Take a new snapshot and verify the postcondition before continuing. A
successful tool response proves delivery, not the resulting UI state.
Minimal shape:
cua-driver call get_accessibility_tree '{}'
cua-driver call get_window_state '{"pid":844,"window_id":10725,"session":"superset-computer-<unique>"}'
cua-driver call click '{"pid":844,"element_token":"s0000002a:14","session":"superset-computer-<unique>"}'
cua-driver call verify_state '{"pid":844,"window_id":10725,"session":"superset-computer-<unique>","expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]}'
Re-snapshot before every later action. Never reuse a token after a snapshot of
the same window, navigation, modal transition, or substantial repaint.
Use semantic operations first
- Use
invoke_menu for native application-menu paths.
- Use
set_value or type_text for accessible fields and press_key or
hotkey for non-text keys.
- Use
set_window_frame for window geometry and verify it with list_windows.
- For Chromium or Electron page content, discover the exact browser PID and
window ID, then call
get_browser_state. If it reports
browser_requires_setup, call browser_prepare under its approval rules.
After preparation, use its returned PID to rediscover the window and call
get_browser_state again. Use the returned target_id and tab_id with
browser_click, browser_type, browser_navigate, and browser_pointer,
refreshing the page snapshot before each later browser action.
- For browser pixel actions,
browser_click and browser_pointer coordinates
must be viewport CSS pixels from the latest get_browser_state page
snapshot. For native pixel actions, use coordinates only for canvas, video,
WebGL, or custom-drawn controls absent from the accessibility tree, and take
them from the same fresh get_window_state snapshot used by the action.
Handle failures conservatively
- If a control is absent, refresh the snapshot and inspect dialogs, sheets, and
application menus before escalating.
- If an action has no verified effect, retry only the narrowest failed step:
background accessibility, then background pixel input, then foreground input.
- Escalate to full-desktop control only after semantic, accessibility, pixel,
and foreground-window routes are exhausted, and only through the mechanism
advertised by the installed driver version.
- If the target or resulting state is ambiguous, stop and report what is
visible. Do not click through unknown dialogs or retry destructive actions.
- Leave the user's windows, focus, tabs, and clipboard as you found them unless
changing them is part of the request.
Finish cleanly
Take a final fresh snapshot or semantic readback, report the observable result,
and end only the session created for this task:
cua-driver call end_session '{"session":"superset-computer-<unique>"}'
Do not stop a shared driver daemon or close unrelated windows when the task
ends.
When the screenshots are the point, a verification run someone else has to
read, publish them instead of leaving a list of paths on their disk. Collect
them in a directory beside an index.html that lays them out, and publish the
directory so the images ride along at their relative paths:
superset pages publish ./evidence/ --workspace <id> --title "Export dialog: verified"
A desktop screenshot catches whatever else was on screen: mail, messages, a
password manager, another customer's data. Look at each one, drop or crop what
the report does not need, and pass --visibility just_me unless the org needs
it. Publishing is one of the actions the Safety rules below require you to
confirm first.
Safety
This skill reaches real apps and signed-in sessions. Limit inspection and
actions to the user's request; never extract credentials, cookies, tokens, or
unrelated private data. Obtain confirmation immediately before sending a
message, submitting a form that creates an external commitment, publishing,
purchasing, deleting data, changing an account, accepting legal terms, or
taking another consequential external action. Authentication prompts,
passkeys, passwords, and MFA stay with the user.
1---
2name: computer
3description: Operate the user's real desktop apps and windows on macOS, Windows, or Linux. Use when the user asks to open or drive a native app, click or type in a desktop UI, inspect or arrange a window, use their signed-in system browser, or verify an end-to-end GUI flow, including "open Settings and turn on dark mode", "click Save in that window", "what's on screen". Uses an accessibility-first driver (Cua Driver by default, Peekaboo on macOS). Not for Superset's in-app browser pane, headless scraping, or anything an API or CLI can do directly.
4argument-hint: the app and what to do in it
5allowed-tools: Bash(cua-driver:*) Bash(peekaboo:*) Bash(superset pages publish:*)
6metadata:
7 source: https://skillmd.com/skills/superset-sh/computer
8---
9
10# Superset Computer Control
11
12Operate the user's real desktop through an accessibility-first driver: a local
13CLI that exposes accessibility snapshots, exact window screenshots, native menu
14operations, and targeted input without making the agent guess at stale screen
15coordinates. The contract in this skill (preflight, task-scoped session,
16snapshot, act, verify, semantic operations first, conservative failure
17handling, clean teardown) is driver-agnostic. The command reference below is
18for Cua Driver, the default.
19
20## Choose the right surface
21
22- Use this skill for native apps, OS UI, and browser windows outside Superset.
23- Use `superset:browser` for Superset's in-app browser pane.
24- Prefer an application API, purpose-built CLI, or direct filesystem operation
25 when the requested result does not require GUI interaction.
26
27## Choose a driver
28
29- If the user names a driver, use that one.
30- Otherwise use whichever supported driver is already installed:
31 [Cua Driver](https://cua.ai/docs/cua-driver) (`cua-driver`, cross-platform,
32 the default) or [Peekaboo](https://peekaboo.sh) (`peekaboo`, macOS only). If
33 both are installed, use Cua Driver and mention the alternative once.
34- If neither is installed, ask the user which to install before installing
35 anything; recommend Cua Driver unless the user is macOS-only and prefers
36 Peekaboo.
37- With a driver other than Cua Driver, apply this skill's contract through that
38 driver's own command surface; discover it with the driver's help output and
39 documentation rather than assuming the Cua command shapes below.
40
41## Set up Cua Driver (default)
42
431. Confirm the installed executable with `command -v cua-driver` on macOS or
44 Linux, or `Get-Command cua-driver` in Windows PowerShell, then inspect
45 `cua-driver --version`. Do not silently install or upgrade it. If it is
46 missing, ask the user to authorize an installer from the [Cua Driver
47 installation guide](https://cua.ai/docs/how-to-guides/driver/install). On
48 macOS or Linux, use:
49
50 ```bash
51 /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"
52 ```
53
54 On Windows, use PowerShell:
55
56 ```powershell
57 irm https://cua.ai/driver/install.ps1 | iex
58 cua-driver autostart enable
59 cua-driver autostart kick
60 ```
61
622. Run `cua-driver doctor` and `cua-driver status`. On macOS, also run
63 `cua-driver permissions status`. If Accessibility or Screen Recording is
64 missing, ask the user to run `cua-driver permissions grant` and complete the
65 OS prompts. Never work around missing OS permissions with AppleScript,
66 synthetic shell input, or another GUI driver.
673. Preserve an already-running daemon's permission mode; do not restart it
68 merely to broaden permissions. If the environment needs explicit service
69 setup, follow the platform-specific guidance reported by `doctor`.
704. Inspect the installed tool surface instead of assuming a version-specific
71 schema:
72
73 ```bash
74 cua-driver list-tools
75 cua-driver describe get_window_state
76 cua-driver describe click
77 ```
78
79Invoke tools with `cua-driver call <tool> '<json>'`. Use `describe` whenever an
80argument is uncertain.
81
82## Declare a task session
83
84Create a unique session before the first GUI observation:
85
86```bash
87cua-driver call start_session '{"session":"superset-computer-<unique>"}'
88```
89
90Pass the same `session` to every tool that accepts it. It owns this run's agent
91cursor and lifecycle cleanup. It does not select the capture mode or grant
92access beyond the daemon's fixed permission mode.
93
94## Snapshot, act, verify
95
96Follow this loop for every GUI action:
97
981. State the exact postcondition, such as "Settings shows Dark mode selected."
992. Discover the target with `get_accessibility_tree` or another semantic tool,
100 then take a fresh `get_window_state` snapshot for the exact PID and window.
1013. Prefer the snapshot's `element_token` over `element_index`, and prefer both
102 over pixel coordinates. Tokens identify the control and fail closed when the
103 snapshot becomes stale.
1044. Perform one action. Default to background delivery; use foreground delivery
105 only when fresh evidence shows the background action did not land.
1065. Take a new snapshot and verify the postcondition before continuing. A
107 successful tool response proves delivery, not the resulting UI state.
108
109Minimal shape:
110
111```bash
112cua-driver call get_accessibility_tree '{}'
113cua-driver call get_window_state '{"pid":844,"window_id":10725,"session":"superset-computer-<unique>"}'
114cua-driver call click '{"pid":844,"element_token":"s0000002a:14","session":"superset-computer-<unique>"}'
115cua-driver call verify_state '{"pid":844,"window_id":10725,"session":"superset-computer-<unique>","expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]}'
116```
117
118Re-snapshot before every later action. Never reuse a token after a snapshot of
119the same window, navigation, modal transition, or substantial repaint.
120
121## Use semantic operations first
122
123- Use `invoke_menu` for native application-menu paths.
124- Use `set_value` or `type_text` for accessible fields and `press_key` or
125 `hotkey` for non-text keys.
126- Use `set_window_frame` for window geometry and verify it with `list_windows`.
127- For Chromium or Electron page content, discover the exact browser PID and
128 window ID, then call `get_browser_state`. If it reports
129 `browser_requires_setup`, call `browser_prepare` under its approval rules.
130 After preparation, use its returned PID to rediscover the window and call
131 `get_browser_state` again. Use the returned `target_id` and `tab_id` with
132 `browser_click`, `browser_type`, `browser_navigate`, and `browser_pointer`,
133 refreshing the page snapshot before each later browser action.
134- For browser pixel actions, `browser_click` and `browser_pointer` coordinates
135 must be viewport CSS pixels from the latest `get_browser_state` page
136 snapshot. For native pixel actions, use coordinates only for canvas, video,
137 WebGL, or custom-drawn controls absent from the accessibility tree, and take
138 them from the same fresh `get_window_state` snapshot used by the action.
139
140## Handle failures conservatively
141
142- If a control is absent, refresh the snapshot and inspect dialogs, sheets, and
143 application menus before escalating.
144- If an action has no verified effect, retry only the narrowest failed step:
145 background accessibility, then background pixel input, then foreground input.
146- Escalate to full-desktop control only after semantic, accessibility, pixel,
147 and foreground-window routes are exhausted, and only through the mechanism
148 advertised by the installed driver version.
149- If the target or resulting state is ambiguous, stop and report what is
150 visible. Do not click through unknown dialogs or retry destructive actions.
151- Leave the user's windows, focus, tabs, and clipboard as you found them unless
152 changing them is part of the request.
153
154## Finish cleanly
155
156Take a final fresh snapshot or semantic readback, report the observable result,
157and end only the session created for this task:
158
159```bash
160cua-driver call end_session '{"session":"superset-computer-<unique>"}'
161```
162
163Do not stop a shared driver daemon or close unrelated windows when the task
164ends.
165
166When the screenshots are the point, a verification run someone else has to
167read, publish them instead of leaving a list of paths on their disk. Collect
168them in a directory beside an `index.html` that lays them out, and publish the
169directory so the images ride along at their relative paths:
170
171```bash
172superset pages publish ./evidence/ --workspace <id> --title "Export dialog: verified"
173```
174
175A desktop screenshot catches whatever else was on screen: mail, messages, a
176password manager, another customer's data. Look at each one, drop or crop what
177the report does not need, and pass `--visibility just_me` unless the org needs
178it. Publishing is one of the actions the Safety rules below require you to
179confirm first.
180
181## Safety
182
183This skill reaches real apps and signed-in sessions. Limit inspection and
184actions to the user's request; never extract credentials, cookies, tokens, or
185unrelated private data. Obtain confirmation immediately before sending a
186message, submitting a form that creates an external commitment, publishing,
187purchasing, deleting data, changing an account, accepting legal terms, or
188taking another consequential external action. Authentication prompts,
189passkeys, passwords, and MFA stay with the user.