Computer Use Automation

The tool

computer_use_tool runs Python on the computer you are operating — its screen, its files, its apps. It is not a sandbox and not your own machine.

Keep the browser plane fixed

cu controls browsers installed on the Windows desktop. bu controls a separate product browser and cannot access the local browser's window or session.

For each browser workflow, choose one plane and keep it fixed. If the user requests a local or installed browser (such as Chrome, Edge, or Firefox on the desktop), or the task depends on an existing desktop browser window or profile:

• Use cu for every browser action until the task is complete. A webpage displayed in that browser remains a cu target.
• Do not use bu to perform, inspect, continue, retry, or verify any part of the workflow.
• If the local browser is unavailable or blocked, report the blocker or ask before changing planes; never switch silently.

If the user explicitly requests the product browser, use bu instead and keep that browser workflow on bu.

computer_use_tool(plane="cu", code="...", title="Open notification settings")

plane declares which library the cell drives; a desktop cell declares "cu", and it is checked against the code. A cell that calls no plane library at all is refused: file, parsing and shell work goes to Read / Write / Glob / Grep and the shell tool — for a longer program, Write a .py file and run it there.

The desktop library is already installed:

import seed_computer_use as cu   # the desktop, 0-1000 screen coordinates

The whole standard library works too. That matters more than it sounds: checking a file you just saved, polling for a download, looping over rows, computing coordinates — all of it belongs in the same call as the cu actions it serves, not spread across several. A loop that fills twelve cells is one call, not twelve. Opening an application is no exception: discovery and launch belong in the same call.

Since nothing survives a call, anything you need later goes onto the machine (a file) or into your own message — a value you computed is only usable in the call that computed it.

title is shown to a human as the activity label. Write the intent in a few words ("Fill the year column"), not the mechanism ("call cu.click four times").

Required user takeover

When a step requires the user to act personally—for example signing in, entering credentials, completing a CAPTCHA or MFA challenge, granting consent, or taking manual control of the current app or browser—call the session's user-takeover tool (currently interaction.request_action). Calling the tool is required: do not merely ask the user in chat, keep retrying automation, or work around the gate.

After control returns, assume the UI may have changed. Re-observe the exact app or desktop before any further action and derive new coordinates from that new observation. If no user-takeover tool is available, report the blocker and stop instead of claiming that a plain-text instruction transferred control.

Looking at the screen

import seed_computer_use as cu     # every call, not just the first
cu.click(504, 358)
cu.screenshot()          # <- the frame comes back as an image in this result

Asking for the screen is the single biggest habit to change. Acting without ever looking is the most common way to fail: you will click a menu item that moved, type into a window that never took focus, and never find out.

Observe when the next decision depends on what is on screen. Do not take a frame after every single action — a batch of actions followed by one frame is usually right, and images are expensive.

Coordinates

cu uses 0-1000 on both axes over the whole screen: (0,0) top-left, (1000,1000) bottom-right, (500,500) dead centre. Read a target's position off a screenshot as a fraction of the image and give it in those units. Never pass raw pixels.

Apps and launching

launch_app(name) only accepts an exact App.name bound by a list_apps() call in this session, so discovery comes first — in the same cell is fine:

import seed_computer_use as cu
apps = cu.list_apps()
matches = [app for app in apps if "notepad" in app.name.casefold()]
if len(matches) != 1:
    print(matches[:5])
    raise RuntimeError("expected exactly one match")
result = cu.launch_app(matches[0].name)
print(result)                 # status="ready" or "accepted_unverified"
cu.screenshot()               # mandatory fresh whole-desktop observation

Print the candidates you filtered down to, as above — the launch itself is verified against the catalog, but a wrong choice is only visible to you.

The two public application methods are:

• list_apps() → complete AppCatalog[App]; each App is only {name}, and every name in one listing is unique ignoring case
• launch_app(name) → launch one exact App.name from the current listing, then return status="ready" with matching window evidence or status="accepted_unverified" when Windows accepted the request but no matching visible window was confirmed yet

name is the application's own Start-menu name. When two of them share one, the listing qualifies both — Calculator (Store) and Calculator (Desktop) — so every application in a listing has exactly one spelling. The match is exact: a differently-cased name, a padded one, or part of one is refused. It is never a path, an executable, an AppID/AUMID, or a launch target. Keep the selected name in your own message for the later launch cell.

AppCatalog is a normal complete list for iteration, indexing, slicing, and local filtering. Its raw repr is bounded to a short preview, but the underlying list is not truncated. Keep it in apps, filter in Python, and print only matches; an app beyond the preview remains discoverable by local code.

Every list_apps() call creates a new discovery generation and supersedes the previous listing's bindings; a new session invalidates them too. launch_app rejects missing, unknown, or changed-descriptor names, and it rejects a differently-cased or partial name, a URI, a bare executable, or a path even when Windows itself could launch it. list_apps() → launch_app(exact App.name) is the only way an application gets opened.

For Win32 applications, launch_app uses the resolved executable, arguments, and working directory from the live catalog; packaged apps use AppsFolder. It returns status="ready" only after a matching visible window appears. An accepted request that is still unconfirmed returns accepted_unverified; this is not an error and must not be launched a second time. Observe the whole screen and wait for the existing request instead. To switch to an app that is already open, use cu.screenshot(), cu.hotkey("alt", "tab"), then take another screenshot; do not use launch_app as a focus operation.

A cu.screenshot() covers the whole screen. Observe again after focus, layout, menu, dialog, or application changes before deriving another coordinate.

Batching coordinates: when it is safe

Batching is why this tool exists, but coordinates come from a specific frame and some actions invalidate them.

Safe — the actions do not change the layout:

import seed_computer_use as cu
cells = [(77, 306, "2015"), (77, 322, "2016"), (77, 338, "2017")]
for x, y, v in cells:
    cu.click(x, y)
    cu.type(v)
    cu.hotkey("enter")
cu.screenshot()

Not safe — the action changes what is on screen, so anything after it needs a new frame:

cu.click(120, 40)      # opens a menu
cu.click(140, 95)      # WRONG: this coordinate did not exist when you looked

Opening a menu, opening a dialog, switching tabs and scrolling all fall in this group. Do the state-changing action, take a frame, and end the cell there. Writing both halves in one cell cannot work — the second coordinate has not been observed yet.

The helpers

cu.click(x, y)              cu.left_double(x, y)      cu.triple_click(x, y)
cu.right_single(x, y)       cu.hover(x, y)            cu.drag(x1, y1, x2, y2)
cu.scroll(x, y, delta)      cu.type(text)             cu.hotkey(*keys)
cu.press(key)               cu.key_down(key)          cu.key_up(key)
cu.wait(seconds)
cu.screenshot(tag=None)     cu.list_apps()            cu.launch_app(name)
cu.get_clipboard()          cu.set_clipboard(text)    cu.copy_selection()

• cu.type("hi\n") — a trailing newline presses Enter. Non-ASCII text (Chinese, say) is handled for you, as is text too long to type key-by-key.
• cu.copy_selection() presses Ctrl+C and hands back what was caught — the cheapest way to check a selection landed where you meant. Use cu.get_clipboard rather than tkinter, raw ctypes, or Get-Clipboard; those all misbehave here.
• cu.scroll(x, y, delta) accepts a non-zero integer from -10 to 10; positive scrolls up and negative scrolls down. Exact aliases "up", "down", "left", and "right" are also supported, for example cu.scroll(500, 500, "right"). There is no amount= keyword here; that one belongs to the page API.
• cu.hotkey("enter") taps one key; cu.hotkey("ctrl", "s") taps a combination. Pass each key separately rather than writing "ctrl+s". cu.press(key) also taps one key. To hold a key across actions, pair cu.key_down(key) with cu.key_up(key).
• cu.wait(seconds) is measured in seconds (maximum 180). Millisecond-style values such as 1000 are rejected; use cu.wait(1) instead.
• cu.hover(x, y) moves the pointer without clicking; cu.move and cu.move_mouse are the same function.
• cu.screenshot(tag="after") — the tag only matters when one cell takes more than one frame; plain cu.screenshot() is the normal call. It returns the PNG bytes, and .width / .height on that value give you the screen size.
• Windows paths: use a raw string such as r"C:\Users\Public\report.txt", doubled backslashes such as "C:\\Users\\Public\\report.txt", or os.path.join — never paste C:\Users\... into an ordinary Python string (\U starts an escape).

Verify by reading, not only by looking

A screenshot tells you what a window is showing; the file system tells you what actually happened. When you have just saved, exported or downloaded something, read it back in the same cell:

import os, time
import seed_computer_use as cu
cu.hotkey("ctrl", "s")
cu.wait(2)
p = os.path.join(os.environ["USERPROFILE"], "Documents", "report.xlsx")
print("saved:", os.path.exists(p), os.path.getsize(p) if os.path.exists(p) else 0)

That check costs nothing, needs no image, and catches the case where the save dialog silently went somewhere else.