ZachRouan gridhand
Desktop GUI automation CLI. All commands return JSON to stdout (errors to stderr).
gridhand screenshot --output /tmp/screen.png # full screen
gridhand screenshot --window-id 123 --output /tmp/app.png # cropped to window
gridhand screenshot --window-id 123 --grid --output /tmp/grid.png # grid overlay
gridhand screenshot --window-id 123 --grid --cell B2 --output /tmp/z.png # zoom into cell
gridhand windows list # JSON list of all windows
gridhand windows raise 123 # bring window to front
gridhand mouse click # click at current position
gridhand mouse click --button right # right-click
gridhand mouse click --cell B2.C1 --window-id 123 # click cell (recursive zoom OK)
gridhand key type "hello" --window-id 123 # type text
gridhand key press "ctrl+a" --window-id 123 # key combo (ctrl/shift/alt/super + key)
Grid Targeting
No pixel coordinates exist. The grid is the only way to click. Each cell has a cyan crosshair (+) at its center — a click on that cell lands exactly on that crosshair, nowhere else. If the crosshair isn't on your target, the click will miss. Zoom until a crosshair sits directly on the target.
Orient → Zoom → Click → Verify
Orient. Take a grid screenshot. Identify which cell contains your target and note where within that cell it sits (e.g. "the button is in D1, near the bottom-left").
Zoom. Crop into that cell to get a sub-grid. Pick the sub-cell matching your position note — don't re-read text or hunt for icons in zoomed views, they'll be blurry. Translate spatially: "bottom-left of D1" → sub-cell B5 or C5. Keep zooming (append with dots: D1.C5.F3) until a crosshair is on the target. 2–3 levels is typical.
Click. gridhand mouse click --cell D1.C5 --window-id 123
Verify. Zoom into the area you just clicked to check the result — a full-page screenshot is too small to see subtle state changes (button color, focus ring, selection highlight). If the click missed, re-orient from scratch — the screen state may have changed.
Navigating Without Zooming Out
Zoomed views show dimmed context from adjacent parent cells with their labels visible. If your target is in a neighboring cell, reference it directly — do not zoom all the way out:
# Zoomed into G1, but target is actually in H1 (visible in dimmed context)
# WRONG: take a fresh full-grid screenshot and start over
# RIGHT: just use H1 directly
gridhand screenshot --window-id 123 --grid --cell H1 --output /tmp/zoom.png
Only take a fresh full-grid screenshot (no --cell) when the screen has changed (after clicking, typing, or switching windows) or when you're genuinely lost.
Between-Cell Targeting
When you locate a target, report every cell it touches, not just one — a target on a grid line is in two cells, and picking one silently biases your click toward the wrong side. Then:
- One cell: click it normally.
- Two adjacent cells (side by side or stacked): use
+to click the shared boundary. - Three or four cells (target sits on a corner, or spans further): don't guess — zoom into one of them first.
gridhand mouse click --cell D3+E3 --window-id 123 # horizontal
gridhand mouse click --cell D3+D4 --window-id 123 # vertical
gridhand mouse click --cell D3+E4 --window-id 123 # diagonal
Small Icons and Buttons
Tiny targets (like/dislike buttons, close icons, checkboxes) need 3+ zoom levels. On the first zoom, note the icon's position carefully — at deeper zooms it becomes a few blurry pixels. Trust your spatial note, not what the zoomed crop looks like.
Dot Notation
B2.C1 = sub-cell C1 within parent B2. Change the last segment to try a neighbor at the same depth (B2.D2). Append to go deeper (B2.C1.F3).
Known Failure Modes
- Non-ASCII text on Linux:
gridhand key typeerrors on non-ASCII characters on Linux (the uinput backend only synthesizes ASCII keycodes). Stick to ASCII text, or split out non-ASCII characters and handle them another way (e.g. clipboard paste via a key combo, if the target app supports it). - Zoom chains bottoming out: Recursive zoom (
B2.C1.F3...) fails with a "cell size reaches zero" error around 3 levels deep, once the cropped region gets too small to subdivide further. If you hit this, back off one level and either retry with fewer zoom levels, or take a fresh screenshot with an explicit coarser--grid(e.g.--grid 4x4) so each cell covers more pixels.