# Grid UI Automation

> Skill: grid GUI automation (positional clicking on a regular grid)

- Skill: `tinqiao-oss/grid-ui-automation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add tinqiao-oss/grid-ui-automation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tinqiao-oss/grid-ui-automation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tinqiao-oss (https://skillmd.com/u/tinqiao-oss)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tinqiao-oss/grid-ui-automation

---


# Skill: grid GUI automation (positional clicking on a regular grid)

For LLM agents driving a **regularly-spaced grid UI** through `clawtouch-mcp`
when keyboard navigation isn't available and the only way in is to click
cells by their screen position — a minesweeper-style board, an on-screen
number pad or keyboard, spreadsheet cells, a tile/board game, a calendar or
seat-map picker.

Two things bite agents on grids, and this skill is the disciplined recipe
that avoids both:

1. **Guessed or hardcoded coordinates drift.** A pixel that's right at one
   window size / DPI / theme is wrong at another. This skill never bakes in
   coordinates — it **derives them at runtime** from the live screen, so the
   same recipe survives layout changes. (That's why this repo otherwise
   keeps fixed coordinates out of skills; runtime calibration is the
   exception that's actually robust.)
2. **Back-to-back clicks get dropped.** Fire several clicks with no gap and
   the OS can fold or discard the earlier ones — the cursor was still
   settling, or two clicks landed too close in time to count as separate
   single-clicks. You then think you clicked five cells and only the last
   one registered.

## When to use this skill

Load this when the task is **clicking cells on a repeating grid by
position** and there is no menu / keyboard route. Skip it for ordinary app
UIs (forms, ribbons, web pages), where keyboard paths are stabler across
layouts — use the app's own skill there.

## Step 1 — calibrate the grid from a screenshot

You need a screenshot of the board. Either call `hid.screenshot` (requires
the server's `--allow-screenshot` flag) or use whatever screen capture your
client already has. **Locating the grid in that image is your job (vision /
OCR); this skill is the math and the click discipline once you have it.**

Capture the **whole grid with a margin on all four sides**. A tight crop
that clips the edge row or column makes you miscount the grid and calibrate
against the wrong cells (a clipped bottom row reads as one row short).

Pick **two well-separated cells** whose centers you can read off the image —
the wider apart, the smaller the pitch error:

- top-left cell center → `(x0, y0)`
- a cell `N` columns right and `M` rows down → `(xN, yM)`

The per-cell spacing ("pitch") is then:

```
pitch_x = (xN - x0) / N
pitch_y = (yM - y0) / M
```

Measuring across many cells and dividing beats eyeballing one cell's width.
Anchor on the **true center of a cell**, not the board's outer frame or a
3-D bevel edge — on a framed grid (minesweeper) the dark border isn't a
cell, and taking it as `(x0, y0)` shifts every coordinate by one cell.

## Step 2 — map a (row, col) to a screen pixel

With the origin and pitch, any cell center is plain arithmetic:

```
x = round(x0 + col * pitch_x)
y = round(y0 + row * pitch_y)
```

These are **absolute screen coordinates** — exactly what `hid.click` and
`hid.batch` click ops expect by default (the server reads the OS cursor
position and converges onto the target).

## Step 3 — probe once, then click the cells in one hid.batch

**Before firing a batch, do two cheap checks — a wrong calibration clicks
ten wrong cells at once, and on an irreversible grid like minesweeper that's
unrecoverable (you detonate a mine you meant to flag):**

1. **Focus the target window first.** On an unfocused window the first click
   often only *activates* it and is swallowed — the cell action doesn't
   fire. Expect to click once to focus (a screenshot will show nothing
   changed), then again to act. Some apps pass the activating click through;
   many don't.
2. **Validate the calibration with a single click.** Click one cell,
   screenshot, and confirm it landed on exactly the cell you intended. Only
   once that's confirmed do you fire a 10-cell batch.

Then send the targets as one `hid.batch` instead of one tool call per click.
It runs the ops in strict order and **spaces consecutive clicks
automatically** (a small default settle on click ops) so the OS doesn't
merge them:

```json
{
  "ops": [
    {"type": "click", "x": 320, "y": 200},
    {"type": "click", "x": 352, "y": 200},
    {"type": "click", "x": 384, "y": 200}
  ]
}
```

- **Right-click** (e.g. flagging a cell): add `"button": "right"`.
- **The cap is 10 ops per batch.** For a bigger sweep, send several batches.
- **Don't pass `relative: true`** here — you computed absolute pixels in
  Step 2, and absolute mode is what converges onto them. (`relative` is for
  raw deltas, e.g. on Wayland where the OS cursor can't be read.)
- A sluggish app may need more room than the default gap — set
  `"delay_ms": 120` on the click ops (keep it within the 2000 ms per-op
  ceiling).

## Step 4 — verify, then retry the misses (the step agents skip)

This is what turns "I sent clicks" into "the cells actually changed".
`hid.batch` returning `clicked: true` per op only means the HID report went
out and the cursor converged — **it cannot see whether the app registered
the click** (USB HID has no app-level feedback). So:

1. **Screenshot again** after the batch.
2. **Compare** against what you expected — did those cells reveal / toggle /
   fill?
3. **Re-click only the cells that didn't change** — and at most once or
   twice.

Re-clicking only fixes a genuinely *dropped* click. If a cell still hasn't
changed after one or two retries, the cause is usually **not** a drop — you
misread the board, or the calibration drifted (the window moved or
scrolled), and re-clicking the same pixel won't help. Stop, take a fresh
screenshot, re-read the whole board, and recalibrate (Step 1) before
continuing. Sanity-check your *reading* too — a cell that "didn't change"
may be one you misidentified, not one you missed. This
screenshot → act → screenshot loop is what makes grid automation reliable;
the click itself is open-loop.

## Worked example — Minesweeper

A minesweeper-style board is the canonical grid: fixed cell pitch, left-click
to reveal, right-click to flag.

1. **Calibrate**: screenshot the board; read the top-left cell center and a
   far-corner cell; compute `pitch_x` / `pitch_y` (Step 1).
2. **Decide** (your logic, not this skill): from the revealed numbers, work
   out which cells are safe and which are mines.
3. **Reveal** a batch of safe cells — up to 10 left-clicks in one
   `hid.batch` (Step 3).
4. **Flag** known mines — right-clicks (`"button": "right"`) in another
   batch.
5. **Verify**: screenshot; revealed cells should now show numbers and
   flagged ones a marker. Re-click any that didn't take (Step 4), then
   recompute and continue.

The calibrate → batch → verify loop is identical for an on-screen keypad
(enter a PIN by clicking digit cells), a tile puzzle, or selecting cells in
a sheet — only the "which cells" logic changes.

## Known gotchas

- **`clicked: true` is not "the app acted".** It confirms the HID report and
  cursor landing only. Always verify by screenshot (Step 4). This is the
  single most common way grid runs go wrong.
- **High-DPI / Retina**: `hid.screenshot` returns the image in the same
  logical-point space your click coordinates use (its `scale_x` / `scale_y`
  metadata is ~1.0 after auto-resize), so pixels read off that screenshot
  map 1:1 to click coordinates — no manual ×2. If you capture the screen
  some other way at physical resolution, divide by the DPI scale first.
- **The board moved or scrolled → recalibrate.** Pitch and origin are only
  valid for the window position you measured. After any scroll, resize, or
  window move, take a fresh screenshot and redo Step 1.
- **10-op batch cap.** A large grid needs several batches; verify between
  them so you don't pile retries onto an already-wrong state.
- **Absolute clicks need the OS cursor.** On Wayland (or any host where the
  cursor query fails) absolute mode returns an error — fall back to
  `relative: true` deltas, tracking the cursor position yourself. Inside a
  batch, that op comes back `{ok: false, error: ...}`, and with the default
  `stop_on_error: true` the rest of the batch halts there.
- **Set `--screen WxH`** on the server to your real display so coordinates
  are clamped to the right monitor.

## When this skill doesn't help

- **Non-grid UIs** — forms, ribbons, menus, web pages. Prefer keyboard paths
  (stabler across layouts) or the app's own skill.
- **No way to read the screen** — this loop needs a screenshot to calibrate
  and to verify. With only `hid.*` and no vision/capture, you'd be clicking
  blind; don't.
- **Irregular or non-uniform grids** — cells of varying size, merged cells,
  or perspective/rotation break the constant-pitch assumption. Calibrate per
  region, or click each cell individually with its own verification.
- **A keyboard route exists** — many "grids" (spreadsheets, calendars)
  navigate by arrow keys / Tab / a cell-reference box. That's more reliable
  than positional clicking; use it when available.