Appsnap

appsnap

Capture one selected macOS window and return the screenshot plus extracted text as context.

Scope

• Use local macOS APIs only. Do not use network or model calls inside scripts.
• Treat the result as passive context. Do not click, type, or control the app.
• Exception: default auto mode and --fullpage / {"mode":"fullpage"} may be ACTIVE. Auto hover-picks a window and scroll-stitches only when a bounded AX probe finds a scrollable area. --visible forces a passive single visible screenshot.
• The screenshot is primary context; you (the consuming agent) read it with your own vision. Accessibility (AX) text is supplementary: its unique value is off-screen / not-scrolled-into-view content. Local OCR is OFF by default — your own vision reads a high-res screenshot more accurately than macOS Vision OCR. Only opt into OCR for non-vision / headless consumers.
• Be honest for canvas-heavy apps (Google Docs, Sheets, Slides, Gmail, many browsers): AX may expose little or no text. In that case assemble.py returns read_image:true and you must read the screenshot for the visible content. Off-screen canvas text is not reachable by anyone unless the app exposes it through AX.

One-time setup

Scripts need macOS PyObjC frameworks. Create a venv and install:

python3 -m venv <skill-dir>/.venv
<skill-dir>/.venv/bin/pip install -r <skill-dir>/requirements.txt

Run every script below with <skill-dir>/.venv/bin/python (not the system python3).

Flow

1. Check permissions:

   python3 <skill-dir>/scripts/check_permissions.py
   

   If it exits nonzero, stop and show its one-line stderr guidance.

2. Capture a window:

   python3 <skill-dir>/scripts/capture.py
   

   Stdin is optional. Default is {"mode":"auto"}. Use {"mode":"visible"} for hover-pick without scrolling, {"mode":"fullpage"} to force active scroll-stitching, or {"mode":"frontmost"} only when the user explicitly wants the frontmost window.

3. Extract AX text with the capture output:

   python3 <skill-dir>/scripts/ax_extract.py
   

   Pass {"pid":123,"windowid":456} from the capture JSON. If ax_extract.py is unavailable or returns ax_available:false, continue to OCR.

4. Assemble the final bundle (OCR stays off by default):

   python3 <skill-dir>/scripts/assemble.py
   

   Pass {"capture":{...},"ax":{...},"ocr":null}. To force local OCR for a non-vision consumer, pass {"ocr_requested":true} or set APPSNAP_OCR=1.

5. Present the bundle to the current agent and ACT ON read_image:

   • Always surface image_path (the high-res screenshot) and text_markdown. For full-page capture, surface every path in image_paths.
   • If read_image is true (AX was thin/absent — common for browsers and canvas apps), read the screenshot yourself for the visible content; text_markdown only covers what AX exposed.
   • If read_image is false, AX returned the full document; text_markdown is the primary context and the image is confirmation.

scripts/ocr_fallback.py exists for the opt-in / headless path: pass {"image_path":"<abs png>"}; it recovers visible text only.

Script Contracts

scripts/capture.py

Input:

{"mode":"auto"}

Output:

{"ok":true,"pid":123,"windowid":456,"app":"Safari","title":"...","bounds":{"x":0,"y":0,"w":1440,"h":900},"image_path":"/abs/window.png"}

For opt-in active full-page mode:

{"ok":true,"pid":123,"windowid":456,"app":"Safari","title":"...","bounds":{"x":0,"y":0,"w":1440,"h":900},"image_paths":["/abs/page-01.png"],"stitched":true,"frames":5,"stopped_by":"bottom"}

scripts/ax_extract.py

Input:

{"pid":123,"windowid":456}

Output:

{"ok":true,"source":"ax","text_markdown":"...","char_count":123,"element_count":45,"truncated":false,"ax_available":true,"note":""}

scripts/ocr_fallback.py

Input:

{"image_path":"/abs/window.png"}

Output:

{"ok":true,"source":"ocr","text_markdown":"...","char_count":123}

scripts/assemble.py

Input:

{"capture":{},"ax":{},"ocr":null}

Output:

{"image_path":"/abs/window.png","image_paths":["/abs/window.png"],"text_markdown":"...","text_source":"ax|ax+ocr|ocr|image-only","read_image":true,"primary_context":"image|text","app":"Safari","title":"...","truncated":false,"summary_line":"appsnap of Safari «title» — 123 chars via AX; read the screenshot for visible content"}

read_image:true ⇒ AX was thin/absent; the consuming agent must read the screenshot. text_source:"image-only" ⇒ no text from AX (and OCR not requested); the screenshot is the only context — this is a valid result, not an error.

Housekeeping

Captures are saved to ~/Pictures/appsnap/ (override with APPSNAP_DIR). The one-shot appsnap.sh prints an in-session reminder when screenshots older than APPSNAP_RETENTION_DAYS (default 7) exist, and appsnap.sh --clean deletes them. When you see that reminder, surface it to the user and offer to run --clean — never auto-delete.

Failure Rules

• Scripts read JSON from stdin and write JSON to stdout.
• Scripts write one-line stderr and exit nonzero on errors.
• If permissions are missing, stop before capture or extraction.
• If both AX and OCR produce no text, report that no text was available and keep the screenshot path as the only usable context.