# Drive Tui > Drive interactive terminal (TUI) programs — CLIs, REPLs, installers, menu apps, agent CLIs, and editors like vim — through a PTY, reading semantic screen snapshots. A pattern library classifies a screen (REPL, menu, pager, fzf search, confirm dialog, form, spinner, wizard) and drives it with a ready recipe. Use when a program expects a live terminal (arrow-key menus, prompts, spinners, password fields, curses UIs), or when a piped command hangs or prints nothing. - Skill: `mediar-ai/drive-tui` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add mediar-ai/drive-tui` - Raw SKILL.md: https://api.skillmd.com/api/skills/mediar-ai/drive-tui/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: mediar-ai (https://skillmd.com/u/mediar-ai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mediar-ai/drive-tui --- # drive-tui Drive interactive terminal programs the way a human does: look at the screen, decide, press keys, wait, look again. A normal `subprocess`/pipe can't do this — line-buffered pipes hide menus, spinners, and prompts, and many programs refuse to run without a real TTY. This skill uses `smartcli_core` (a PTY + `pyte` screen model + semantic snapshots + readiness waits) behind a thin CLI, `scripts/tui.py`. ## When to use - Launching a program that draws a full-screen or line-mode UI and expects keystrokes (REPLs, installers, `vim`, agent CLIs, arrow-key menus, y/n prompts, password entry). - A piped command hangs, prints nothing, or says "not a terminal". - You need to see what an interactive program is showing right now, then act on it. - Do NOT use for non-interactive commands whose full output you just want captured — run those directly. ## Core mental model: the perceive → decide → act loop Never fire keystrokes blind and never guess with `sleep`. Every interaction is one turn of this loop: 1. **Perceive** — take a snapshot. Read the header line (cursor position, selected row, status bar, errors) and the row-numbered body. 2. **Decide** — classify the screen: is it a menu, a text prompt, a confirmation, a spinner still loading, an error, or a finished result? Decide the single next action. 3. **Act** — translate intent into key tokens and send them (`send-line`, `send-text`, or `keys`). 4. **Wait** — call `wait` / `wait-regex`. NEVER `sleep`. The wait pumps output until the screen settles or an expected marker appears. 5. **Confirm** — re-snapshot and verify the screen changed as expected before the next action. ## Setup `scripts/tui.py` adds the repo root to `sys.path` itself, so run it from anywhere. It needs `smartcli_core` importable (default: three levels up from the script) and the deps `pyte` + `pywinpty` (Windows) already installed. Override the repo location with `SMARTCLI_ROOT` if needed. All commands below are shown from the repo root. Use the exact form: `python skills/drive-tui/scripts/tui.py ...` ## Two ways to drive ### A. Persistent session (interactive, multi-turn) — the default A detached daemon owns one live program; each command connects over a localhost-only socket, so state survives across separate shell calls. This is the real perceive→decide→act loop. 1. Start (prints a session id — capture it): `python skills/drive-tui/scripts/tui.py start --cmd "python" --cols 100 --rows 30` 2. Wait for the first prompt, then snapshot is printed automatically: `python skills/drive-tui/scripts/tui.py wait-regex --id ">>> " --timeout-ms 15000` 3. Act, then wait, then it re-snapshots: `python skills/drive-tui/scripts/tui.py send-line --id "print(6*7)"` `python skills/drive-tui/scripts/tui.py wait --id ` 4. Snapshot any time without acting: `python skills/drive-tui/scripts/tui.py snapshot --id ` Add `--json` for the structured `Snapshot.to_json()` form (selected span, menu_items, errors). 5. Check liveness / list / close: `python skills/drive-tui/scripts/tui.py alive --id ` (exit 0 alive, 1 dead) `python skills/drive-tui/scripts/tui.py list` `python skills/drive-tui/scripts/tui.py close --id ` (always close when done) Subcommands: `start`, `snapshot`, `send-text`, `send-line`, `keys`, `wait`, `wait-regex`, `alive`, `close`, `list`. `wait`/`wait-regex` print liveness + reason to stderr and the snapshot to stdout. ### B. One-shot script (batch, non-interactive) When you already know the whole sequence, run it in a single process against a fresh program. No session to manage. Write a JSON list of steps to a file and run: `python skills/drive-tui/scripts/tui.py run --cmd "python" --steps steps.json --cols 100 --rows 30` Step objects (executed in order; each `wait_*`/`snapshot` prints a snapshot): - `{"action":"send_text","text":"..."}` — type literally, no Enter. - `{"action":"send_line","text":"..."}` — type + Enter. - `{"action":"send_keys","keys":["Down","Down","Enter"]}` — key tokens. - `{"action":"wait_ready","marker":"regex"}` — wait for marker OR stability (marker optional). - `{"action":"wait_regex","pattern":"regex","timeout_ms":10000}` — wait strictly for the regex. - `{"action":"snapshot"}` — print the current screen. ## Reading a snapshot `to_text()` output = one header line + row-numbered body: ``` [screen 24x80] cursor=r3c4 selected=r3[0:80]">>>"(cursor_line) status="..." errors=1 0 | Python 3.14.6 ... 3 | >>> ``` - Header: `cursor=rNcM`, optional `selected=...`, `status="..."`, `title`, `errors=N`, `screen_reverse`. - Body: `<*>| text`. A `*` marks a selected/highlighted row. Blank runs collapse to `...`. - **`selected_line` is the key menu signal.** In an arrow-key menu the highlighted (reverse-video / colored) row is the current choice — the snapshot surfaces it as the `*` row and in the header `selected`. Use it to know where the cursor sits before pressing Up/Down. ## Deciding: classify the screen - Prompt (`>>> `, `$ `, `Password:`, `Continue? [y/N]`) → send the answer with `send-line`, or a single key with `keys`. - Menu (a `*`/`selected` row, list of options) → move with `keys Up`/`keys Down`, choose with `keys Enter`. Do not type the option text. - Spinner / progress / still loading → do not act; call `wait` again (it caps at `max_wait_ms` and returns the last screen). - Error (`errors=N` in header, red text, "Traceback"/"failed") → stop and read; do not keep sending input. - Finished result → capture the snapshot; act or close. ## Translating intent to keys (`keys` subcommand / `send_keys`) Named tokens: `Enter Tab BackTab Space Backspace Delete Escape Up Down Left Right Home End PageUp PageDown Insert F1`–`F12`. Combos: `C-c` / `^C` (Ctrl+C), `M-x` (Alt+x). Anything else is sent literally. - Menus/keys → `keys`. Free text + submit → `send-line`. Text without submit (fills a field) → `send-text`. - `send-line` submits with `\r`. A raw-mode app that needs `\n` → use `send-text "...\n"`. ## Choosing the right wait — do NOT sleep - Know the exact text that means "ready" (a prompt, a banner) → `wait-regex `. **Prefer this.** It watches strictly for the regex and will NOT return early. - Don't know the marker (screen just needs to settle after a keystroke) → `wait` (marker optional): returns on stability or a marker, capped by timeout. - **Startup gotcha:** a program can sit quiet for seconds before its banner appears (Python's REPL banner arrives ~3s after spawn on Windows ConPTY). Plain `wait`/`wait_ready` may declare `STABLE` on the still-blank screen during that gap. So for the FIRST prompt after `start`, use `wait-regex` with a generous `--timeout-ms` (e.g. 15000), not bare `wait`. - Regex tips: match against the whole screen text. Anchoring with `$` needs multiline semantics — prefer a loose marker like `">>> "` over `">>> $"`. Confirm the result with a follow-up `snapshot`. ## Recovering when a program doesn't respond 1. Re-`snapshot` — confirm it actually didn't change (you may have missed output). 2. `alive --id ` — if dead, the child exited; read the final snapshot for the reason, then `close`. 3. Still alive but stuck: try `keys Enter` (dismiss a pager/prompt) or `keys Escape` (leave a mode). 4. Interrupt a running command: `keys C-c`. **Windows caveat:** under ConPTY a raw Ctrl-C byte does NOT reliably raise an interrupt in a line-mode child (verified — `time.sleep` in the Python REPL was not interrupted). On Windows, if `C-c` fails, the reliable recovery is `close --id ` and `start` a fresh session. On POSIX, `C-c` works normally. 5. Always `close` sessions you started (frees the daemon and the child). ## Constraints / gotchas - Never blind-`sleep` to wait for output — use `wait` / `wait-regex`; they pump the PTY and have hard timeouts. - Always re-snapshot after acting; the snapshot before your keystroke is stale. - Navigation sequences are standard xterm/VT100 (normal cursor mode). An app in application-cursor-keys mode may need different Up/Down bytes — if arrows don't move the selection, re-snapshot and try `keys Down` once more or use the app's letter/number shortcuts. - The daemon binds `127.0.0.1` only — local process control, no network exposure. - Paths above are relative to this skill's folder; run the script by its path from the repo root. - Deeper API details: read `references/core_api.md` only when you need field-level specifics. ## Knowledge base — look before you build Before scripting a new interaction or pattern, consult the SmartCLI knowledge graph at `D:/Project/SmartCLI/knowledge/INDEX.md`. Its `tui-patterns/` domain is a 1:1 conceptual map of the 8 recipes below — [[list-menu-navigation]], [[fuzzy-search-filter]], [[pager-navigation]], [[confirm-yes-no-dialog]], [[progress-spinner-waiting]], [[form-field-input]], [[repl-session]], [[wizard-installer-flow]] — plus the sourced foundations behind every gotcha in this skill: [[application-cursor-mode]] (the exact arrow-keys-don't-move bug), [[quiescence-detection]] and [[snapshot-stability-hash]] (how `wait` decides "settled"), [[key-encoding-reference]] (key tokens → bytes), [[cursor-row-binding]] and [[verify-movement-step-by-step]] (why matches bind to the cursor row and every action re-snapshots). Read the relevant note before inventing a recipe from scratch. ## Pattern library — classify a screen, drive it with a recipe Above is the manual loop through `scripts/tui.py`. For recurring paradigms there is a Python pattern library (`skills/drive-tui/patterns/`) that recognizes a screen and drives it for you. It works on `smartcli_core` `Snapshot`/`PtySession` objects, so use it from Python (import `patterns`), not the `tui.py` CLI. Add `skills/drive-tui` to `sys.path`, or run from there. ### Classify / explain a screen - `classify(snapshot, threshold=0.05) -> [(Pattern, confidence), ...]` — every registered pattern scored 0..1 against the snapshot, highest first. A pattern whose `matches()` raises is scored 0, never poisoning the ranking. - `explain(snapshot) -> str` — a human/agent-readable description: raw facts (size, cursor, highlighted row + reason, menu spans, status bar, errors, screen-reverse) followed by the ranked paradigms with confidences. ```python import sys; sys.path.insert(0, "skills/drive-tui") from patterns import registry from patterns.classify import classify, explain snap = session.snapshot() # a smartcli_core PtySession print(explain(snap)) # what does this screen look like? best, conf = classify(snap)[0] # top paradigm ``` ### The 8 recipes (registry names → tags) Live catalog: `[p.name for p in registry.all_patterns()]`. | name | drives | intent | tags | |------|--------|--------|------| | `repl` | interactive REPL/shell prompt | one code line to run | repl, shell, prompt, interactive | | `menu_select` | arrow-key vertical menu | int index or substring | menu, list, navigation | | `pager` | less/more/man/git-log | `to_end` / `next_page` / `search:` | pager, scroll | | `search_filter` | fzf / Ctrl-R / palette | the query string | fuzzy, filter, incremental | | `confirm` | `[y/N]` / `(yes/no)` dialog | bool (or yes/no string) | dialog, yesno | | `form` | labelled input fields | dict/list/str of field values | form, input | | `progress` | spinner / progress bar | optional completion regex | spinner, progress, wait | | `wizard` | multi-step installer | list of per-step actions | wizard, multistep, installer | ### Call a recipe Get the registered singleton and call `.drive(session, intent, **kw)`. It runs the perceive→act→wait→confirm loop against the live session and returns a `PatternResult(ok, detail, snapshot, data)` — `ok` is a real screen assertion, `snapshot` is the last screen as evidence, `data` is the pattern-specific payload. ```python from patterns import registry # REPL: run a line, confirm a fresh prompt returned + cursor advanced res = registry.get("repl").drive(session, intent="6*7") assert res.ok; print(res.data["output"]) # ['42'] # menu: move the highlight to a target and press Enter (confirms the bar moved) registry.get("menu_select").drive(session, intent="Charlie") # or intent=2 registry.get("menu_select").drive(session, intent=0, press=False) # position only # pager: page to the bottom registry.get("pager").drive(session, intent="to_end", max_pages=200) # search: narrow a fuzzy list, then pick the top match registry.get("search_filter").drive(session, intent="grapef", stop_at=1, accept=True) # confirm: answer yes (commits with Enter if a line-mode child only echoed) registry.get("confirm").drive(session, intent=True) # form: fill fields, Tab between, Enter to submit registry.get("form").drive(session, intent={"Name": "Ada", "Email": "a@b.c"}) # progress: wait for a done marker, else for the animation to settle (hard ceiling) registry.get("progress").drive(session, intent=r"DONE!", max_wait_ms=60000) # wizard: drive a scripted multi-step flow registry.get("wizard").drive(session, intent=[ {"keys": ["Down", "Enter"]}, {"pattern": "confirm", "value": True}, None]) ``` Module-level convenience helpers: `patterns.recipes.repl_session.run_line(session, code, **kw)` and `patterns.recipes.paginate.read_all(session, max_pages=200, key="Space") -> (text, PatternResult)` (accumulates the pager's full text across pages, dedup'ing repainted boundary rows). Recipe `drive` kwargs worth knowing: - `repl`: `expect=` (confirm a marker instead of waiting for STABLE), `timeout_ms`, `quiet_ms`, `min_wait_ms`. - `menu_select`: `max_steps`, `settle_ms`, `press` (send Enter, default True), `enter_wait_ms`. - `pager`: `max_pages`, `key` (`Space`|`PageDown`|`f`, default `Space`; `f` is the less/more forward-page key). - `search_filter`: `incremental` (default True), `stop_at` (match-count threshold, default 1), `accept` (Enter to pick), `settle_ms`. - `form`: `next_key` (default `Tab`), `submit` (default `Enter`); intent may be a dict, a list of `(label, value)`, or a single string. - `progress`: `max_wait_ms`, `quiet_ms` (pure observer, sends no keys; returns `timeout` rather than a false complete if it never settles). - `wizard`: `advance_key` (default `Enter`), `max_steps`. Each intent item is `{"pattern": name, "value": intent}`, `{"keys": [...]}`, or `None`/`{}` for auto (classify + best recipe, fallback `advance_key`). Recipes fail loud on bad params (empty/mistyped intent raises `ValueError`) rather than doing something silently wrong. ### Add a NEW pattern Drop one module in `patterns/recipes/` — pkgutil auto-imports it and `@register` wires it into `all_patterns`/`classify`. A module that fails to import is logged (stderr + `registry.load_errors()`) and skipped; it never breaks the rest of the catalog. Avoid import-time side effects that can't fail-soft. Contract (`patterns/base.py`): - Subclass `Pattern`; set `name` (lowercase, unique), `description`, `tags`. - `matches(self, snapshot) -> float` (0..1): cheap, **no side effects**. Bind signals to the CURSOR ROW and cell attributes (`selected`, `menu_items`, `status_bar`) — NOT bare body text, since prompts and `[y/N]` strings also echo in scrollback. Use the helpers `Pattern.cursor_row_text(snap)`, `Pattern.last_rows_text(snap, n)`, `Pattern.visible_text(snap)`, `Pattern.rx(pat, text, flags)`. - `drive(self, session, intent=None, **kw) -> PatternResult`: perceive→act→wait→confirm. NEVER blind-`sleep`; always re-snapshot after acting; always bound waits (`wait_stable`/`wait_for` with a hard ceiling). Return `PatternResult(ok, detail, snapshot=, data={...})` where `ok` is asserted from the screen. - Decorate with `@patterns.registry.register` above the class. ```python from smartcli_core import PtySession, Snapshot from patterns.base import Pattern, PatternResult from patterns import registry @registry.register class TabBar(Pattern): name = "tabbar" description = "a top tab bar; switch tabs with the arrow/Tab keys." tags = ("tabs", "navigation") def matches(self, snapshot: Snapshot) -> float: row0 = next((e[1] for e in snapshot.lines if e != "..." and e[0] == 0), "") return 0.8 if self.rx(r"\[.+\]\s+\[.+\]", row0) else 0.0 def drive(self, session: PtySession, intent=None, **kw) -> PatternResult: before = session.snapshot() session.send_keys(["Right"]) session.wait_stable(quiet_ms=120, max_wait_ms=1500) after = session.snapshot() moved = after.cursor != before.cursor return PatternResult(ok=moved, detail="switched tab" if moved else "no move", snapshot=after, data={"moved": moved}) ``` Fault isolation is verified: a recipe module that crashes at import leaves all other recipes registered, and `classify()` is safe on a degenerate snapshot. ## Worked example — driving the Python REPL (persistent session) ``` # 1. start -> capture the id it prints $ python skills/drive-tui/scripts/tui.py start --cmd "python" --cols 80 --rows 24 s21576_90197 # 2. PERCEIVE the first prompt (wait-regex, generous timeout for the ~3s banner gap) $ python skills/drive-tui/scripts/tui.py wait-regex --id s21576_90197 ">>> " --timeout-ms 15000 # matched=True alive=True [screen 24x80] cursor=r3c4 selected=r3[0:80]">>>"(cursor_line) 0 | Python 3.14.6 ... on win32 2 | Type "help", "copyright", "credits" or "license" for more information. 3 | >>> # 3. DECIDE: it's a prompt. ACT: send an expression. WAIT + CONFIRM. $ python skills/drive-tui/scripts/tui.py send-line --id s21576_90197 "print(6*7)" $ python skills/drive-tui/scripts/tui.py wait --id s21576_90197 [screen 24x80] cursor=r5c4 ... 3 | >>> print(6*7) 4 | 42 5 | >>> # 4. done -> exit cleanly, then close the session $ python skills/drive-tui/scripts/tui.py send-line --id s21576_90197 "exit()" $ python skills/drive-tui/scripts/tui.py close --id s21576_90197 closed s21576_90197 ```