# Omniverse UI Inspector

> Use this skill when the user asks to inspect, screenshot, automate, click, type into, or drive a running ovui/ovui-widgets application through the ovui-inspect FastAPI inspector. The skill provides an importable `ovuiinspect` module, an `ovui-inspect` CLI, and a screenshot-first mouse/keyboard workflow for controlling ovui apps launched with PYTHONPATH.

- Skill: `nvidia-omniverse-ovui/omniverse-ui-inspector` (Agent Skill, multi-file: 13 files)
- Install (CLI): `npx skillmds@latest add nvidia-omniverse-ovui/omniverse-ui-inspector`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-omniverse-ovui/omniverse-ui-inspector/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- License: LicenseRef-NVIDIA
- Author: NVIDIA-Omniverse (https://skillmd.com/u/nvidia-omniverse-ovui)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/nvidia-omniverse-ovui/omniverse-ui-inspector

---


# ovui Inspector

Drive standalone ovui applications through the `ovui-inspect` FastAPI server.

## Quick Start

The tracked skill root in this repository is `skills/omniverse-ui-inspector`. Refer to
it from any clone with the repo-relative path; do not bake in machine-specific
absolute paths.

Launch the app with the skill root on `PYTHONPATH`. Set `OVUI_REPO` (or any
equivalent variable) to the local checkout of the ovui repository:

```bash
PYTHONPATH="${OVUI_REPO}/skills/omniverse-ui-inspector:${PYTHONPATH:-}" \
  python -m ovui_widgets.app.headless --width 1280 --height 720
```

For a windowed app, use:

```bash
PYTHONPATH="${OVUI_REPO}/skills/omniverse-ui-inspector:${PYTHONPATH:-}" \
  python -m ovui_widgets.app
```

The app imports `ovuiinspect` during startup. Importing the module starts a
localhost FastAPI server in a background thread and the app drains inspector
input actions on the ovui frame loop.

When using Codex or another command runner, keep the app command alive as a
managed long-running foreground session, or launch it with an explicit
detached-process wrapper that records the exact PID. Do not run the app through
a command invocation that has a default timeout and then continue sending
inspector commands after that invocation exits. If the app process is killed by
the runner, follow-up keyboard commands will fail with connection refused even
though the key is supported.

Defaults:

- Host: `127.0.0.1`
- Port: `9910`
- Override with `OVUIINSPECT_HOST` and `OVUIINSPECT_PORT`
- Application state and checkpoint endpoints disabled by default; enable them
  for state-guided QA with `OVUIINSPECT_ENABLE_STATE=1`
- Python execution endpoint disabled by default; enable with `OVUIINSPECT_ENABLE_EXECUTE=1`

## Interface Priority

1. `skills/omniverse-ui-inspector/scripts/ovui-inspect <command>` from the repo root is the preferred CLI.
2. `python skills/omniverse-ui-inspector/scripts/ovui-inspect.py <command>` is equivalent.
3. Raw HTTP against `http://127.0.0.1:9910` is for debugging.

Always wait for health after launch. Use the repo-relative path from the
current working directory or prefix it with the local repo root variable:

```bash
skills/omniverse-ui-inspector/scripts/ovui-inspect wait --timeout 60
skills/omniverse-ui-inspector/scripts/ovui-inspect health
```

For automated QA, also record the listening PID before the first UI action and
check it again if a command reports connection refused:

```bash
ss -tlnp sport = :9910
ps -p <exact-pid> -o pid,stat,cmd
```

## Screenshot-First Workflow

Use the same loop as Kit Inspector, corrected for ovui:

1. Take a screenshot.
2. Inspect the screenshot and identify the target coordinate.
3. Perform one mouse or keyboard action.
4. Take another screenshot and verify what changed.
5. Repeat.

Do not guess coordinates. Do not batch a full workflow between screenshots.
Strict screenshot-first evidence chooses every action coordinate from the
latest screenshot. If a workflow instead takes target geometry from `/state`,
it is state-guided UI evidence even when it records screenshots before and
after the real mouse or keyboard action. In a strict run, optional application
state may verify the result after the action, but it must not choose the target.

```bash
scripts/ovui-inspect screenshot --out proof/01_before.png
scripts/ovui-inspect move 240 32
scripts/ovui-inspect screenshot --out proof/02_hover.png
scripts/ovui-inspect click 240 32
scripts/ovui-inspect screenshot --out proof/03_after_click.png
```

## Command Reference

Core commands:

- `health` / `status`: check server, app attachment, queue depth, and opt-in settings.
- `state [--out FILE]`: capture the attached application's read-only scene and
  interaction state on the ovui frame loop. It requires
  `OVUIINSPECT_ENABLE_STATE=1`. Use it after a strict Inspector input action to
  check the active provider's scene state (native OVStage for the `ovstage`
  provider; USD stage state only when the OpenUSD provider is selected) against
  the provider-neutral adapter and UI views, or label geometry-guided use as
  state-guided QA; never use it to mutate UI.
- `wait --timeout 60`: wait until `/health` responds.
- `screenshot --out file.png`: capture the full ovui framebuffer.
- `move X Y`: move the injected ovui cursor.
- `click X Y [--button left|right|middle] [--double]`: click in screenshot coordinates.
- `drag X1 Y1 X2 Y2 [--button left|right|middle] [--duration SEC] [--steps N] [--modifiers ctrl,shift]`: drag over frame-stepped positions. Modifiers are held for the complete press/move/release gesture; `--duration` is converted to movement samples by the server and `--steps` wins when both are provided.
- `scroll up|down|left|right [--amount N] [--x X --y Y]`: scroll at the current or specified position.
- `type "text"`: send text input to the focused widget.
- `press KEY [--modifiers ctrl,shift]`: send Enter, Escape, Tab, arrows, Delete, letters, digits, or F-keys.
- `combo "CTRL+A"`: convenience wrapper around key press with modifiers.
- `execute "code"`: run Python on the ovui frame loop only when `OVUIINSPECT_ENABLE_EXECUTE=1`.
- `shutdown`: request `Application.request_exit()`.

Read `references/api-endpoints.md` for raw HTTP paths and payloads.

For evidence that must correlate a read-only application snapshot with the
exact screenshot request registered beside it, use the raw `POST /checkpoint`
endpoint described in `references/api-endpoints.md`. It returns frozen state,
request-scoped capture metadata, and the captured image without recapturing or
overwriting the initial evidence. This QA-only endpoint also requires
`OVUIINSPECT_ENABLE_STATE=1`. Exact native request correlation requires a built
ovui module with `_get_screenshot_result()`; the older schedule/poll fallback
keeps capture available but supplies only Inspector-local identity.

CLI commands exit `0` only when the HTTP request succeeds and the inspector
response does not report `success: false`. HTTP errors, validation failures,
disabled execute, timeouts, and connection failures exit nonzero while still
printing the JSON error payload.

## Operational Rules

- Bind to localhost unless the user explicitly asks for a remote bind.
- Enable `OVUIINSPECT_ENABLE_STATE=1` only when a QA run needs application
  state or correlated checkpoints; the payload can contain file paths, scene
  values, and interaction geometry.
- Prefer mouse and keyboard commands over `/execute`; use `/execute` only for setup tasks that cannot reasonably be done through the visible UI.
- If `/execute` is required, relaunch the app with `OVUIINSPECT_ENABLE_EXECUTE=1` and keep the code snippet small and auditable.
- If FastAPI or uvicorn is missing, `ovuiinspect` logs the startup error and leaves the ovui app running without inspector endpoints.
- If a command times out, take `status`, inspect the app log, then retry a smaller single action.

## Known Limits

- The Inspector covers full-application screenshots, mouse input, keyboard
  input, opt-in read-only application/scene snapshots, optional Python
  execution, and clean shutdown.
- It does not expose widget-tree inspection, named-window capture, menu-path clicking, dialog helpers, logs, or streaming execution yet.
- ovui input is injected through ovui's in-process APIs, not OS-level tools. This is correct for app UI QA, but native OS dialogs may need a different method.
- `combo` sends modifier key events through the same key-injection path as
  `press`. Visible shortcut effects still depend on the target widget's real
  keyboard focus. In the Stage filter, the headless remote text fallback appends
  printable text directly to the filter model, so do not use Ctrl+A plus `type`
  as proof that selected text replacement works there.