# MCP AI Wpoos Dictation App

> Operational guide for the NV oOS Dictation desktop companion app (Tauri v2 + Rust, examples/nvoos-dictate) — build/run on Windows/MSVC, the local-first engine track (parakeet/whisper/fastconformer/moonshine/silero-vad), the embedded remote-STT fallback, the Stage-2 polish provider track (disabled/openai_compat/nvoos plus local LM Studio and Ollama), the file/media decode seam (Symphonia 0.5), the live camera + local object-detection/tracking track (Proposals 073 + 075) with an opt-in VLM label-polish layer (Proposal 076), the CDP browser-control agent (feature agent) + FastConformer engine + NV oOS multi-SPA hub, model registry + verification harnesses, the plugins.updater boot-crash config rule, and the full test/clippy gate matrix. Use when building, extending, or debugging the dictation app, adding an engine/model, a polish provider, the browser agent, file/video transcription, camera/vision detection, or VLM relabeling.

- Skill: `nvdigitalsolutions/mcp-ai-wpoos-dictation-app` (Agent Skill)
- Install (CLI): `npx get-skill add nvdigitalsolutions/mcp-ai-wpoos-dictation-app`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvdigitalsolutions/mcp-ai-wpoos-dictation-app/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: Proprietary. See examples/nvoos-dictate/LICENSE.txt
- Author: nvdigitalsolutions (https://skillmd.com/u/nvdigitalsolutions)
- Updated: 2026-11-15
- Page: https://skillmd.com/skills/nvdigitalsolutions/mcp-ai-wpoos-dictation-app

---

# NV oOS Dictation App — Build, Engines, and Verification Guide

Operational guide for `examples/nvoos-dictate` (Proposal 067, PR #7011):
a local-first push-to-talk dictation + meeting transcription desktop app
with the NV oOS plugin as the optional AI-enrichment backend.

## When to use this skill

- Building/running the app on Windows, or debugging a boot failure
- Adding or swapping an STT engine, a model catalog entry, or VAD behavior
- Changing the deterministic cleanup pipeline (golden fixtures are the backstop)
- Wiring the Embedded addon remote-STT fallback or its auth
- Running the benchmark harness (`transcribe_file`) or the model downloader
- Embedding the hosted Pro SPA in the Assistant tab, or changing its reachability probe / CSP
- Producing a "complete" hand-off exe (the full feature union, not just STT flags)
- Continuing P2 (loopback + PTT matrix), P4/P5 ports, or distribution
- Verifying the macOS port from CI or a remote Mac (boot smoke, interactive SSH session, VNC options)

## Layout

```
examples/nvoos-dictate/
├── src/                  # React webview (Settings/History/Meetings/Overlay)
├── src-tauri/            # the Rust crate (manifest lives HERE, not at app root)
│   ├── src/{audio,stt,cleanup,nvoos,hotkey,paste,meetings,mcp,storage}
│   ├── tests/            # integration tests (must stay under src-tauri/tests!)
│   ├── examples/         # transcribe_file.rs (WER/e2e_ms) + download_model.rs
│   └── capabilities/ + icons/
├── docs/decisions/       # ADRs 0001 (paste strategy), 0002 (MCP transport), 0003 (engines)
└── e2e/                  # Playwright (webview UI)
```

## Build & run on Windows/MSVC

- Prerequisites: Rust + MSVC (VS 2022), Node 20+, WebView2 runtime
  (Win11 ships it), tauri-cli via npm (`npm run tauri`) — no cargo install needed.
- **MSYS shells**: every cargo command needs
  `export PATH="/c/Users/rasta/.cargo/bin:$PATH"` first; tauri-cli also needs
  cargo on PATH (`npm run tauri` fails with "cargo metadata: program not found"
  otherwise).
- Dev: `npm run tauri dev` (vite :1420). Debug exe:
  `npm run tauri -- build --debug --no-bundle` → `src-tauri/target/debug/nvoos-dictate.exe`.
  **The debug exe must include the feature for the engine the settings
  configure** — the default build is mock+embedded only, and a whisper
  config on a default build boots to "engine 'whisper' is not compiled
  into this build". Build with the features you need:
  `npm run tauri -- build --debug --no-bundle --features whisper,parakeet,silero-vad`
  (whisper needs CMake + libclang on PATH — see below; whisper+parakeet
  co-link because the vendored whisper-rs-sys compiles whisper.cpp with
  the static CRT /MT, matching sherpa-onnx's prebuilt libs).
  **"Complete" build** (every opt-in surface) is the union of all
  feature-gated modules — engines + vision + agent + mcp:
  `npm run tauri -- build --debug --no-bundle --features fastconformer,parakeet,whisper,silero-vad,camera,vision,agent,mcp`.
  **Trap — a build that omits a feature silently boots with that surface
  "not compiled", not an error.** `vision` (YOLO detection/tracking + camera
  preview + VLM label-polish) and `agent` (browser control) are each
  `cfg!`-gated and report `available() == false` when absent, so a
  "working" exe can still be missing them — always specify the FULL union
  for a hand-off exe, and verify with the string probe below.
  `moonshine` is intentionally **excluded** from the complete list (it is a
  documented spike — the streaming loop is unimplemented and returns
  `Unsupported`; including it only adds an unusable engine to the dropdown).
  Quick engine-load check without the GUI: build with `mcp` too and run
  `target/debug/nvoos-dictate.exe --mcp < /dev/null` — headless mode loads
  the configured engine and prints whisper.cpp's model-load log on
  success, or "engine unavailable in headless mode: …" on failure.
  **Debug-exe size is roughly 56 MB — that is correct, not a partial build.**
  `--debug --no-bundle` skips the installer/MSI + WebView2 bootstrap (the app
  runs on the system WebView2 Win11 ships), and the engine *weights* (whisper
  ~142 MB, parakeet ~600 MB, fastconformer ~162 MB) are downloaded at runtime
  into `%APPDATA%/nvoos-dictate/models/`, never compiled in. To **prove** a
  build has everything, scan the PE for feature strings (MSYS `strings` is
  usually missing; write a `.ps1` file and use
  `[System.IO.File]::ReadAllBytes` + a regex count — plain `powershell
  -Command` with `$var` gets mangled by MSYS, so always `-File x.ps1`):
  the presence of `fastconformer`/`parakeet`/`whisper`/`sherpa-onnx` means
  the engines are linked; `Assistant` means the SPA tab shipped.
  **Trap:** a plain `cargo build`/`cargo run` (debug profile) compiles with
  `devUrl` (`http://localhost:1420`) baked in, so the webview loads the
  Vite dev server at runtime and shows `ERR_CONNECTION_REFUSED` in the app
  body when nothing is serving there — and it silently OVERWRITES
  `target/debug/nvoos-dictate.exe`. Only `tauri build --debug --no-bundle`
  (or `npm run tauri dev` with vite running) produces a usable GUI exe;
  after any plain-cargo headless check, rebuild via tauri before handing
  the exe back.
- `generate_context!` embeds the frontend `dist` at compile time — run
  `npm run build` before any cargo step in CI.
- Boot smoke test: `cmd //c start "" <exe>` → `tasklist //FI "IMAGENAME eq
  nvoos-dictate.exe"` after ~10 s (process alive = boot OK); kill with
  `taskkill //IM nvoos-dictate.exe //F`. A running app locks the exe — kill
  it before `cargo build`/`test` or you get "failed to remove file" errors.
- **MSYS quoting traps:** `%APPDATA%`/`$_`/`$VAR` inside `cmd`/`powershell
  -Command` get mangled by the MSYS shell — write PowerShell to a `.ps1`
  file and run `powershell -NoProfile -ExecutionPolicy Bypass -File x.ps1`
  instead; paths like `C:\Users\…` stay literal. `cmdkey //list | findstr
  nvoos` (double slashes) shows what actually landed in Windows Credential
  Manager — the ground truth for "did the credential stick".

## Critical config rule: `plugins.updater` block

`tauri.conf.json` MUST contain a `plugins.updater` block (endpoints +
pubkey). Without it the app panics at boot:

```
error while running NV oOS Dictation: PluginInitialization("updater",
"Error deserializing 'plugins.updater' within your Tauri configuration:
invalid type: null, expected struct Config")
```

The endpoint uses the runtime placeholder `{{current_version}}` matching the
`nvoos-dictate-v*` tag convention; `pubkey: ""` = unsigned dev builds.

## Gate matrix (all verified green on the reference PC)

```bash
cd src-tauri
cargo fmt --check
cargo clippy --all-targets -- -D warnings                          # default
cargo clippy --all-targets --features mcp -- -D warnings
cargo clippy --all-targets --features camera -- -D warnings
cargo clippy --all-targets --features vision -- -D warnings
cargo test            # default (mock/embedded)
cargo test --features vision   # 94 lib (+9 vision/tracker/polish tests)
cargo test --features mcp       # +4 toolbox tests
cargo test --features parakeet  # 46 lib (+1 tar.bz2 extraction test)
```

Native feature compile checks: `parakeet`, `whisper`, `moonshine`,
`silero-vad`, and the combined `--features parakeet,silero-vad,moonshine`.
**whisper needs extra env** (below). CI (`build-nvoos-dictate.yml`) runs
fmt/clippy/test default + `mcp` on windows/ubuntu/macos, plus two macOS
runtime gates: a headless `--mcp` engine-load probe (rmcp's exit-1
"connection closed" on stdin EOF is expected teardown — fail only on
"engine unavailable") and a `boot-smoke` job (real debug `.app`, 30 s
boot-residency gate, `.app` zip + boot-log artifacts). The native engines
are local gates.

## Verifying the macOS port from a non-Mac machine

The GitHub macOS runner is the remote Mac. The full remote loop:

- **Every push:** `boot-smoke` in `build-nvoos-dictate.yml` builds the
  debug **universal** `.app` (`--target universal-apple-darwin` — Rust
  lipo's the binary; `build:helper` with `NVOOS_HELPER_ARCH=universal`
  builds both Swift slices in isolated scratch dirs, lipo's them over the
  `-universal-apple-darwin` externalBin stub, and self-checks with
  `lipo -info`) — production frontend + sidecar — boots **both slices**
  (native arm64, then the Intel slice via `arch -x86_64` under Rosetta 2,
  with a defensive `softwareupdate --install-rosetta` step and a teardown
  wait so the single-instance guard doesn't redirect the second boot),
  enforces 30 s boot residency per slice, hard-verifies the bundle is
  universal (`lipo -info` on the main binary AND the sidecar), and uploads
  the app zipped as artifact `nvoos-dictate-macos-app` (7 days) + boot
  evidence. **One download serves Apple Silicon AND Intel Macs** (macOS
  13+, incl. Sequoia 15.x) — an arm64-only app fails on Intel with "this
  application is not supported on this Mac". Release DMGs are a single
  universal DMG via publish-nvoos-dictate.yml; targeted builds land under
  `src-tauri/target/<triple>/debug/bundle/macos/`
  (`universal-apple-darwin` for universal), not `target/debug/`.
- **Interactive:** run the `NV oOS Dictation — macOS interactive session`
  workflow (`workflow_dispatch`) — builds the bundle, runs
  `scripts/macos-matrix-remote.sh` (the automatable subset of
  `docs/ptt-matrix-macos.md`: bundle surface, sidecar `--check`, boot
  residency, screenshots), uploads evidence, then holds the runner for
  SSH via tmate (up to 360 min, `limit-access-to-actor`).
- **Human matrix (TCC grants):** mic / Screen Recording / Accessibility
  clicks need a person at the GUI — GitHub-hosted runners have no VNC.
  Namespace runners support VNC into macOS (swap `runs-on` to a
  Namespace profile), or use a physical/rented Mac with
  `scripts/macos-matrix-remote.sh`.
- **Signed + notarized DMG:** Apple Developer provisioning needs **no
  Mac** — openssl CSR → Developer ID Application cert at
  developer.apple.com → p12 (base64) → repo secrets (`APPLE_CERTIFICATE`,
  `APPLE_CERTIFICATE_PASSWORD`, `APPLE_SIGNING_IDENTITY`, `APPLE_ID`,
  `APPLE_PASSWORD`, `APPLE_TEAM_ID`) → tag `nvoos-dictate-v*`
  (`publish-nvoos-dictate.yml`).

Remote-Mac landscape (2026-10): **Cirrus CI shut down 2026-06-01**
(Cirrus Labs joined OpenAI) — do not recommend it. Docker/QEMU macOS
(`dockurr/macos`, `docker-osx`) needs a Linux KVM host and violates
Apple's EULA (DMCA takedown 2024) — not a path from Windows.

Platform pitfalls that already cost CI runs:

- **Tauri v2 keeps the Cargo binary name as the bundle executable** —
  `Contents/MacOS/nvoos-dictate`, NOT productName
  (`Contents/MacOS/NV oOS Dictation` does not exist). Discover it with
  `find "$APP/Contents/MacOS" -maxdepth 1 -type f -perm -u+x ! -name
  '*helper*' -print -quit` or read `CFBundleExecutable`.
- **`screencapture -x` produces no image on GitHub macOS runners** —
  treat it as best-effort evidence, never a hard gate (boot residency is
  the gate).

## Engine track (ADR 0003: local-first, embedded as fallback)

Ordering: mock → parakeet → fastconformer → whisper → moonshine → embedded.
The factory is `stt::create_engine(&EngineConfig, &EngineResources)` —
`EngineResources` carries `models_dir` plus nvoos secrets (`nvoos_site_url`,
`nvoos_credential`, `wp_username`, `wp_app_password`) so the `embedded`
engine can build auth without touching the keychain itself. Call sites:
`state.rs ensure_engine` (fetches secrets), `main.rs` headless +
`examples/*` use `EngineResources::local_only`.

### parakeet (sherpa-onnx, feature `parakeet`) — API VERIFIED 1.13.8

Use the crate's own doc example shape (the old stub had drift everywhere):

```rust
let mut config = OfflineRecognizerConfig::default();
config.model_config.transducer = OfflineTransducerModelConfig {
    encoder: Some(".../encoder.int8.onnx".into()),
    decoder: Some(".../decoder.int8.onnx".into()),
    joiner:  Some(".../joiner.int8.onnx".into()),
};
config.model_config.tokens     = Some(".../tokens.txt".into());
config.model_config.model_type = Some("nemo_transducer".into());
config.model_config.num_threads = n as i32;
config.model_config.provider   = Some("cpu".into());
config.decoding_method         = Some("greedy_search".into());
let recognizer = OfflineRecognizer::create(&config).ok_or(...)?;  // -> Option
let stream = recognizer.create_stream();               // infallible
stream.accept_waveform(sample_rate as i32, samples);   // &self, infallible
recognizer.decode(&stream);                            // infallible
stream.get_result().map(|r| r.text)                    // -> Option
```

Model archive: `sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2` extracts
a top-level dir named after the model id — **unpack into `models_dir` (the
parent)**, not `models_dir/{id}`. Files: `encoder.int8.onnx`,
`decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (v2 int8 = feature_dim
80 default; v3 uses 128). TDT limit ≈ 4–5 min → meetings must be chunked
(`meetings/chunker.rs`).

### fastconformer (sherpa-onnx, feature `fastconformer`) — API VERIFIED 1.13.8

NeMo FastConformer-CTC runs through the SAME `OfflineRecognizer` as parakeet,
but via the `nemo_ctc` config slot instead of `transducer`. The key API shapes:

```rust
let mut config = OfflineRecognizerConfig::default();
config.model_config.nemo_ctc = OfflineNemoEncDecCtcModelConfig {
    model: Some(".../model.int8.onnx".into()),   // the ONLY nemo_ctc field
};
config.model_config.tokens     = Some(".../tokens.txt".into());
config.model_config.model_type = Some("nemo_ctc".into());      // NOT nemo_transducer
config.model_config.num_threads = n as i32;
config.model_config.provider   = Some("cpu".into());
config.decoding_method         = Some("greedy_search".into());
// then the identical create_stream -> accept_waveform -> decode -> get_result flow.
```

Important API facts (verified against docs.rs sherpa-onnx 1.13.8):
- `OfflineModelConfig` has a `nemo_ctc: OfflineNemoEncDecCtcModelConfig` field
  whose only member is `model: Option<String>` — there is **no** separate
  encoder/decoder/joiner for CTC, just one ONNX file.
- `model_type = "nemo_ctc"`; `tokens`/`num_threads`/`provider` live on the
  PARENT `OfflineModelConfig`, not on the `nemo_ctc` sub-struct.

The official sherpa-onnx download for this family is **
`sherpa-onnx-nemo-ctc-en-conformer-large.tar.bz2`** (NGC's
`stt_en_conformer_ctc_large` — *not* the hyphenated
`stt-en-fastconformer-ctc-large` name; the NGC model IS a fastconformer
architecture). Archive contains `model.int8.onnx` (162 MB) + `tokens.txt` +
`test_wavs/`; unpack into `models_dir` (the parent) like parakeet. Registry
id is `sherpa-onnx-nemo-ctc-en-conformer-large`.

**Registry `is_installed()` is engine-aware:** the CTC archive uses
`model.int8.onnx` + `tokens.txt`, NOT the transducer
`encoder/decoder/joiner` trio — `model_registry.rs::is_installed` branches on
`spec.engine == "fastconformer"` for the CTC layout. When adding any archive
engine with a differing file layout, extend that match arm.

**Archive extraction is NOT parakeet-only anymore.** `extract_tar_bz2` /
`install_archive` / `make_writable_recursive` were gated `#[cfg(feature =
"parakeet")]` but are now `#[cfg(any(feature = "parakeet", feature =
"fastconformer"))]` (and the `download` inline block mirrors it). Any new
archive-based engine must be added to that `any(...)` list or its download
will hit the "archive models require the … feature" error.

### whisper (whisper-rs 0.16, feature `whisper`) — API VERIFIED

- `full_n_segments() -> c_int` (NOT a Result); segments via
  `state.get_segment(i) -> Option<WhisperSegment>` +
  `seg.to_str() -> Result<&str>` (the old `full_get_segment_text` is gone).
- **MSVC Release-config trap (the "portable exe takes forever" bug):** with
  the VS generator, whisper.cpp's `Release` CMake configuration maps to an
  unoptimized vcxproj (empty `<Optimization>` → /Od) — base.en decodes at
  RTF ~5 instead of ~0.3, so dictation "hangs" for tens of seconds while
  parakeet stays fast. The app therefore wires
  `whisper = ["dep:whisper-rs", "whisper-rs-sys/force-debug"]` — a direct
  optional `whisper-rs-sys` dep (unified with the `[patch.crates-io]`
  entry) forwards the vendor's `force-debug` feature, which compiles
  whisper.cpp with the RelWithDebInfo profile (verified fast on MSVC) even
  in release builds. Do not remove that wiring; verify speed with
  `cargo run --release --features whisper --example transcribe_file --
  <wav> --engine whisper --model base.en` (expect RTF < 1).
- Windows build prerequisites: **CMake on PATH** (VS-bundled CMake at
  `C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`
  works) + **libclang** (`winget install LLVM.LLVM` → set
  `LIBCLANG_PATH=/c/Program Files/LLVM/bin`).
- `WHISPER_DONT_GENERATE_BINDINGS=1` is **Linux-only** — the checked-in
  bindings are glibc-specific and fail to compile on MSVC (`_IO_FILE` size
  errors). Always use real bindgen on Windows.
- The vendored `vendor/whisper-rs-sys/whisper.cpp/` tree is **trimmed** (no
  `bindings/`); its `CMakeLists.txt` has the upstream
  `configure_file(bindings/javascript/package-tmpl.json …)` line REMOVED.
  Never restore that line without restoring `bindings/javascript/` too — a
  fresh build dir (new feature-set hash, CI clean checkout, or after
  `cargo clean`) re-copies the trimmed tree into OUT_DIR and CMake configure
  fails. A stale pre-trim OUT_DIR masks the bug locally, which is why this
  only surfaces on fresh builds. If whisper-rs-sys CMake errors appear,
  `rm -rf target/{debug,release}/build/whisper-rs-sys-*` and rebuild.

### ort / moonshine / silero-vad (features `moonshine`, `silero-vad`)

- **ort pin**: `ort = "=2.0.0-rc.9"` + `ort-sys = "=2.0.0-rc.9"` (direct).
  Without the ort-sys pin, the caret range drifts to rc.13 whose bindings
  break the rc.9 high-level API (233 errors), and rc.13's `download-binaries`
  build script demands a TLS feature rc.9 doesn't expose. Add `dep:ort-sys`
  to both `moonshine` and `silero-vad` features.
- Moonshine engine is a documented spike (`Unsupported` error) — the
  streaming ONNX loop is intentionally unimplemented.
- **silero-vad-rs 0.1.2 real API**: `SileroVAD::new(path)` +
  `model.process_chunk(&arr.view(), 16000) -> Result<Array1<f32>>` with
  exactly **512-sample chunks @ 16 kHz** (32 ms). The adapter in
  `audio/vad.rs` buffers partial input and mirrors the EnergyVad hangover
  semantics. Model: `silero_vad.onnx` from sherpa-onnx releases, catalogued
  as engine `vad` (UI shows `vad` entries alongside the selected engine).

### embedded (remote fallback, always compiled — no feature flag)

**Settings → NV oOS "Test connection" contract:** `GET
{site}/wp-json/mcp-ai/v1/assistants` with `Authorization: Bearer
<credential>` — 200/201 = "connected; credential accepted", 401/403 =
"site reached, credential rejected", anything else = transport/status
error surfaced verbatim. The credential must be issued on the site first
(`wp mcp-ai assistant create …` + `wp mcp-ai credential issue <id>
--porcelain`, or the admin UI) — a stored-but-invalid credential shows
401, a missing one errors before any network call (`nvoos_client()`
requires both site URL and keychain credential).

**Windows credential storage (the "I stored it but it did not stick" bug):**
the app does NOT use the `keyring` crate on Windows — keyring 3.6.3 hard-
codes `CRED_PERSIST_ENTERPRISE`, which from an unelevated app writes
"successfully" into a namespace the app cannot read back (nothing appears
in `cmdkey /list` either). `storage/win_cred.rs`
(`WindowsCredentialStore`) calls `CredReadW`/`CredWriteW`/`CredDeleteW`
directly with `CRED_PERSIST_LOCAL_MACHINE` (machine-local, survives
reboots, visible in Credential Manager — the persistence git credential
manager uses), keeping keyring's `{user}.{service}` target convention and
UTF-16LE blob format for compatibility. keyring remains the backend on
macOS/Linux. Every IPC secret write (`persist_secret_verified`) verifies
the read-back before reporting success. Diagnostics: `cargo run --example
keychain_probe` (set/read-back/delete); visibility check while held:
`cmdkey /list | findstr nvoos`.

Contract: `POST {site}/wp-json/mcp-ai/v1/embedded/transcribe` with
`{"audio": "<base64 16-bit PCM WAV>", "language": "en"}`; server limit
10 MB, 30 req/min, 60 s timeout (Embedded addon → Gemma 4 audio endpoint).
Auth: **WP application password → Basic** (satisfies the route's
`is_user_logged_in()`; stored in keychain via `set_wp_app_password` IPC);
assistant credential → Bearer requires the addon `permission_callback`
tweak (follow-up PR). WAV encoder: `audio/wav.rs::encode_f32_to_wav`.
Contract tests in `stt/embedded.rs` (endpoint/body/parse/auth) — no network.

## Model registry conventions

`stt/model_registry.rs`: pinned `ModelSpec` catalogue (`engine`, `id`, url,
optional sha256, `archive`, `file_name` override for non-`.bin` files).
Rules: streaming SHA-256 (never read models into memory), atomic rename,
measured-hash logging when unpinned, tar.bz2 extraction behind the
`parakeet` feature. Downloads live in
`%APPDATA%/nvoos-dictate/models/` (GUI: Settings → Engine & Models).

Two guardrails added after the "parakeet model 'base.en' not found" wedge:

- **`validate_pairing(engine, model)`** rejects catalogue-known cross-engine
  pairings (a whisper id under parakeet) BEFORE settings are persisted —
  called from `state::update_settings` and `create_engine`. Engines without
  catalogue entries (mock/embedded) and unknown ids always pass: the shipped
  default settings store the parakeet id under the `mock` engine and MUST
  keep saving. The Settings UI model field is a dropdown filtered by engine,
  so a mismatch cannot be typed anymore; custom ids still work via
  config.json.
- **`download(spec, dir, force, progress)`** — `force` restarts from zero
  instead of resuming the `.download` partial (resume can never repair a
  corrupt file). The Settings UI shows a **Re-download** button on installed
  models. `download_model` IPC re-runs `ensure_engine` after install so the
  fresh model is usable without an app restart.

Re-download hardening: archive installs extract into a `.staging` dir and
atomically swap into place via `install_archive` — never extract over a live
install. Both the fresh tree and the previous install get a
`make_writable_recursive` pass (Windows `FILE_ATTRIBUTE_READONLY` clear,
Unix owner-write) because tar extraction preserves archive metadata and can
leave read-only model files that break later truncating overwrites.
Non-archive renames get `make_writable(dest)` first. The `download_model`
IPC **drops the resident engine before** replacing files (a loaded engine
can hold model files open/mapped on Windows) and re-creates it after.
**The actual root cause of the persistent "Download failed: I/O error:
creating or truncating a file requires write or append access" was NOT an
OS error — it is `OpenOptions` client-side validation** (`ErrorKind::
InvalidInput`): the tmp-file open had `.create(true).truncate(!resuming)`
with `.append(resuming)` but NO write access mode, so every fresh/forced
(non-resuming) download failed before touching disk, while resumed
downloads worked (append implies write). Fix: `.write(!resuming)` added;
the error now also carries the failing path. Regression tests:
`reinstall_over_readonly_previous_install_succeeds` and
`truncating_open_requires_write_access` (both feature `parakeet` for the
first; the second runs everywhere).

## Verification harnesses

```bash
# Registry-driven download (same path the GUI uses)
cargo run --features parakeet --example download_model -- parakeet sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8
cargo run --example download_model -- whisper tiny.en

# Live-capture probe: does a fresh stream on a device actually deliver?
# (distinguishes OS/device problems from app-side capture bugs)
cargo run --example mic_probe              # default device
cargo run --example mic_probe -- "ROG EYE S"  # device whose name contains ...

# Live-capture integration test on real hardware (ignored in CI)
cargo test -- --ignored live_capture_delivers_samples

# WER + e2e_ms + RTF against a speech file (sidecar .txt = reference)
cargo run --features parakeet --example transcribe_file -- ../speech.wav --engine parakeet --model sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8
```

- Speech samples without a mic: PowerShell SAPI TTS —
  `Add-Type -AssemblyName System.Speech; $s.Speak(...)` →
  `SetOutputToWaveFile(...)`.
- Reference-PC numbers (debug build, 4.27 s sample): parakeet 1187 ms
  (RTF 0.28), whisper tiny.en 1095 ms (WER 7.7%). Both beat the 1.5 s
  key-up→paste budget; release is faster.

## Known Rust pitfalls (this codebase)

- `cpal::Stream` is `!Send + !Sync` on WASAPI — streams must live on
  detached worker threads; only Send handles (ring buffer Arcs, channels,
  atomics) may enter `AppState`. Do not regress this.
  **And the stream must be HELD by the worker** — `build_stream` returns the
  live `cpal::Stream` alongside the ring, and the worker binds it for its
  lifetime. The original code dropped the stream at the end of
  `build_stream` (the "hold it forever" comment was aspirational): dropping
  a playing stream stops capture, so the ring NEVER filled — every dictation
  reported "no speech detected" on perfectly healthy mics. The
  `MicCapture` health counters (`callback_count()`, `stall_ms()`, stream
  errors) surface this on the Diagnostics page, `restart_microphone` IPC
  rebuilds capture without an app restart, and the ignored integration test
  `live_capture_delivers_samples` (`cargo test -- --ignored`) reproduces
  the failure on real hardware (ignored in CI — no audio device).
- **Bare expression paths do not resolve sibling modules declared in a
  parent module** (proven with a minimal repro): inside
  `stt/engine.rs`, `parakeet::X` fails while `crate::stt::parakeet::X`
  works — always use crate-qualified paths for cfg-gated siblings.
- **Hotkey swallow discipline (Windows hook)**: an armed PTT press must
  consume BOTH the key-down and the matching key-up (`SWALLOWED_DOWN`
  tracks the pair; auto-repeat downs stay swallowed). Swallowing only the
  key-up while the down passes through leaves Windows believing RAlt is
  permanently held — the "keyboard breaks, Ctrl+Alt+Del fixes it" stuck-Alt
  failure. Non-armed events (AltGr combos) always pass through via
  `ptt_pressed_alone`. **Exit discipline**: if the app shuts down with a
  swallow pending (Alt+F4 close while RAlt is held is the classic repro),
  the hook thread injects a synthetic key-up via `SendInput` AFTER
  `UnhookWindowsHookEx` (`release_pending_swallow` / `take_pending_swallow`
  bookkeeping) so the modifier is never left stuck for the session. The
  hook tests serialize on a shared `TEST_LOCK` because the hook state lives
  in process-global statics.
- **VAD gating (dictation)**: the energy gate is adaptive per window —
  `max(0.02, min(10th-percentile frame RMS × 6, 0.15))` (WebRTC-style noise
  gating); silero keeps its probability default. The VAD only trims silence
  edges — it is never the final gate: `vad_empty_fallback` transcribes the
  whole window whenever the peak exceeds 0.004 (false negatives must never
  drop user audio; muted/unplugged mics peak at ~0 and still error). Every
  attempt records `LastDictationStats` (window/utterance ms, RMS/peak, gate,
  outcome) surfaced by `get_diagnostics`.
- **Settings saves that fail validation must not persist**: `update_settings`
  runs `validate_pairing` BEFORE `settings.save()` so a broken combo can
  never wedge the next boot. If the engine still fails to load after a valid
  save (missing model files), the IPC error says exactly that — settings
  ARE saved in that case; only the engine swap failed.

## PTT / hotkey stopped working — runbook

When "Right Alt doesn't trigger dictation anymore" is reported, **do not
blame the most recent code first**: the hook path (`hotkey/win_hook.rs`,
`dictation.rs`) is deliberately isolated (vision/polish/engine changes never
reference it), so a git diff that only touches `vision/`, `cleanup/`,
`state.rs`, `ipc.rs`, `lib.rs`, or frontend files *cannot* have broken PTT.
Verify with `git diff <base>...HEAD -- src/hotkey/` — empty = hook code
untouched. Then work this checklist in order:

1. **Instance check.** `tasklist //FI "IMAGENAME eq nvoos-dictate.exe"` —
   multiple rows mean a stale instance is holding the global keyboard hook or
   the single-instance guard redirected focus. Kill all
   (`taskkill //IM nvoos-dictate.exe //F`) and relaunch ONE fresh instance;
   the hook reinstalls clean. This session's "broken RAlt" was exactly this —
   many build/kill/relaunch cycles, not a code regression.
2. **Config check.** `cat "$APPDATA/nvoos-dictate/config.json"` (or
   Settings → Hotkey) — confirm `hotkey.ptt_key` is `RAlt` (Windows default;
   macOS uses `toggle` mode + `ptt_combo`). A leftover `toggle`/combo-only
   config on Windows means RAlt *hold* was never armed.
3. **Hook health (Diagnostics → Push-to-talk hotkey), the decisive signal:**
   - **Low-level listener** `running` (green) vs `not running` —
     `not running` = the WH_KEYBOARD_LL hook failed to install or the thread
     died; the `GetAsyncKeyState` fallback may still cover basic hold-PTT.
   - **Listener heartbeat age** — small (~0–500 ms) = alive; large/growing =
     the thread stalled or died.
   - **Listener callback invocations** — must increment while you tap RAlt;
     flat means the hook isn't receiving key events (another app grabbed the
     keyboard, or the hook wasn't installed).
4. **Engine, not hook.** If the listener is `running` AND callbacks fire but
   nothing pastes, the trigger is fine — the failure is downstream. Check
   Settings → Engine: the configured engine/model (e.g. `whisper/base.en`)
   must be **downloaded** (Diagnostics → Speech engine flags `not installed`).
   A valid-but-missing model errors on transcription, not on trigger.
5. **Stuck-Alt recovery.** If the keyboard itself "broke" (RAlt held,
   Ctrl+Alt+Del fixes), that's the swallowed key-up path (see the swallow
   discipline pitfall above) — close the app cleanly so `release_pending_swallow`
   injects the synthetic key-up, or press RAlt once more to clear it.


## Cleanup pipeline (golden backstop)

Order matters: whitespace → vocabulary → numbers → fillers → capitalize →
punctuate → per_app. Any rule change must keep `src-tauri/tests/cleanup_golden.rs`
(17 cases + `meeting1` fixture) green or be an intentional, reviewed change.
The fixture previously said "So we need…" but the leading-marker rule strips
`so` after filler removal — "We need…" is the correct expectation.

## Camera & live vision (Proposal 073 + 075)

Two opt-in local sources on the webcam, both feature-gated and off by default:

- **Camera preview (073)** — `camera/` module, `nokhwa` frames only, gated
  behind the `camera` feature. `nokhwa::Camera` is `!Sync` so it lives on a
  dedicated `nvoos-camera` worker thread; only `Send` handles (the
  `PreviewFrame` ring `Arc<Mutex<_>>`, atomics) cross into `AppState`.
  Camera *audio* is **not** nokhwa — a webcam mic is an ordinary `cpal` input
  device and transcribes through the existing mic path. `PreviewFrame` is
  downsampled to ≤320 px wide for the preview. IPC: `list_camera_devices`,
  `camera_available`, `start_camera`, `stop_camera`, `get_preview_frame`.
- **Object detection + tracking (075)** — `vision/` module, behind the
  `vision` feature (which implies `camera`: `vision = ["camera", "dep:ort",
  "dep:ort-sys", "dep:ndarray16"]`). Runs a YOLO ONNX detector (`yolov11n`
  default, `yolov8n` fallback) via the pinned `ort` rc.9, decodes + NMS-filters
  `[1, 84, 8400]` → source-pixel `Detection`s, then a dependency-free SORT
  tracker assigns stable ids + a click-to-select pin. **No facial
  recognition / identity** — bright-line non-goal, enforced in tool/UI copy.

`vision/` layout mirrors `camera` for feature stability:
`mod.rs` (always compiled: `Detection`, `available()`) + `detect.rs`
(letterbox + `ort` + NMS; **both** a `#[cfg(feature = "vision")]` real body
and a stub), `tracker.rs` (pure SORT, always compiled + unit-tested),
`models.rs` (vision ONNX catalogue: `yolov11n`/`yolov8n`, SHA-256 + streaming
+ atomic install, mirroring `stt::model_registry`), and `controller.rs`
(`VisionController` — detector+tracker on a worker, snapshot + pin channel
handoff). `ort` rc.9 has **no** crate-root `Session`/`Value` re-export — use
`ort::session::Session`, `ort::value::Value`, and positional `ort::inputs!`
(`inputs![Value::from_array(tensor)?]`, NOT the named form). `ort` rc.9 maps
`ndarray` **0.16** while the app pins `ndarray` 0.15 for `silero-vad` — use the
`ndarray16` alias (already in `Cargo.toml`).

`Settings.vision` gains `enabled`, `detector`, `confidence`, `iou_threshold`,
`track_objects`, `hand_model` (hand model + ByteTrack are deferred M5
follow-ups). `StatusInfo` adds `vision_available/running/detector/hand_model`;
`get_diagnostics` adds a `vision` block (availability, running, model install
state, fps-equivalent `frames`, `last_infer_ms`, `active_tracks`,
`track_events`, `errors`). IPC commands: `vision_available`, `start_vision`,
`stop_vision`, `get_detections`, `vision_select_track(id)`, `get_vision_models`,
`download_vision_model`. The Home "Camera preview" panel draws detections on a
`<canvas>` overlay with click-to-pin; the detector is started from the Home
panel and the model is downloaded from Diagnostics → Object detection (or
Settings → Vision).

The de-risk spike `examples/vision_probe.rs` (feature `vision`) proved
`yolov11n.onnx` (exported `opset=12`, `onnx` Python module required to
regenerate) runs on `ort` rc.9 with the confirmed `[1, 84, 8400]` output and
~49 ms CPU inference. Weights export: install `onnx` (`pip install onnx`),
download `yolo11n.pt` from
`https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.pt`
(note: the id is `yolo11n`, not `yolov11n`), then
`python -c "from ultralytics import YOLO; YOLO('yolo11n.pt').export(format='onnx', opset=12, imgsz=640)"`.
No `onnxslim` = the simplifier step logs a non-fatal warning.

**Vision label polish (Proposal 076):** the stock `yolov11n` is a
closed-vocabulary 80-class COCO model, so it mislabels arbitrary objects
(a pen → "knife"). The opt-in VLM polish layer fixes this: when
`Settings.vision.polish_enabled` is on, the detect loop crops each
low-confidence box, PNG-encodes it as a base64 data-URI, and sends a
**multimodal** OpenAI-compatible request (image + text) through the *same*
`LocalOpenAiProvider` transport used by text polish (`local_openai.rs`). The
returned `{label, confidence}` replaces the COCO label (badged `VLM` in the
overlay). Implementation: `vision/polish.rs` (`VisionPolisher` + `prepare_request`
/`resolve`, crop→base64 PNG), `openai_shape::{build_vision_body, parse_label_response,
vision_polish_prompt}`, `LocalOpenAiProvider::describe_crop`, and a dedicated
`nvoos-vision-polish` thread (owns a tokio runtime) so the ~500 ms VLM
round-trip never stalls the 15 fps detect loop. Default model
`qwen/qwen2.5-vl-7b` (the reference VLM for structured-JSON grounding; verified
live on the reference PC — it returns **fenced** JSON, which
`parse_label_response` tolerates). The no-facial-recognition bright line is in
the system prompt. `VisionDiagnostics` gained `polish_enabled/provder/model/calls/results`.

## Stage-2 polish providers (local LLM cleanup)

Stage-2 is the optional LLM polish that runs **after** Stage-1 deterministic
rules, gated by `cleanup.wait_for_llm_ms` (ADR 0001). Provider dispatch lives
in `cleanup/providers/mod.rs::create()` — `disabled` | `openai_compat` |
`lm_studio` | `ollama` | `nvoos`. Each provider receives text + target-app
category only (never audio); failure always falls back to Stage-1 text and
can never block a paste.

**Provider architecture (refactor from this session):**

- `openai_shape.rs` owns the provider-agnostic OpenAI `/v1/chat/completions`
  shape: `system_prompt_for(category)`, `build_chat_body(model: Option<&str>,
  text, category, temperature)` (omits the `model` field entirely when `None`),
  and `parse_chat_response` (defensive string-or-segment-array content).
- `local_openai.rs` owns the shared **local**-server behaviour
  (`LocalOpenAiProvider` parameterized by a `LocalKind { name, id }`): the
  local-http-vs-remote-https invariant, optional bearer auth, `GET /v1/models`
  discovery, `ttft_ms`/`total_ms` timing, `load_timeout_ms` cold-load budget
  enforcement, and actionable error mapping. LM Studio and Ollama are thin
  `LocalKind` wrappers over it — they share ~95% of their logic.
- `lm_studio.rs` (`LmStudioProvider`) and `ollama.rs` (`OllamaProvider`) supply
  only their identity (`name`/`id`) and read their own keychain token
  (`KEY_LM_STUDIO_TOKEN`, `KEY_OLLAMA_TOKEN`). `id()` returns `lm_studio` /
  `ollama` so transcript/history labelling stays correct.

**Key facts that shaped the design:**

- **Model identifiers are discovered, not typed.** Both local servers expose
  `GET /v1/models` → `{ data: [{ id, … }] }`; the `id` is what goes back on
  `chat/completions.model` (an LM Studio "model identifier", an Ollama model
  name like `llama3.2` — NOT a GGUF filename). The Settings tabs surface a
  discovered-model dropdown + auto-select the single loaded model.
- **Auth defaults to none.** The local server needs no token (any non-empty
  `Authorization: Bearer …` is accepted as a placeholder). The optional token
  is really for a remote/headless server and lives in the OS keychain — never
  in `config.json` (the `Settings` struct holds no secrets).
- **Single-loaded-model behaviour:** if exactly one model is loaded, the server
  serves it regardless of the `model` field; omitting `model` when the config
  is empty is the correct default.
- **Two distinct timeouts:** `cleanup.wait_for_llm_ms` bounds the *whole*
  polish (dictation.rs); `load_timeout_ms` (per local provider) is a separate
  `tokio::time::timeout` budget for a Just-In-Time cold model load, plus a 60 s
  soft reqwest-client default so a hung server can't block forever.
- **`temperature` is provider-specific:** `0.2` for `openai_compat` (cloud),
  `0.1` for local models (they drift more; cleanup is deterministic editing).
- **Polish latency:** `PolishMeta` carries `ttft_ms`/`total_ms` (local providers
  report them; `nvoos`/`openai_compat` leave `None`); `LastDictationStats`
  surfaces them as `polish_ttft_ms`/`polish_total_ms`, shown on the Diagnostics
  page under "Polish latency (TTFT / total)".

**Settings plumbing per local provider:** config block (`LmStudioConfig` /
`OllamaConfig` with `base_url`, `model`, `load_timeout_ms`) wired into
`Settings`; keychain key + `has_*_token()` + `*_provider()` on `AppState`;
three IPC commands each (`test_*_connection`, `set_*_token`, `clear_*_token`)
registered in `lib.rs::run`; a Settings tab + provider `<option>` in
`Settings.tsx`; a credential row in `Diagnostics.tsx`. Default base URLs:
LM Studio `http://localhost:1234/v1`, Ollama `http://localhost:11434/v1`.

**Pitfall (this session):** adding a field to `PolishMeta` (a struct with other
literal constructions) breaks `nvoos/types.rs::extract_polish_meta`, which
builds it field-by-field — use `..PolishMeta::default()` there. Also, a
`reqwest::Client` in a struct means it can't trivially `#[derive(Debug)]`, so
`unwrap_err()` on a `Result<Provider, _>` won't compile — match on it instead.

The design rationale is captured in
`examples/nvoos-dictate/docs/lm-studio-integration-plan.md`.

## File / media decode seam (Symphonia 0.5)

File transcription (`Transcribe file…`, `util/decode.rs::decode_file`) decodes
mp3/m4a/wav/flac/ogg to mono f32 PCM via Symphonia and feeds the resident STT
engine through `ipc::transcribe_file`. Key facts:

- **Symphonia 0

…(truncated)
