NV oOS Dictation App — Build, Engines, and Verification Guide
Operational guide for examples/nvoos-dictate (Proposal 067, PR #7011):
a local-first push-to-talk dictation + meeting transcription desktop app
with the NV oOS plugin as the optional AI-enrichment backend.
When to use this skill
- Building/running the app on Windows, or debugging a boot failure
- Adding or swapping an STT engine, a model catalog entry, or VAD behavior
- Changing the deterministic cleanup pipeline (golden fixtures are the backstop)
- Wiring the Embedded addon remote-STT fallback or its auth
- Running the benchmark harness (
transcribe_file) or the model downloader
- Embedding the hosted Pro SPA in the Assistant tab, or changing its reachability probe / CSP
- Producing a "complete" hand-off exe (the full feature union, not just STT flags)
- Continuing P2 (loopback + PTT matrix), P4/P5 ports, or distribution
- Verifying the macOS port from CI or a remote Mac (boot smoke, interactive SSH session, VNC options)
Layout
examples/nvoos-dictate/
├── src/ # React webview (Settings/History/Meetings/Overlay)
├── src-tauri/ # the Rust crate (manifest lives HERE, not at app root)
│ ├── src/{audio,stt,cleanup,nvoos,hotkey,paste,meetings,mcp,storage}
│ ├── tests/ # integration tests (must stay under src-tauri/tests!)
│ ├── examples/ # transcribe_file.rs (WER/e2e_ms) + download_model.rs
│ └── capabilities/ + icons/
├── docs/decisions/ # ADRs 0001 (paste strategy), 0002 (MCP transport), 0003 (engines)
└── e2e/ # Playwright (webview UI)
Build & run on Windows/MSVC
- Prerequisites: Rust + MSVC (VS 2022), Node 20+, WebView2 runtime
(Win11 ships it), tauri-cli via npm (
npm run tauri) — no cargo install needed.
- MSYS shells: every cargo command needs
export PATH="/c/Users/rasta/.cargo/bin:$PATH" first; tauri-cli also needs
cargo on PATH (npm run tauri fails with "cargo metadata: program not found"
otherwise).
- Dev:
npm run tauri dev (vite :1420). Debug exe:
npm run tauri -- build --debug --no-bundle → src-tauri/target/debug/nvoos-dictate.exe.
The debug exe must include the feature for the engine the settings
configure — the default build is mock+embedded only, and a whisper
config on a default build boots to "engine 'whisper' is not compiled
into this build". Build with the features you need:
npm run tauri -- build --debug --no-bundle --features whisper,parakeet,silero-vad
(whisper needs CMake + libclang on PATH — see below; whisper+parakeet
co-link because the vendored whisper-rs-sys compiles whisper.cpp with
the static CRT /MT, matching sherpa-onnx's prebuilt libs).
"Complete" build (every opt-in surface) is the union of all
feature-gated modules — engines + vision + agent + mcp:
npm run tauri -- build --debug --no-bundle --features fastconformer,parakeet,whisper,silero-vad,camera,vision,agent,mcp.
Trap — a build that omits a feature silently boots with that surface
"not compiled", not an error. vision (YOLO detection/tracking + camera
preview + VLM label-polish) and agent (browser control) are each
cfg!-gated and report available() == false when absent, so a
"working" exe can still be missing them — always specify the FULL union
for a hand-off exe, and verify with the string probe below.
moonshine is intentionally excluded from the complete list (it is a
documented spike — the streaming loop is unimplemented and returns
Unsupported; including it only adds an unusable engine to the dropdown).
Quick engine-load check without the GUI: build with mcp too and run
target/debug/nvoos-dictate.exe --mcp < /dev/null — headless mode loads
the configured engine and prints whisper.cpp's model-load log on
success, or "engine unavailable in headless mode: …" on failure.
Debug-exe size is roughly 56 MB — that is correct, not a partial build.
--debug --no-bundle skips the installer/MSI + WebView2 bootstrap (the app
runs on the system WebView2 Win11 ships), and the engine weights (whisper
~142 MB, parakeet ~600 MB, fastconformer ~162 MB) are downloaded at runtime
into %APPDATA%/nvoos-dictate/models/, never compiled in. To prove a
build has everything, scan the PE for feature strings (MSYS strings is
usually missing; write a .ps1 file and use
[System.IO.File]::ReadAllBytes + a regex count — plain powershell -Command with $var gets mangled by MSYS, so always -File x.ps1):
the presence of fastconformer/parakeet/whisper/sherpa-onnx means
the engines are linked; Assistant means the SPA tab shipped.
Trap: a plain cargo build/cargo run (debug profile) compiles with
devUrl (http://localhost:1420) baked in, so the webview loads the
Vite dev server at runtime and shows ERR_CONNECTION_REFUSED in the app
body when nothing is serving there — and it silently OVERWRITES
target/debug/nvoos-dictate.exe. Only tauri build --debug --no-bundle
(or npm run tauri dev with vite running) produces a usable GUI exe;
after any plain-cargo headless check, rebuild via tauri before handing
the exe back.
generate_context! embeds the frontend dist at compile time — run
npm run build before any cargo step in CI.
- Boot smoke test:
cmd //c start "" <exe> → tasklist //FI "IMAGENAME eq nvoos-dictate.exe" after ~10 s (process alive = boot OK); kill with
taskkill //IM nvoos-dictate.exe //F. A running app locks the exe — kill
it before cargo build/test or you get "failed to remove file" errors.
- MSYS quoting traps:
%APPDATA%/$_/$VAR inside cmd/powershell -Command get mangled by the MSYS shell — write PowerShell to a .ps1
file and run powershell -NoProfile -ExecutionPolicy Bypass -File x.ps1
instead; paths like C:\Users\… stay literal. cmdkey //list | findstr nvoos (double slashes) shows what actually landed in Windows Credential
Manager — the ground truth for "did the credential stick".
Critical config rule: plugins.updater block
tauri.conf.json MUST contain a plugins.updater block (endpoints +
pubkey). Without it the app panics at boot:
error while running NV oOS Dictation: PluginInitialization("updater",
"Error deserializing 'plugins.updater' within your Tauri configuration:
invalid type: null, expected struct Config")
The endpoint uses the runtime placeholder {{current_version}} matching the
nvoos-dictate-v* tag convention; pubkey: "" = unsigned dev builds.
Gate matrix (all verified green on the reference PC)
cd src-tauri
cargo fmt --check
cargo clippy --all-targets -- -D warnings # default
cargo clippy --all-targets --features mcp -- -D warnings
cargo clippy --all-targets --features camera -- -D warnings
cargo clippy --all-targets --features vision -- -D warnings
cargo test # default (mock/embedded)
cargo test --features vision # 94 lib (+9 vision/tracker/polish tests)
cargo test --features mcp # +4 toolbox tests
cargo test --features parakeet # 46 lib (+1 tar.bz2 extraction test)
Native feature compile checks: parakeet, whisper, moonshine,
silero-vad, and the combined --features parakeet,silero-vad,moonshine.
whisper needs extra env (below). CI (build-nvoos-dictate.yml) runs
fmt/clippy/test default + mcp on windows/ubuntu/macos, plus two macOS
runtime gates: a headless --mcp engine-load probe (rmcp's exit-1
"connection closed" on stdin EOF is expected teardown — fail only on
"engine unavailable") and a boot-smoke job (real debug .app, 30 s
boot-residency gate, .app zip + boot-log artifacts). The native engines
are local gates.
Verifying the macOS port from a non-Mac machine
The GitHub macOS runner is the remote Mac. The full remote loop:
- Every push:
boot-smoke in build-nvoos-dictate.yml builds the
debug universal .app (--target universal-apple-darwin — Rust
lipo's the binary; build:helper with NVOOS_HELPER_ARCH=universal
builds both Swift slices in isolated scratch dirs, lipo's them over the
-universal-apple-darwin externalBin stub, and self-checks with
lipo -info) — production frontend + sidecar — boots both slices
(native arm64, then the Intel slice via arch -x86_64 under Rosetta 2,
with a defensive softwareupdate --install-rosetta step and a teardown
wait so the single-instance guard doesn't redirect the second boot),
enforces 30 s boot residency per slice, hard-verifies the bundle is
universal (lipo -info on the main binary AND the sidecar), and uploads
the app zipped as artifact nvoos-dictate-macos-app (7 days) + boot
evidence. One download serves Apple Silicon AND Intel Macs (macOS
13+, incl. Sequoia 15.x) — an arm64-only app fails on Intel with "this
application is not supported on this Mac". Release DMGs are a single
universal DMG via publish-nvoos-dictate.yml; targeted builds land under
src-tauri/target/<triple>/debug/bundle/macos/
(universal-apple-darwin for universal), not target/debug/.
- Interactive: run the
NV oOS Dictation — macOS interactive session
workflow (workflow_dispatch) — builds the bundle, runs
scripts/macos-matrix-remote.sh (the automatable subset of
docs/ptt-matrix-macos.md: bundle surface, sidecar --check, boot
residency, screenshots), uploads evidence, then holds the runner for
SSH via tmate (up to 360 min, limit-access-to-actor).
- Human matrix (TCC grants): mic / Screen Recording / Accessibility
clicks need a person at the GUI — GitHub-hosted runners have no VNC.
Namespace runners support VNC into macOS (swap
runs-on to a
Namespace profile), or use a physical/rented Mac with
scripts/macos-matrix-remote.sh.
- Signed + notarized DMG: Apple Developer provisioning needs no
Mac — openssl CSR → Developer ID Application cert at
developer.apple.com → p12 (base64) → repo secrets (
APPLE_CERTIFICATE,
APPLE_CERTIFICATE_PASSWORD, APPLE_SIGNING_IDENTITY, APPLE_ID,
APPLE_PASSWORD, APPLE_TEAM_ID) → tag nvoos-dictate-v*
(publish-nvoos-dictate.yml).
Remote-Mac landscape (2026-10): Cirrus CI shut down 2026-06-01
(Cirrus Labs joined OpenAI) — do not recommend it. Docker/QEMU macOS
(dockurr/macos, docker-osx) needs a Linux KVM host and violates
Apple's EULA (DMCA takedown 2024) — not a path from Windows.
Platform pitfalls that already cost CI runs:
- Tauri v2 keeps the Cargo binary name as the bundle executable —
Contents/MacOS/nvoos-dictate, NOT productName
(Contents/MacOS/NV oOS Dictation does not exist). Discover it with
find "$APP/Contents/MacOS" -maxdepth 1 -type f -perm -u+x ! -name '*helper*' -print -quit or read CFBundleExecutable.
screencapture -x produces no image on GitHub macOS runners —
treat it as best-effort evidence, never a hard gate (boot residency is
the gate).
Engine track (ADR 0003: local-first, embedded as fallback)
Ordering: mock → parakeet → fastconformer → whisper → moonshine → embedded.
The factory is stt::create_engine(&EngineConfig, &EngineResources) —
EngineResources carries models_dir plus nvoos secrets (nvoos_site_url,
nvoos_credential, wp_username, wp_app_password) so the embedded
engine can build auth without touching the keychain itself. Call sites:
state.rs ensure_engine (fetches secrets), main.rs headless +
examples/* use EngineResources::local_only.
parakeet (sherpa-onnx, feature parakeet) — API VERIFIED 1.13.8
Use the crate's own doc example shape (the old stub had drift everywhere):
let mut config = OfflineRecognizerConfig::default();
config.model_config.transducer = OfflineTransducerModelConfig {
encoder: Some(".../encoder.int8.onnx".into()),
decoder: Some(".../decoder.int8.onnx".into()),
joiner: Some(".../joiner.int8.onnx".into()),
};
config.model_config.tokens = Some(".../tokens.txt".into());
config.model_config.model_type = Some("nemo_transducer".into());
config.model_config.num_threads = n as i32;
config.model_config.provider = Some("cpu".into());
config.decoding_method = Some("greedy_search".into());
let recognizer = OfflineRecognizer::create(&config).ok_or(...)?; // -> Option
let stream = recognizer.create_stream(); // infallible
stream.accept_waveform(sample_rate as i32, samples); // &self, infallible
recognizer.decode(&stream); // infallible
stream.get_result().map(|r| r.text) // -> Option
Model archive: sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2 extracts
a top-level dir named after the model id — unpack into models_dir (the
parent), not models_dir/{id}. Files: encoder.int8.onnx,
decoder.int8.onnx, joiner.int8.onnx, tokens.txt (v2 int8 = feature_dim
80 default; v3 uses 128). TDT limit ≈ 4–5 min → meetings must be chunked
(meetings/chunker.rs).
fastconformer (sherpa-onnx, feature fastconformer) — API VERIFIED 1.13.8
NeMo FastConformer-CTC runs through the SAME OfflineRecognizer as parakeet,
but via the nemo_ctc config slot instead of transducer. The key API shapes:
let mut config = OfflineRecognizerConfig::default();
config.model_config.nemo_ctc = OfflineNemoEncDecCtcModelConfig {
model: Some(".../model.int8.onnx".into()), // the ONLY nemo_ctc field
};
config.model_config.tokens = Some(".../tokens.txt".into());
config.model_config.model_type = Some("nemo_ctc".into()); // NOT nemo_transducer
config.model_config.num_threads = n as i32;
config.model_config.provider = Some("cpu".into());
config.decoding_method = Some("greedy_search".into());
// then the identical create_stream -> accept_waveform -> decode -> get_result flow.
Important API facts (verified against docs.rs sherpa-onnx 1.13.8):
OfflineModelConfig has a nemo_ctc: OfflineNemoEncDecCtcModelConfig field
whose only member is model: Option<String> — there is no separate
encoder/decoder/joiner for CTC, just one ONNX file.
model_type = "nemo_ctc"; tokens/num_threads/provider live on the
PARENT OfflineModelConfig, not on the nemo_ctc sub-struct.
The official sherpa-onnx download for this family is **
sherpa-onnx-nemo-ctc-en-conformer-large.tar.bz2** (NGC's
stt_en_conformer_ctc_large — not the hyphenated
stt-en-fastconformer-ctc-large name; the NGC model IS a fastconformer
architecture). Archive contains model.int8.onnx (162 MB) + tokens.txt +
test_wavs/; unpack into models_dir (the parent) like parakeet. Registry
id is sherpa-onnx-nemo-ctc-en-conformer-large.
Registry is_installed() is engine-aware: the CTC archive uses
model.int8.onnx + tokens.txt, NOT the transducer
encoder/decoder/joiner trio — model_registry.rs::is_installed branches on
spec.engine == "fastconformer" for the CTC layout. When adding any archive
engine with a differing file layout, extend that match arm.
Archive extraction is NOT parakeet-only anymore. extract_tar_bz2 /
install_archive / make_writable_recursive were gated #[cfg(feature = "parakeet")] but are now #[cfg(any(feature = "parakeet", feature = "fastconformer"))] (and the download inline block mirrors it). Any new
archive-based engine must be added to that any(...) list or its download
will hit the "archive models require the … feature" error.
whisper (whisper-rs 0.16, feature whisper) — API VERIFIED
full_n_segments() -> c_int (NOT a Result); segments via
state.get_segment(i) -> Option<WhisperSegment> +
seg.to_str() -> Result<&str> (the old full_get_segment_text is gone).
- MSVC Release-config trap (the "portable exe takes forever" bug): with
the VS generator, whisper.cpp's
Release CMake configuration maps to an
unoptimized vcxproj (empty <Optimization> → /Od) — base.en decodes at
RTF ~5 instead of ~0.3, so dictation "hangs" for tens of seconds while
parakeet stays fast. The app therefore wires
whisper = ["dep:whisper-rs", "whisper-rs-sys/force-debug"] — a direct
optional whisper-rs-sys dep (unified with the [patch.crates-io]
entry) forwards the vendor's force-debug feature, which compiles
whisper.cpp with the RelWithDebInfo profile (verified fast on MSVC) even
in release builds. Do not remove that wiring; verify speed with
cargo run --release --features whisper --example transcribe_file -- <wav> --engine whisper --model base.en (expect RTF < 1).
- Windows build prerequisites: CMake on PATH (VS-bundled CMake at
C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin
works) + libclang (winget install LLVM.LLVM → set
LIBCLANG_PATH=/c/Program Files/LLVM/bin).
WHISPER_DONT_GENERATE_BINDINGS=1 is Linux-only — the checked-in
bindings are glibc-specific and fail to compile on MSVC (_IO_FILE size
errors). Always use real bindgen on Windows.
- The vendored
vendor/whisper-rs-sys/whisper.cpp/ tree is trimmed (no
bindings/); its CMakeLists.txt has the upstream
configure_file(bindings/javascript/package-tmpl.json …) line REMOVED.
Never restore that line without restoring bindings/javascript/ too — a
fresh build dir (new feature-set hash, CI clean checkout, or after
cargo clean) re-copies the trimmed tree into OUT_DIR and CMake configure
fails. A stale pre-trim OUT_DIR masks the bug locally, which is why this
only surfaces on fresh builds. If whisper-rs-sys CMake errors appear,
rm -rf target/{debug,release}/build/whisper-rs-sys-* and rebuild.
ort / moonshine / silero-vad (features moonshine, silero-vad)
- ort pin:
ort = "=2.0.0-rc.9" + ort-sys = "=2.0.0-rc.9" (direct).
Without the ort-sys pin, the caret range drifts to rc.13 whose bindings
break the rc.9 high-level API (233 errors), and rc.13's download-binaries
build script demands a TLS feature rc.9 doesn't expose. Add dep:ort-sys
to both moonshine and silero-vad features.
- Moonshine engine is a documented spike (
Unsupported error) — the
streaming ONNX loop is intentionally unimplemented.
- silero-vad-rs 0.1.2 real API:
SileroVAD::new(path) +
model.process_chunk(&arr.view(), 16000) -> Result<Array1<f32>> with
exactly 512-sample chunks @ 16 kHz (32 ms). The adapter in
audio/vad.rs buffers partial input and mirrors the EnergyVad hangover
semantics. Model: silero_vad.onnx from sherpa-onnx releases, catalogued
as engine vad (UI shows vad entries alongside the selected engine).
embedded (remote fallback, always compiled — no feature flag)
Settings → NV oOS "Test connection" contract: GET {site}/wp-json/mcp-ai/v1/assistants with Authorization: Bearer <credential> — 200/201 = "connected; credential accepted", 401/403 =
"site reached, credential rejected", anything else = transport/status
error surfaced verbatim. The credential must be issued on the site first
(wp mcp-ai assistant create … + wp mcp-ai credential issue <id> --porcelain, or the admin UI) — a stored-but-invalid credential shows
401, a missing one errors before any network call (nvoos_client()
requires both site URL and keychain credential).
Windows credential storage (the "I stored it but it did not stick" bug):
the app does NOT use the keyring crate on Windows — keyring 3.6.3 hard-
codes CRED_PERSIST_ENTERPRISE, which from an unelevated app writes
"successfully" into a namespace the app cannot read back (nothing appears
in cmdkey /list either). storage/win_cred.rs
(WindowsCredentialStore) calls CredReadW/CredWriteW/CredDeleteW
directly with CRED_PERSIST_LOCAL_MACHINE (machine-local, survives
reboots, visible in Credential Manager — the persistence git credential
manager uses), keeping keyring's {user}.{service} target convention and
UTF-16LE blob format for compatibility. keyring remains the backend on
macOS/Linux. Every IPC secret write (persist_secret_verified) verifies
the read-back before reporting success. Diagnostics: cargo run --example keychain_probe (set/read-back/delete); visibility check while held:
cmdkey /list | findstr nvoos.
Contract: POST {site}/wp-json/mcp-ai/v1/embedded/transcribe with
{"audio": "<base64 16-bit PCM WAV>", "language": "en"}; server limit
10 MB, 30 req/min, 60 s timeout (Embedded addon → Gemma 4 audio endpoint).
Auth: WP application password → Basic (satisfies the route's
is_user_logged_in(); stored in keychain via set_wp_app_password IPC);
assistant credential → Bearer requires the addon permission_callback
tweak (follow-up PR). WAV encoder: audio/wav.rs::encode_f32_to_wav.
Contract tests in stt/embedded.rs (endpoint/body/parse/auth) — no network.
Model registry conventions
stt/model_registry.rs: pinned ModelSpec catalogue (engine, id, url,
optional sha256, archive, file_name override for non-.bin files).
Rules: streaming SHA-256 (never read models into memory), atomic rename,
measured-hash logging when unpinned, tar.bz2 extraction behind the
parakeet feature. Downloads live in
%APPDATA%/nvoos-dictate/models/ (GUI: Settings → Engine & Models).
Two guardrails added after the "parakeet model 'base.en' not found" wedge:
validate_pairing(engine, model) rejects catalogue-known cross-engine
pairings (a whisper id under parakeet) BEFORE settings are persisted —
called from state::update_settings and create_engine. Engines without
catalogue entries (mock/embedded) and unknown ids always pass: the shipped
default settings store the parakeet id under the mock engine and MUST
keep saving. The Settings UI model field is a dropdown filtered by engine,
so a mismatch cannot be typed anymore; custom ids still work via
config.json.
download(spec, dir, force, progress) — force restarts from zero
instead of resuming the .download partial (resume can never repair a
corrupt file). The Settings UI shows a Re-download button on installed
models. download_model IPC re-runs ensure_engine after install so the
fresh model is usable without an app restart.
Re-download hardening: archive installs extract into a .staging dir and
atomically swap into place via install_archive — never extract over a live
install. Both the fresh tree and the previous install get a
make_writable_recursive pass (Windows FILE_ATTRIBUTE_READONLY clear,
Unix owner-write) because tar extraction preserves archive metadata and can
leave read-only model files that break later truncating overwrites.
Non-archive renames get make_writable(dest) first. The download_model
IPC drops the resident engine before replacing files (a loaded engine
can hold model files open/mapped on Windows) and re-creates it after.
The actual root cause of the persistent "Download failed: I/O error:
creating or truncating a file requires write or append access" was NOT an
OS error — it is OpenOptions client-side validation (ErrorKind:: InvalidInput): the tmp-file open had .create(true).truncate(!resuming)
with .append(resuming) but NO write access mode, so every fresh/forced
(non-resuming) download failed before touching disk, while resumed
downloads worked (append implies write). Fix: .write(!resuming) added;
the error now also carries the failing path. Regression tests:
reinstall_over_readonly_previous_install_succeeds and
truncating_open_requires_write_access (both feature parakeet for the
first; the second runs everywhere).
Verification harnesses
# Registry-driven download (same path the GUI uses)
cargo run --features parakeet --example download_model -- parakeet sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8
cargo run --example download_model -- whisper tiny.en
# Live-capture probe: does a fresh stream on a device actually deliver?
# (distinguishes OS/device problems from app-side capture bugs)
cargo run --example mic_probe # default device
cargo run --example mic_probe -- "ROG EYE S" # device whose name contains ...
# Live-capture integration test on real hardware (ignored in CI)
cargo test -- --ignored live_capture_delivers_samples
# WER + e2e_ms + RTF against a speech file (sidecar .txt = reference)
cargo run --features parakeet --example transcribe_file -- ../speech.wav --engine parakeet --model sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8
- Speech samples without a mic: PowerShell SAPI TTS —
Add-Type -AssemblyName System.Speech; $s.Speak(...) →
SetOutputToWaveFile(...).
- Reference-PC numbers (debug build, 4.27 s sample): parakeet 1187 ms
(RTF 0.28), whisper tiny.en 1095 ms (WER 7.7%). Both beat the 1.5 s
key-up→paste budget; release is faster.
Known Rust pitfalls (this codebase)
cpal::Stream is !Send + !Sync on WASAPI — streams must live on
detached worker threads; only Send handles (ring buffer Arcs, channels,
atomics) may enter AppState. Do not regress this.
And the stream must be HELD by the worker — build_stream returns the
live cpal::Stream alongside the ring, and the worker binds it for its
lifetime. The original code dropped the stream at the end of
build_stream (the "hold it forever" comment was aspirational): dropping
a playing stream stops capture, so the ring NEVER filled — every dictation
reported "no speech detected" on perfectly healthy mics. The
MicCapture health counters (callback_count(), stall_ms(), stream
errors) surface this on the Diagnostics page, restart_microphone IPC
rebuilds capture without an app restart, and the ignored integration test
live_capture_delivers_samples (cargo test -- --ignored) reproduces
the failure on real hardware (ignored in CI — no audio device).
- Bare expression paths do not resolve sibling modules declared in a
parent module (proven with a minimal repro): inside
stt/engine.rs, parakeet::X fails while crate::stt::parakeet::X
works — always use crate-qualified paths for cfg-gated siblings.
- Hotkey swallow discipline (Windows hook): an armed PTT press must
consume BOTH the key-down and the matching key-up (
SWALLOWED_DOWN
tracks the pair; auto-repeat downs stay swallowed). Swallowing only the
key-up while the down passes through leaves Windows believing RAlt is
permanently held — the "keyboard breaks, Ctrl+Alt+Del fixes it" stuck-Alt
failure. Non-armed events (AltGr combos) always pass through via
ptt_pressed_alone. Exit discipline: if the app shuts down with a
swallow pending (Alt+F4 close while RAlt is held is the classic repro),
the hook thread injects a synthetic key-up via SendInput AFTER
UnhookWindowsHookEx (release_pending_swallow / take_pending_swallow
bookkeeping) so the modifier is never left stuck for the session. The
hook tests serialize on a shared TEST_LOCK because the hook state lives
in process-global statics.
- VAD gating (dictation): the energy gate is adaptive per window —
max(0.02, min(10th-percentile frame RMS × 6, 0.15)) (WebRTC-style noise
gating); silero keeps its probability default. The VAD only trims silence
edges — it is never the final gate: vad_empty_fallback transcribes the
whole window whenever the peak exceeds 0.004 (false negatives must never
drop user audio; muted/unplugged mics peak at ~0 and still error). Every
attempt records LastDictationStats (window/utterance ms, RMS/peak, gate,
outcome) surfaced by get_diagnostics.
- Settings saves that fail validation must not persist:
update_settings
runs validate_pairing BEFORE settings.save() so a broken combo can
never wedge the next boot. If the engine still fails to load after a valid
save (missing model files), the IPC error says exactly that — settings
ARE saved in that case; only the engine swap failed.
PTT / hotkey stopped working — runbook
When "Right Alt doesn't trigger dictation anymore" is reported, do not
blame the most recent code first: the hook path (hotkey/win_hook.rs,
dictation.rs) is deliberately isolated (vision/polish/engine changes never
reference it), so a git diff that only touches vision/, cleanup/,
state.rs, ipc.rs, lib.rs, or frontend files cannot have broken PTT.
Verify with git diff <base>...HEAD -- src/hotkey/ — empty = hook code
untouched. Then work this checklist in order:
- Instance check.
tasklist //FI "IMAGENAME eq nvoos-dictate.exe" —
multiple rows mean a stale instance is holding the global keyboard hook or
the single-instance guard redirected focus. Kill all
(taskkill //IM nvoos-dictate.exe //F) and relaunch ONE fresh instance;
the hook reinstalls clean. This session's "broken RAlt" was exactly this —
many build/kill/relaunch cycles, not a code regression.
- Config check.
cat "$APPDATA/nvoos-dictate/config.json" (or
Settings → Hotkey) — confirm hotkey.ptt_key is RAlt (Windows default;
macOS uses toggle mode + ptt_combo). A leftover toggle/combo-only
config on Windows means RAlt hold was never armed.
- Hook health (Diagnostics → Push-to-talk hotkey), the decisive signal:
- Low-level listener
running (green) vs not running —
not running = the WH_KEYBOARD_LL hook failed to install or the thread
died; the GetAsyncKeyState fallback may still cover basic hold-PTT.
- Listener heartbeat age — small (~0–500 ms) = alive; large/growing =
the thread stalled or died.
- Listener callback invocations — must increment while you tap RAlt;
flat means the hook isn't receiving key events (another app grabbed the
keyboard, or the hook wasn't installed).
- Engine, not hook. If the listener is
running AND callbacks fire but
nothing pastes, the trigger is fine — the failure is downstream. Check
Settings → Engine: the configured engine/model (e.g. whisper/base.en)
must be downloaded (Diagnostics → Speech engine flags not installed).
A valid-but-missing model errors on transcription, not on trigger.
- Stuck-Alt recovery. If the keyboard itself "broke" (RAlt held,
Ctrl+Alt+Del fixes), that's the swallowed key-up path (see the swallow
discipline pitfall above) — close the app cleanly so
release_pending_swallow
injects the synthetic key-up, or press RAlt once more to clear it.
Cleanup pipeline (golden backstop)
Order matters: whitespace → vocabulary → numbers → fillers → capitalize →
punctuate → per_app. Any rule change must keep src-tauri/tests/cleanup_golden.rs
(17 cases + meeting1 fixture) green or be an intentional, reviewed change.
The fixture previously said "So we need…" but the leading-marker rule strips
so after filler removal — "We need…" is the correct expectation.
Camera & live vision (Proposal 073 + 075)
Two opt-in local sources on the webcam, both feature-gated and off by default:
- Camera preview (073) —
camera/ module, nokhwa frames only, gated
behind the camera feature. nokhwa::Camera is !Sync so it lives on a
dedicated nvoos-camera worker thread; only Send handles (the
PreviewFrame ring Arc<Mutex<_>>, atomics) cross into AppState.
Camera audio is not nokhwa — a webcam mic is an ordinary cpal input
device and transcribes through the existing mic path. PreviewFrame is
downsampled to ≤320 px wide for the preview. IPC: list_camera_devices,
camera_available, start_camera, stop_camera, get_preview_frame.
- Object detection + tracking (075) —
vision/ module, behind the
vision feature (which implies camera: vision = ["camera", "dep:ort", "dep:ort-sys", "dep:ndarray16"]). Runs a YOLO ONNX detector (yolov11n
default, yolov8n fallback) via the pinned ort rc.9, decodes + NMS-filters
[1, 84, 8400] → source-pixel Detections, then a dependency-free SORT
tracker assigns stable ids + a click-to-select pin. No facial
recognition / identity — bright-line non-goal, enforced in tool/UI copy.
vision/ layout mirrors camera for feature stability:
mod.rs (always compiled: Detection, available()) + detect.rs
(letterbox + ort + NMS; both a #[cfg(feature = "vision")] real body
and a stub), tracker.rs (pure SORT, always compiled + unit-tested),
models.rs (vision ONNX catalogue: yolov11n/yolov8n, SHA-256 + streaming
- atomic install, mirroring
stt::model_registry), and controller.rs
(VisionController — detector+tracker on a worker, snapshot + pin channel
handoff). ort rc.9 has no crate-root Session/Value re-export — use
ort::session::Session, ort::value::Value, and positional ort::inputs!
(inputs![Value::from_array(tensor)?], NOT the named form). ort rc.9 maps
ndarray 0.16 while the app pins ndarray 0.15 for silero-vad — use the
ndarray16 alias (already in Cargo.toml).
Settings.vision gains enabled, detector, confidence, iou_threshold,
track_objects, hand_model (hand model + ByteTrack are deferred M5
follow-ups). StatusInfo adds vision_available/running/detector/hand_model;
get_diagnostics adds a vision block (availability, running, model install
state, fps-equivalent frames, last_infer_ms, active_tracks,
track_events, errors). IPC commands: vision_available, start_vision,
stop_vision, get_detections, vision_select_track(id), get_vision_models,
download_vision_model. The Home "Camera preview" panel draws detections on a
<canvas> overlay with click-to-pin; the detector is started from the Home
panel and the model is downloaded from Diagnostics → Object detection (or
Settings → Vision).
The de-risk spike examples/vision_probe.rs (feature vision) proved
yolov11n.onnx (exported opset=12, onnx Python module required to
regenerate) runs on ort rc.9 with the confirmed [1, 84, 8400] output and
~49 ms CPU inference. Weights export: install onnx (pip install onnx),
download yolo11n.pt from
https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.pt
(note: the id is yolo11n, not yolov11n), then
python -c "from ultralytics import YOLO; YOLO('yolo11n.pt').export(format='onnx', opset=12, imgsz=640)".
No onnxslim = the simplifier step logs a non-fatal warning.
Vision label polish (Proposal 076): the stock yolov11n is a
closed-vocabulary 80-class COCO model, so it mislabels arbitrary objects
(a pen → "knife"). The opt-in VLM polish layer fixes this: when
Settings.vision.polish_enabled is on, the detect loop crops each
low-confidence box, PNG-encodes it as a base64 data-URI, and sends a
multimodal OpenAI-compatible request (image + text) through the same
LocalOpenAiProvider transport used by text polish (local_openai.rs). The
returned {label, confidence} replaces the COCO label (badged VLM in the
overlay). Implementation: vision/polish.rs (VisionPolisher + prepare_request
/resolve, crop→base64 PNG), openai_shape::{build_vision_body, parse_label_response, vision_polish_prompt}, LocalOpenAiProvider::describe_crop, and a dedicated
nvoos-vision-polish thread (owns a tokio runtime) so the ~500 ms VLM
round-trip never stalls the 15 fps detect loop. Default model
qwen/qwen2.5-vl-7b (the reference VLM for structured-JSON grounding; verified
live on the reference PC — it returns fenced JSON, which
parse_label_response tolerates). The no-facial-recognition bright line is in
the system prompt. VisionDiagnostics gained polish_enabled/provder/model/calls/results.
Stage-2 polish providers (local LLM cleanup)
Stage-2 is the optional LLM polish that runs after Stage-1 deterministic
rules, gated by cleanup.wait_for_llm_ms (ADR 0001). Provider dispatch lives
in cleanup/providers/mod.rs::create() — disabled | openai_compat |
lm_studio | ollama | nvoos. Each provider receives text + target-app
category only (never audio); failure always falls back to Stage-1 text and
can never block a paste.
Provider architecture (refactor from this session):
openai_shape.rs owns the provider-agnostic OpenAI /v1/chat/completions
shape: system_prompt_for(category), build_chat_body(model: Option<&str>, text, category, temperature) (omits the model field entirely when None),
and parse_chat_response (defensive string-or-segment-array content).
local_openai.rs owns the shared local-server behaviour
(LocalOpenAiProvider parameterized by a LocalKind { name, id }): the
local-http-vs-remote-https invariant, optional bearer auth, GET /v1/models
discovery, ttft_ms/total_ms timing, load_timeout_ms cold-load budget
enforcement, and actionable error mapping. LM Studio and Ollama are thin
LocalKind wrappers over it — they share ~95% of their logic.
lm_studio.rs (LmStudioProvider) and ollama.rs (OllamaProvider) supply
only their identity (name/id) and read their own keychain token
(KEY_LM_STUDIO_TOKEN, KEY_OLLAMA_TOKEN). id() returns lm_studio /
ollama so transcript/history labelling stays correct.
Key facts that shaped the design:
- Model identifiers are discovered, not typed. Both local servers expose
GET /v1/models → { data: [{ id, … }] }; the id is what goes back on
chat/completions.model (an LM Studio "model identifier", an Ollama model
name like llama3.2 — NOT a GGUF filename). The Settings tabs surface a
discovered-model dropdown + auto-select the single loaded model.
- Auth defaults to none. The local server needs no token (any non-empty
Authorization: Bearer … is accepted as a placeholder). The optional token
is really for a remote/headless server and lives in the OS keychain — never
in config.json (the Settings struct holds no secrets).
- Single-loaded-model behaviour: if exactly one model is loaded, the server
serves it regardless of the
model field; omitting model when the config
is empty is the correct default.
- Two distinct timeouts:
cleanup.wait_for_llm_ms bounds the whole
polish (dictation.rs); load_timeout_ms (per local provider) is a separate
tokio::time::timeout budget for a Just-In-Time cold model load, plus a 60 s
soft reqwest-client default so a hung server can't block forever.
temperature is provider-specific: 0.2 for openai_compat (cloud),
0.1 for local models (they drift more; cleanup is deterministic editing).
- Polish latency:
PolishMeta carries ttft_ms/total_ms (local providers
report them; nvoos/openai_compat leave None); LastDictationStats
surfaces them as polish_ttft_ms/polish_total_ms, shown on the Diagnostics
page under "Polish latency (TTFT / total)".
Settings plumbing per local provider: config block (LmStudioConfig /
OllamaConfig with base_url, model, load_timeout_ms) wired into
Settings; keychain key + has_*_token() + *_provider() on AppState;
three IPC commands each (test_*_connection, set_*_token, clear_*_token)
registered in lib.rs::run; a Settings tab + provider <option> in
Settings.tsx; a credential row in Diagnostics.tsx. Default base URLs:
LM Studio http://localhost:1234/v1, Ollama http://localhost:11434/v1.
Pitfall (this session): adding a field to PolishMeta (a struct with other
literal constructions) breaks nvoos/types.rs::extract_polish_meta, which
builds it field-by-field — use ..PolishMeta::default() there. Also, a
reqwest::Client in a struct means it can't trivially #[derive(Debug)], so
unwrap_err() on a Result<Provider, _> won't compile — match on it instead.
The design rationale is captured in
examples/nvoos-dictate/docs/lm-studio-integration-plan.md.
File / media decode seam (Symphonia 0.5)
File transcription (Transcribe file…, util/decode.rs::decode_file) decodes
mp3/m4a/wav/flac/ogg to mono f32 PCM via Symphonia and feeds the resident STT
engine through ipc::transcribe_file. Key facts:
…(truncated)
1---2type: Skill3name: mcp-ai-wpoos-dictation-app4description: Operational guide for the NV oOS Dictation desktop companion app (Tauri v2 + Rust, examples/nvoos-dictate) — build/run on Windows/MSVC, the local-first engine track (parakeet/whisper/fastconformer/moonshine/silero-vad), the embedded remote-STT fallback, the Stage-2 polish provider track (disabled/openai_compat/nvoos plus local LM Studio and Ollama), the file/media decode seam (Symphonia 0.5), the live camera + local object-detection/tracking track (Proposals 073 + 075) with an opt-in VLM label-polish layer (Proposal 076), the CDP browser-control agent (feature agent) + FastConformer engine + NV oOS multi-SPA hub, model registry + verification harnesses, the plugins.updater boot-crash config rule, and the full test/clippy gate matrix. Use when building, extending, or debugging the dictation app, adding an engine/model, a polish provider, the browser agent, file/video transcription, camera/vision detection, or VLM relabeling.5license: Proprietary. See examples/nvoos-dictate/LICENSE.txt6metadata:7 source: https://skillmd.com/skills/nvdigitalsolutions/mcp-ai-wpoos-dictation-app8 app: nvoos-dictate9 app-version: "0.2.0"10 proposal: "067"11 pr: "7043"12 last-updated: "2026-10-14"13---1415# NV oOS Dictation App — Build, Engines, and Verification Guide1617Operational guide for `examples/nvoos-dictate` (Proposal 067, PR #7011):18a local-first push-to-talk dictation + meeting transcription desktop app19with the NV oOS plugin as the optional AI-enrichment backend.2021## When to use this skill2223- Building/running the app on Windows, or debugging a boot failure24- Adding or swapping an STT engine, a model catalog entry, or VAD behavior25- Changing the deterministic cleanup pipeline (golden fixtures are the backstop)26- Wiring the Embedded addon remote-STT fallback or its auth27- Running the benchmark harness (`transcribe_file`) or the model downloader28- Embedding the hosted Pro SPA in the Assistant tab, or changing its reachability probe / CSP29- Producing a "complete" hand-off exe (the full feature union, not just STT flags)30- Continuing P2 (loopback + PTT matrix), P4/P5 ports, or distribution31- Verifying the macOS port from CI or a remote Mac (boot smoke, interactive SSH session, VNC options)3233## Layout3435```36examples/nvoos-dictate/37├── src/ # React webview (Settings/History/Meetings/Overlay)38├── src-tauri/ # the Rust crate (manifest lives HERE, not at app root)39│ ├── src/{audio,stt,cleanup,nvoos,hotkey,paste,meetings,mcp,storage}40│ ├── tests/ # integration tests (must stay under src-tauri/tests!)41│ ├── examples/ # transcribe_file.rs (WER/e2e_ms) + download_model.rs42│ └── capabilities/ + icons/43├── docs/decisions/ # ADRs 0001 (paste strategy), 0002 (MCP transport), 0003 (engines)44└── e2e/ # Playwright (webview UI)45```4647## Build & run on Windows/MSVC4849- Prerequisites: Rust + MSVC (VS 2022), Node 20+, WebView2 runtime50 (Win11 ships it), tauri-cli via npm (`npm run tauri`) — no cargo install needed.51- **MSYS shells**: every cargo command needs52 `export PATH="/c/Users/rasta/.cargo/bin:$PATH"` first; tauri-cli also needs53 cargo on PATH (`npm run tauri` fails with "cargo metadata: program not found"54 otherwise).55- Dev: `npm run tauri dev` (vite :1420). Debug exe:56 `npm run tauri -- build --debug --no-bundle` → `src-tauri/target/debug/nvoos-dictate.exe`.57 **The debug exe must include the feature for the engine the settings58 configure** — the default build is mock+embedded only, and a whisper59 config on a default build boots to "engine 'whisper' is not compiled60 into this build". Build with the features you need:61 `npm run tauri -- build --debug --no-bundle --features whisper,parakeet,silero-vad`62 (whisper needs CMake + libclang on PATH — see below; whisper+parakeet63 co-link because the vendored whisper-rs-sys compiles whisper.cpp with64 the static CRT /MT, matching sherpa-onnx's prebuilt libs).65 **"Complete" build** (every opt-in surface) is the union of all66 feature-gated modules — engines + vision + agent + mcp:67 `npm run tauri -- build --debug --no-bundle --features fastconformer,parakeet,whisper,silero-vad,camera,vision,agent,mcp`.68 **Trap — a build that omits a feature silently boots with that surface69 "not compiled", not an error.** `vision` (YOLO detection/tracking + camera70 preview + VLM label-polish) and `agent` (browser control) are each71 `cfg!`-gated and report `available() == false` when absent, so a72 "working" exe can still be missing them — always specify the FULL union73 for a hand-off exe, and verify with the string probe below.74 `moonshine` is intentionally **excluded** from the complete list (it is a75 documented spike — the streaming loop is unimplemented and returns76 `Unsupported`; including it only adds an unusable engine to the dropdown).77 Quick engine-load check without the GUI: build with `mcp` too and run78 `target/debug/nvoos-dictate.exe --mcp < /dev/null` — headless mode loads79 the configured engine and prints whisper.cpp's model-load log on80 success, or "engine unavailable in headless mode: …" on failure.81 **Debug-exe size is roughly 56 MB — that is correct, not a partial build.**82 `--debug --no-bundle` skips the installer/MSI + WebView2 bootstrap (the app83 runs on the system WebView2 Win11 ships), and the engine *weights* (whisper84 ~142 MB, parakeet ~600 MB, fastconformer ~162 MB) are downloaded at runtime85 into `%APPDATA%/nvoos-dictate/models/`, never compiled in. To **prove** a86 build has everything, scan the PE for feature strings (MSYS `strings` is87 usually missing; write a `.ps1` file and use88 `[System.IO.File]::ReadAllBytes` + a regex count — plain `powershell89 -Command` with `$var` gets mangled by MSYS, so always `-File x.ps1`):90 the presence of `fastconformer`/`parakeet`/`whisper`/`sherpa-onnx` means91 the engines are linked; `Assistant` means the SPA tab shipped.92 **Trap:** a plain `cargo build`/`cargo run` (debug profile) compiles with93 `devUrl` (`http://localhost:1420`) baked in, so the webview loads the94 Vite dev server at runtime and shows `ERR_CONNECTION_REFUSED` in the app95 body when nothing is serving there — and it silently OVERWRITES96 `target/debug/nvoos-dictate.exe`. Only `tauri build --debug --no-bundle`97 (or `npm run tauri dev` with vite running) produces a usable GUI exe;98 after any plain-cargo headless check, rebuild via tauri before handing99 the exe back.100- `generate_context!` embeds the frontend `dist` at compile time — run101 `npm run build` before any cargo step in CI.102- Boot smoke test: `cmd //c start "" <exe>` → `tasklist //FI "IMAGENAME eq103 nvoos-dictate.exe"` after ~10 s (process alive = boot OK); kill with104 `taskkill //IM nvoos-dictate.exe //F`. A running app locks the exe — kill105 it before `cargo build`/`test` or you get "failed to remove file" errors.106- **MSYS quoting traps:** `%APPDATA%`/`$_`/`$VAR` inside `cmd`/`powershell107 -Command` get mangled by the MSYS shell — write PowerShell to a `.ps1`108 file and run `powershell -NoProfile -ExecutionPolicy Bypass -File x.ps1`109 instead; paths like `C:\Users\…` stay literal. `cmdkey //list | findstr110 nvoos` (double slashes) shows what actually landed in Windows Credential111 Manager — the ground truth for "did the credential stick".112113## Critical config rule: `plugins.updater` block114115`tauri.conf.json` MUST contain a `plugins.updater` block (endpoints +116pubkey). Without it the app panics at boot:117118```119error while running NV oOS Dictation: PluginInitialization("updater",120"Error deserializing 'plugins.updater' within your Tauri configuration:121invalid type: null, expected struct Config")122```123124The endpoint uses the runtime placeholder `{{current_version}}` matching the125`nvoos-dictate-v*` tag convention; `pubkey: ""` = unsigned dev builds.126127## Gate matrix (all verified green on the reference PC)128129```bash130cd src-tauri131cargo fmt --check132cargo clippy --all-targets -- -D warnings # default133cargo clippy --all-targets --features mcp -- -D warnings134cargo clippy --all-targets --features camera -- -D warnings135cargo clippy --all-targets --features vision -- -D warnings136cargo test # default (mock/embedded)137cargo test --features vision # 94 lib (+9 vision/tracker/polish tests)138cargo test --features mcp # +4 toolbox tests139cargo test --features parakeet # 46 lib (+1 tar.bz2 extraction test)140```141142Native feature compile checks: `parakeet`, `whisper`, `moonshine`,143`silero-vad`, and the combined `--features parakeet,silero-vad,moonshine`.144**whisper needs extra env** (below). CI (`build-nvoos-dictate.yml`) runs145fmt/clippy/test default + `mcp` on windows/ubuntu/macos, plus two macOS146runtime gates: a headless `--mcp` engine-load probe (rmcp's exit-1147"connection closed" on stdin EOF is expected teardown — fail only on148"engine unavailable") and a `boot-smoke` job (real debug `.app`, 30 s149boot-residency gate, `.app` zip + boot-log artifacts). The native engines150are local gates.151152## Verifying the macOS port from a non-Mac machine153154The GitHub macOS runner is the remote Mac. The full remote loop:155156- **Every push:** `boot-smoke` in `build-nvoos-dictate.yml` builds the157 debug **universal** `.app` (`--target universal-apple-darwin` — Rust158 lipo's the binary; `build:helper` with `NVOOS_HELPER_ARCH=universal`159 builds both Swift slices in isolated scratch dirs, lipo's them over the160 `-universal-apple-darwin` externalBin stub, and self-checks with161 `lipo -info`) — production frontend + sidecar — boots **both slices**162 (native arm64, then the Intel slice via `arch -x86_64` under Rosetta 2,163 with a defensive `softwareupdate --install-rosetta` step and a teardown164 wait so the single-instance guard doesn't redirect the second boot),165 enforces 30 s boot residency per slice, hard-verifies the bundle is166 universal (`lipo -info` on the main binary AND the sidecar), and uploads167 the app zipped as artifact `nvoos-dictate-macos-app` (7 days) + boot168 evidence. **One download serves Apple Silicon AND Intel Macs** (macOS169 13+, incl. Sequoia 15.x) — an arm64-only app fails on Intel with "this170 application is not supported on this Mac". Release DMGs are a single171 universal DMG via publish-nvoos-dictate.yml; targeted builds land under172 `src-tauri/target/<triple>/debug/bundle/macos/`173 (`universal-apple-darwin` for universal), not `target/debug/`.174- **Interactive:** run the `NV oOS Dictation — macOS interactive session`175 workflow (`workflow_dispatch`) — builds the bundle, runs176 `scripts/macos-matrix-remote.sh` (the automatable subset of177 `docs/ptt-matrix-macos.md`: bundle surface, sidecar `--check`, boot178 residency, screenshots), uploads evidence, then holds the runner for179 SSH via tmate (up to 360 min, `limit-access-to-actor`).180- **Human matrix (TCC grants):** mic / Screen Recording / Accessibility181 clicks need a person at the GUI — GitHub-hosted runners have no VNC.182 Namespace runners support VNC into macOS (swap `runs-on` to a183 Namespace profile), or use a physical/rented Mac with184 `scripts/macos-matrix-remote.sh`.185- **Signed + notarized DMG:** Apple Developer provisioning needs **no186 Mac** — openssl CSR → Developer ID Application cert at187 developer.apple.com → p12 (base64) → repo secrets (`APPLE_CERTIFICATE`,188 `APPLE_CERTIFICATE_PASSWORD`, `APPLE_SIGNING_IDENTITY`, `APPLE_ID`,189 `APPLE_PASSWORD`, `APPLE_TEAM_ID`) → tag `nvoos-dictate-v*`190 (`publish-nvoos-dictate.yml`).191192Remote-Mac landscape (2026-10): **Cirrus CI shut down 2026-06-01**193(Cirrus Labs joined OpenAI) — do not recommend it. Docker/QEMU macOS194(`dockurr/macos`, `docker-osx`) needs a Linux KVM host and violates195Apple's EULA (DMCA takedown 2024) — not a path from Windows.196197Platform pitfalls that already cost CI runs:198199- **Tauri v2 keeps the Cargo binary name as the bundle executable** —200 `Contents/MacOS/nvoos-dictate`, NOT productName201 (`Contents/MacOS/NV oOS Dictation` does not exist). Discover it with202 `find "$APP/Contents/MacOS" -maxdepth 1 -type f -perm -u+x ! -name203 '*helper*' -print -quit` or read `CFBundleExecutable`.204- **`screencapture -x` produces no image on GitHub macOS runners** —205 treat it as best-effort evidence, never a hard gate (boot residency is206 the gate).207208## Engine track (ADR 0003: local-first, embedded as fallback)209210Ordering: mock → parakeet → fastconformer → whisper → moonshine → embedded.211The factory is `stt::create_engine(&EngineConfig, &EngineResources)` —212`EngineResources` carries `models_dir` plus nvoos secrets (`nvoos_site_url`,213`nvoos_credential`, `wp_username`, `wp_app_password`) so the `embedded`214engine can build auth without touching the keychain itself. Call sites:215`state.rs ensure_engine` (fetches secrets), `main.rs` headless +216`examples/*` use `EngineResources::local_only`.217218### parakeet (sherpa-onnx, feature `parakeet`) — API VERIFIED 1.13.8219220Use the crate's own doc example shape (the old stub had drift everywhere):221222```rust223let mut config = OfflineRecognizerConfig::default();224config.model_config.transducer = OfflineTransducerModelConfig {225 encoder: Some(".../encoder.int8.onnx".into()),226 decoder: Some(".../decoder.int8.onnx".into()),227 joiner: Some(".../joiner.int8.onnx".into()),228};229config.model_config.tokens = Some(".../tokens.txt".into());230config.model_config.model_type = Some("nemo_transducer".into());231config.model_config.num_threads = n as i32;232config.model_config.provider = Some("cpu".into());233config.decoding_method = Some("greedy_search".into());234let recognizer = OfflineRecognizer::create(&config).ok_or(...)?; // -> Option235let stream = recognizer.create_stream(); // infallible236stream.accept_waveform(sample_rate as i32, samples); // &self, infallible237recognizer.decode(&stream); // infallible238stream.get_result().map(|r| r.text) // -> Option239```240241Model archive: `sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2` extracts242a top-level dir named after the model id — **unpack into `models_dir` (the243parent)**, not `models_dir/{id}`. Files: `encoder.int8.onnx`,244`decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (v2 int8 = feature_dim24580 default; v3 uses 128). TDT limit ≈ 4–5 min → meetings must be chunked246(`meetings/chunker.rs`).247248### fastconformer (sherpa-onnx, feature `fastconformer`) — API VERIFIED 1.13.8249250NeMo FastConformer-CTC runs through the SAME `OfflineRecognizer` as parakeet,251but via the `nemo_ctc` config slot instead of `transducer`. The key API shapes:252253```rust254let mut config = OfflineRecognizerConfig::default();255config.model_config.nemo_ctc = OfflineNemoEncDecCtcModelConfig {256 model: Some(".../model.int8.onnx".into()), // the ONLY nemo_ctc field257};258config.model_config.tokens = Some(".../tokens.txt".into());259config.model_config.model_type = Some("nemo_ctc".into()); // NOT nemo_transducer260config.model_config.num_threads = n as i32;261config.model_config.provider = Some("cpu".into());262config.decoding_method = Some("greedy_search".into());263// then the identical create_stream -> accept_waveform -> decode -> get_result flow.264```265266Important API facts (verified against docs.rs sherpa-onnx 1.13.8):267- `OfflineModelConfig` has a `nemo_ctc: OfflineNemoEncDecCtcModelConfig` field268 whose only member is `model: Option<String>` — there is **no** separate269 encoder/decoder/joiner for CTC, just one ONNX file.270- `model_type = "nemo_ctc"`; `tokens`/`num_threads`/`provider` live on the271 PARENT `OfflineModelConfig`, not on the `nemo_ctc` sub-struct.272273The official sherpa-onnx download for this family is **274`sherpa-onnx-nemo-ctc-en-conformer-large.tar.bz2`** (NGC's275`stt_en_conformer_ctc_large` — *not* the hyphenated276`stt-en-fastconformer-ctc-large` name; the NGC model IS a fastconformer277architecture). Archive contains `model.int8.onnx` (162 MB) + `tokens.txt` +278`test_wavs/`; unpack into `models_dir` (the parent) like parakeet. Registry279id is `sherpa-onnx-nemo-ctc-en-conformer-large`.280281**Registry `is_installed()` is engine-aware:** the CTC archive uses282`model.int8.onnx` + `tokens.txt`, NOT the transducer283`encoder/decoder/joiner` trio — `model_registry.rs::is_installed` branches on284`spec.engine == "fastconformer"` for the CTC layout. When adding any archive285engine with a differing file layout, extend that match arm.286287**Archive extraction is NOT parakeet-only anymore.** `extract_tar_bz2` /288`install_archive` / `make_writable_recursive` were gated `#[cfg(feature =289"parakeet")]` but are now `#[cfg(any(feature = "parakeet", feature =290"fastconformer"))]` (and the `download` inline block mirrors it). Any new291archive-based engine must be added to that `any(...)` list or its download292will hit the "archive models require the … feature" error.293294### whisper (whisper-rs 0.16, feature `whisper`) — API VERIFIED295296- `full_n_segments() -> c_int` (NOT a Result); segments via297 `state.get_segment(i) -> Option<WhisperSegment>` +298 `seg.to_str() -> Result<&str>` (the old `full_get_segment_text` is gone).299- **MSVC Release-config trap (the "portable exe takes forever" bug):** with300 the VS generator, whisper.cpp's `Release` CMake configuration maps to an301 unoptimized vcxproj (empty `<Optimization>` → /Od) — base.en decodes at302 RTF ~5 instead of ~0.3, so dictation "hangs" for tens of seconds while303 parakeet stays fast. The app therefore wires304 `whisper = ["dep:whisper-rs", "whisper-rs-sys/force-debug"]` — a direct305 optional `whisper-rs-sys` dep (unified with the `[patch.crates-io]`306 entry) forwards the vendor's `force-debug` feature, which compiles307 whisper.cpp with the RelWithDebInfo profile (verified fast on MSVC) even308 in release builds. Do not remove that wiring; verify speed with309 `cargo run --release --features whisper --example transcribe_file --310 <wav> --engine whisper --model base.en` (expect RTF < 1).311- Windows build prerequisites: **CMake on PATH** (VS-bundled CMake at312 `C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`313 works) + **libclang** (`winget install LLVM.LLVM` → set314 `LIBCLANG_PATH=/c/Program Files/LLVM/bin`).315- `WHISPER_DONT_GENERATE_BINDINGS=1` is **Linux-only** — the checked-in316 bindings are glibc-specific and fail to compile on MSVC (`_IO_FILE` size317 errors). Always use real bindgen on Windows.318- The vendored `vendor/whisper-rs-sys/whisper.cpp/` tree is **trimmed** (no319 `bindings/`); its `CMakeLists.txt` has the upstream320 `configure_file(bindings/javascript/package-tmpl.json …)` line REMOVED.321 Never restore that line without restoring `bindings/javascript/` too — a322 fresh build dir (new feature-set hash, CI clean checkout, or after323 `cargo clean`) re-copies the trimmed tree into OUT_DIR and CMake configure324 fails. A stale pre-trim OUT_DIR masks the bug locally, which is why this325 only surfaces on fresh builds. If whisper-rs-sys CMake errors appear,326 `rm -rf target/{debug,release}/build/whisper-rs-sys-*` and rebuild.327328### ort / moonshine / silero-vad (features `moonshine`, `silero-vad`)329330- **ort pin**: `ort = "=2.0.0-rc.9"` + `ort-sys = "=2.0.0-rc.9"` (direct).331 Without the ort-sys pin, the caret range drifts to rc.13 whose bindings332 break the rc.9 high-level API (233 errors), and rc.13's `download-binaries`333 build script demands a TLS feature rc.9 doesn't expose. Add `dep:ort-sys`334 to both `moonshine` and `silero-vad` features.335- Moonshine engine is a documented spike (`Unsupported` error) — the336 streaming ONNX loop is intentionally unimplemented.337- **silero-vad-rs 0.1.2 real API**: `SileroVAD::new(path)` +338 `model.process_chunk(&arr.view(), 16000) -> Result<Array1<f32>>` with339 exactly **512-sample chunks @ 16 kHz** (32 ms). The adapter in340 `audio/vad.rs` buffers partial input and mirrors the EnergyVad hangover341 semantics. Model: `silero_vad.onnx` from sherpa-onnx releases, catalogued342 as engine `vad` (UI shows `vad` entries alongside the selected engine).343344### embedded (remote fallback, always compiled — no feature flag)345346**Settings → NV oOS "Test connection" contract:** `GET347{site}/wp-json/mcp-ai/v1/assistants` with `Authorization: Bearer348<credential>` — 200/201 = "connected; credential accepted", 401/403 =349"site reached, credential rejected", anything else = transport/status350error surfaced verbatim. The credential must be issued on the site first351(`wp mcp-ai assistant create …` + `wp mcp-ai credential issue <id>352--porcelain`, or the admin UI) — a stored-but-invalid credential shows353401, a missing one errors before any network call (`nvoos_client()`354requires both site URL and keychain credential).355356**Windows credential storage (the "I stored it but it did not stick" bug):**357the app does NOT use the `keyring` crate on Windows — keyring 3.6.3 hard-358codes `CRED_PERSIST_ENTERPRISE`, which from an unelevated app writes359"successfully" into a namespace the app cannot read back (nothing appears360in `cmdkey /list` either). `storage/win_cred.rs`361(`WindowsCredentialStore`) calls `CredReadW`/`CredWriteW`/`CredDeleteW`362directly with `CRED_PERSIST_LOCAL_MACHINE` (machine-local, survives363reboots, visible in Credential Manager — the persistence git credential364manager uses), keeping keyring's `{user}.{service}` target convention and365UTF-16LE blob format for compatibility. keyring remains the backend on366macOS/Linux. Every IPC secret write (`persist_secret_verified`) verifies367the read-back before reporting success. Diagnostics: `cargo run --example368keychain_probe` (set/read-back/delete); visibility check while held:369`cmdkey /list | findstr nvoos`.370371Contract: `POST {site}/wp-json/mcp-ai/v1/embedded/transcribe` with372`{"audio": "<base64 16-bit PCM WAV>", "language": "en"}`; server limit37310 MB, 30 req/min, 60 s timeout (Embedded addon → Gemma 4 audio endpoint).374Auth: **WP application password → Basic** (satisfies the route's375`is_user_logged_in()`; stored in keychain via `set_wp_app_password` IPC);376assistant credential → Bearer requires the addon `permission_callback`377tweak (follow-up PR). WAV encoder: `audio/wav.rs::encode_f32_to_wav`.378Contract tests in `stt/embedded.rs` (endpoint/body/parse/auth) — no network.379380## Model registry conventions381382`stt/model_registry.rs`: pinned `ModelSpec` catalogue (`engine`, `id`, url,383optional sha256, `archive`, `file_name` override for non-`.bin` files).384Rules: streaming SHA-256 (never read models into memory), atomic rename,385measured-hash logging when unpinned, tar.bz2 extraction behind the386`parakeet` feature. Downloads live in387`%APPDATA%/nvoos-dictate/models/` (GUI: Settings → Engine & Models).388389Two guardrails added after the "parakeet model 'base.en' not found" wedge:390391- **`validate_pairing(engine, model)`** rejects catalogue-known cross-engine392 pairings (a whisper id under parakeet) BEFORE settings are persisted —393 called from `state::update_settings` and `create_engine`. Engines without394 catalogue entries (mock/embedded) and unknown ids always pass: the shipped395 default settings store the parakeet id under the `mock` engine and MUST396 keep saving. The Settings UI model field is a dropdown filtered by engine,397 so a mismatch cannot be typed anymore; custom ids still work via398 config.json.399- **`download(spec, dir, force, progress)`** — `force` restarts from zero400 instead of resuming the `.download` partial (resume can never repair a401 corrupt file). The Settings UI shows a **Re-download** button on installed402 models. `download_model` IPC re-runs `ensure_engine` after install so the403 fresh model is usable without an app restart.404405Re-download hardening: archive installs extract into a `.staging` dir and406atomically swap into place via `install_archive` — never extract over a live407install. Both the fresh tree and the previous install get a408`make_writable_recursive` pass (Windows `FILE_ATTRIBUTE_READONLY` clear,409Unix owner-write) because tar extraction preserves archive metadata and can410leave read-only model files that break later truncating overwrites.411Non-archive renames get `make_writable(dest)` first. The `download_model`412IPC **drops the resident engine before** replacing files (a loaded engine413can hold model files open/mapped on Windows) and re-creates it after.414**The actual root cause of the persistent "Download failed: I/O error:415creating or truncating a file requires write or append access" was NOT an416OS error — it is `OpenOptions` client-side validation** (`ErrorKind::417InvalidInput`): the tmp-file open had `.create(true).truncate(!resuming)`418with `.append(resuming)` but NO write access mode, so every fresh/forced419(non-resuming) download failed before touching disk, while resumed420downloads worked (append implies write). Fix: `.write(!resuming)` added;421the error now also carries the failing path. Regression tests:422`reinstall_over_readonly_previous_install_succeeds` and423`truncating_open_requires_write_access` (both feature `parakeet` for the424first; the second runs everywhere).425426## Verification harnesses427428```bash429# Registry-driven download (same path the GUI uses)430cargo run --features parakeet --example download_model -- parakeet sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8431cargo run --example download_model -- whisper tiny.en432433# Live-capture probe: does a fresh stream on a device actually deliver?434# (distinguishes OS/device problems from app-side capture bugs)435cargo run --example mic_probe # default device436cargo run --example mic_probe -- "ROG EYE S" # device whose name contains ...437438# Live-capture integration test on real hardware (ignored in CI)439cargo test -- --ignored live_capture_delivers_samples440441# WER + e2e_ms + RTF against a speech file (sidecar .txt = reference)442cargo run --features parakeet --example transcribe_file -- ../speech.wav --engine parakeet --model sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8443```444445- Speech samples without a mic: PowerShell SAPI TTS —446 `Add-Type -AssemblyName System.Speech; $s.Speak(...)` →447 `SetOutputToWaveFile(...)`.448- Reference-PC numbers (debug build, 4.27 s sample): parakeet 1187 ms449 (RTF 0.28), whisper tiny.en 1095 ms (WER 7.7%). Both beat the 1.5 s450 key-up→paste budget; release is faster.451452## Known Rust pitfalls (this codebase)453454- `cpal::Stream` is `!Send + !Sync` on WASAPI — streams must live on455 detached worker threads; only Send handles (ring buffer Arcs, channels,456 atomics) may enter `AppState`. Do not regress this.457 **And the stream must be HELD by the worker** — `build_stream` returns the458 live `cpal::Stream` alongside the ring, and the worker binds it for its459 lifetime. The original code dropped the stream at the end of460 `build_stream` (the "hold it forever" comment was aspirational): dropping461 a playing stream stops capture, so the ring NEVER filled — every dictation462 reported "no speech detected" on perfectly healthy mics. The463 `MicCapture` health counters (`callback_count()`, `stall_ms()`, stream464 errors) surface this on the Diagnostics page, `restart_microphone` IPC465 rebuilds capture without an app restart, and the ignored integration test466 `live_capture_delivers_samples` (`cargo test -- --ignored`) reproduces467 the failure on real hardware (ignored in CI — no audio device).468- **Bare expression paths do not resolve sibling modules declared in a469 parent module** (proven with a minimal repro): inside470 `stt/engine.rs`, `parakeet::X` fails while `crate::stt::parakeet::X`471 works — always use crate-qualified paths for cfg-gated siblings.472- **Hotkey swallow discipline (Windows hook)**: an armed PTT press must473 consume BOTH the key-down and the matching key-up (`SWALLOWED_DOWN`474 tracks the pair; auto-repeat downs stay swallowed). Swallowing only the475 key-up while the down passes through leaves Windows believing RAlt is476 permanently held — the "keyboard breaks, Ctrl+Alt+Del fixes it" stuck-Alt477 failure. Non-armed events (AltGr combos) always pass through via478 `ptt_pressed_alone`. **Exit discipline**: if the app shuts down with a479 swallow pending (Alt+F4 close while RAlt is held is the classic repro),480 the hook thread injects a synthetic key-up via `SendInput` AFTER481 `UnhookWindowsHookEx` (`release_pending_swallow` / `take_pending_swallow`482 bookkeeping) so the modifier is never left stuck for the session. The483 hook tests serialize on a shared `TEST_LOCK` because the hook state lives484 in process-global statics.485- **VAD gating (dictation)**: the energy gate is adaptive per window —486 `max(0.02, min(10th-percentile frame RMS × 6, 0.15))` (WebRTC-style noise487 gating); silero keeps its probability default. The VAD only trims silence488 edges — it is never the final gate: `vad_empty_fallback` transcribes the489 whole window whenever the peak exceeds 0.004 (false negatives must never490 drop user audio; muted/unplugged mics peak at ~0 and still error). Every491 attempt records `LastDictationStats` (window/utterance ms, RMS/peak, gate,492 outcome) surfaced by `get_diagnostics`.493- **Settings saves that fail validation must not persist**: `update_settings`494 runs `validate_pairing` BEFORE `settings.save()` so a broken combo can495 never wedge the next boot. If the engine still fails to load after a valid496 save (missing model files), the IPC error says exactly that — settings497 ARE saved in that case; only the engine swap failed.498499## PTT / hotkey stopped working — runbook500501When "Right Alt doesn't trigger dictation anymore" is reported, **do not502blame the most recent code first**: the hook path (`hotkey/win_hook.rs`,503`dictation.rs`) is deliberately isolated (vision/polish/engine changes never504reference it), so a git diff that only touches `vision/`, `cleanup/`,505`state.rs`, `ipc.rs`, `lib.rs`, or frontend files *cannot* have broken PTT.506Verify with `git diff <base>...HEAD -- src/hotkey/` — empty = hook code507untouched. Then work this checklist in order:5085091. **Instance check.** `tasklist //FI "IMAGENAME eq nvoos-dictate.exe"` —510 multiple rows mean a stale instance is holding the global keyboard hook or511 the single-instance guard redirected focus. Kill all512 (`taskkill //IM nvoos-dictate.exe //F`) and relaunch ONE fresh instance;513 the hook reinstalls clean. This session's "broken RAlt" was exactly this —514 many build/kill/relaunch cycles, not a code regression.5152. **Config check.** `cat "$APPDATA/nvoos-dictate/config.json"` (or516 Settings → Hotkey) — confirm `hotkey.ptt_key` is `RAlt` (Windows default;517 macOS uses `toggle` mode + `ptt_combo`). A leftover `toggle`/combo-only518 config on Windows means RAlt *hold* was never armed.5193. **Hook health (Diagnostics → Push-to-talk hotkey), the decisive signal:**520 - **Low-level listener** `running` (green) vs `not running` —521 `not running` = the WH_KEYBOARD_LL hook failed to install or the thread522 died; the `GetAsyncKeyState` fallback may still cover basic hold-PTT.523 - **Listener heartbeat age** — small (~0–500 ms) = alive; large/growing =524 the thread stalled or died.525 - **Listener callback invocations** — must increment while you tap RAlt;526 flat means the hook isn't receiving key events (another app grabbed the527 keyboard, or the hook wasn't installed).5284. **Engine, not hook.** If the listener is `running` AND callbacks fire but529 nothing pastes, the trigger is fine — the failure is downstream. Check530 Settings → Engine: the configured engine/model (e.g. `whisper/base.en`)531 must be **downloaded** (Diagnostics → Speech engine flags `not installed`).532 A valid-but-missing model errors on transcription, not on trigger.5335. **Stuck-Alt recovery.** If the keyboard itself "broke" (RAlt held,534 Ctrl+Alt+Del fixes), that's the swallowed key-up path (see the swallow535 discipline pitfall above) — close the app cleanly so `release_pending_swallow`536 injects the synthetic key-up, or press RAlt once more to clear it.537538539## Cleanup pipeline (golden backstop)540541Order matters: whitespace → vocabulary → numbers → fillers → capitalize →542punctuate → per_app. Any rule change must keep `src-tauri/tests/cleanup_golden.rs`543(17 cases + `meeting1` fixture) green or be an intentional, reviewed change.544The fixture previously said "So we need…" but the leading-marker rule strips545`so` after filler removal — "We need…" is the correct expectation.546547## Camera & live vision (Proposal 073 + 075)548549Two opt-in local sources on the webcam, both feature-gated and off by default:550551- **Camera preview (073)** — `camera/` module, `nokhwa` frames only, gated552 behind the `camera` feature. `nokhwa::Camera` is `!Sync` so it lives on a553 dedicated `nvoos-camera` worker thread; only `Send` handles (the554 `PreviewFrame` ring `Arc<Mutex<_>>`, atomics) cross into `AppState`.555 Camera *audio* is **not** nokhwa — a webcam mic is an ordinary `cpal` input556 device and transcribes through the existing mic path. `PreviewFrame` is557 downsampled to ≤320 px wide for the preview. IPC: `list_camera_devices`,558 `camera_available`, `start_camera`, `stop_camera`, `get_preview_frame`.559- **Object detection + tracking (075)** — `vision/` module, behind the560 `vision` feature (which implies `camera`: `vision = ["camera", "dep:ort",561 "dep:ort-sys", "dep:ndarray16"]`). Runs a YOLO ONNX detector (`yolov11n`562 default, `yolov8n` fallback) via the pinned `ort` rc.9, decodes + NMS-filters563 `[1, 84, 8400]` → source-pixel `Detection`s, then a dependency-free SORT564 tracker assigns stable ids + a click-to-select pin. **No facial565 recognition / identity** — bright-line non-goal, enforced in tool/UI copy.566567`vision/` layout mirrors `camera` for feature stability:568`mod.rs` (always compiled: `Detection`, `available()`) + `detect.rs`569(letterbox + `ort` + NMS; **both** a `#[cfg(feature = "vision")]` real body570and a stub), `tracker.rs` (pure SORT, always compiled + unit-tested),571`models.rs` (vision ONNX catalogue: `yolov11n`/`yolov8n`, SHA-256 + streaming572+ atomic install, mirroring `stt::model_registry`), and `controller.rs`573(`VisionController` — detector+tracker on a worker, snapshot + pin channel574handoff). `ort` rc.9 has **no** crate-root `Session`/`Value` re-export — use575`ort::session::Session`, `ort::value::Value`, and positional `ort::inputs!`576(`inputs![Value::from_array(tensor)?]`, NOT the named form). `ort` rc.9 maps577`ndarray` **0.16** while the app pins `ndarray` 0.15 for `silero-vad` — use the578`ndarray16` alias (already in `Cargo.toml`).579580`Settings.vision` gains `enabled`, `detector`, `confidence`, `iou_threshold`,581`track_objects`, `hand_model` (hand model + ByteTrack are deferred M5582follow-ups). `StatusInfo` adds `vision_available/running/detector/hand_model`;583`get_diagnostics` adds a `vision` block (availability, running, model install584state, fps-equivalent `frames`, `last_infer_ms`, `active_tracks`,585`track_events`, `errors`). IPC commands: `vision_available`, `start_vision`,586`stop_vision`, `get_detections`, `vision_select_track(id)`, `get_vision_models`,587`download_vision_model`. The Home "Camera preview" panel draws detections on a588`<canvas>` overlay with click-to-pin; the detector is started from the Home589panel and the model is downloaded from Diagnostics → Object detection (or590Settings → Vision).591592The de-risk spike `examples/vision_probe.rs` (feature `vision`) proved593`yolov11n.onnx` (exported `opset=12`, `onnx` Python module required to594regenerate) runs on `ort` rc.9 with the confirmed `[1, 84, 8400]` output and595~49 ms CPU inference. Weights export: install `onnx` (`pip install onnx`),596download `yolo11n.pt` from597`https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.pt`598(note: the id is `yolo11n`, not `yolov11n`), then599`python -c "from ultralytics import YOLO; YOLO('yolo11n.pt').export(format='onnx', opset=12, imgsz=640)"`.600No `onnxslim` = the simplifier step logs a non-fatal warning.601602**Vision label polish (Proposal 076):** the stock `yolov11n` is a603closed-vocabulary 80-class COCO model, so it mislabels arbitrary objects604(a pen → "knife"). The opt-in VLM polish layer fixes this: when605`Settings.vision.polish_enabled` is on, the detect loop crops each606low-confidence box, PNG-encodes it as a base64 data-URI, and sends a607**multimodal** OpenAI-compatible request (image + text) through the *same*608`LocalOpenAiProvider` transport used by text polish (`local_openai.rs`). The609returned `{label, confidence}` replaces the COCO label (badged `VLM` in the610overlay). Implementation: `vision/polish.rs` (`VisionPolisher` + `prepare_request`611/`resolve`, crop→base64 PNG), `openai_shape::{build_vision_body, parse_label_response,612vision_polish_prompt}`, `LocalOpenAiProvider::describe_crop`, and a dedicated613`nvoos-vision-polish` thread (owns a tokio runtime) so the ~500 ms VLM614round-trip never stalls the 15 fps detect loop. Default model615`qwen/qwen2.5-vl-7b` (the reference VLM for structured-JSON grounding; verified616live on the reference PC — it returns **fenced** JSON, which617`parse_label_response` tolerates). The no-facial-recognition bright line is in618the system prompt. `VisionDiagnostics` gained `polish_enabled/provder/model/calls/results`.619620## Stage-2 polish providers (local LLM cleanup)621622Stage-2 is the optional LLM polish that runs **after** Stage-1 deterministic623rules, gated by `cleanup.wait_for_llm_ms` (ADR 0001). Provider dispatch lives624in `cleanup/providers/mod.rs::create()` — `disabled` | `openai_compat` |625`lm_studio` | `ollama` | `nvoos`. Each provider receives text + target-app626category only (never audio); failure always falls back to Stage-1 text and627can never block a paste.628629**Provider architecture (refactor from this session):**630631- `openai_shape.rs` owns the provider-agnostic OpenAI `/v1/chat/completions`632 shape: `system_prompt_for(category)`, `build_chat_body(model: Option<&str>,633 text, category, temperature)` (omits the `model` field entirely when `None`),634 and `parse_chat_response` (defensive string-or-segment-array content).635- `local_openai.rs` owns the shared **local**-server behaviour636 (`LocalOpenAiProvider` parameterized by a `LocalKind { name, id }`): the637 local-http-vs-remote-https invariant, optional bearer auth, `GET /v1/models`638 discovery, `ttft_ms`/`total_ms` timing, `load_timeout_ms` cold-load budget639 enforcement, and actionable error mapping. LM Studio and Ollama are thin640 `LocalKind` wrappers over it — they share ~95% of their logic.641- `lm_studio.rs` (`LmStudioProvider`) and `ollama.rs` (`OllamaProvider`) supply642 only their identity (`name`/`id`) and read their own keychain token643 (`KEY_LM_STUDIO_TOKEN`, `KEY_OLLAMA_TOKEN`). `id()` returns `lm_studio` /644 `ollama` so transcript/history labelling stays correct.645646**Key facts that shaped the design:**647648- **Model identifiers are discovered, not typed.** Both local servers expose649 `GET /v1/models` → `{ data: [{ id, … }] }`; the `id` is what goes back on650 `chat/completions.model` (an LM Studio "model identifier", an Ollama model651 name like `llama3.2` — NOT a GGUF filename). The Settings tabs surface a652 discovered-model dropdown + auto-select the single loaded model.653- **Auth defaults to none.** The local server needs no token (any non-empty654 `Authorization: Bearer …` is accepted as a placeholder). The optional token655 is really for a remote/headless server and lives in the OS keychain — never656 in `config.json` (the `Settings` struct holds no secrets).657- **Single-loaded-model behaviour:** if exactly one model is loaded, the server658 serves it regardless of the `model` field; omitting `model` when the config659 is empty is the correct default.660- **Two distinct timeouts:** `cleanup.wait_for_llm_ms` bounds the *whole*661 polish (dictation.rs); `load_timeout_ms` (per local provider) is a separate662 `tokio::time::timeout` budget for a Just-In-Time cold model load, plus a 60 s663 soft reqwest-client default so a hung server can't block forever.664- **`temperature` is provider-specific:** `0.2` for `openai_compat` (cloud),665 `0.1` for local models (they drift more; cleanup is deterministic editing).666- **Polish latency:** `PolishMeta` carries `ttft_ms`/`total_ms` (local providers667 report them; `nvoos`/`openai_compat` leave `None`); `LastDictationStats`668 surfaces them as `polish_ttft_ms`/`polish_total_ms`, shown on the Diagnostics669 page under "Polish latency (TTFT / total)".670671**Settings plumbing per local provider:** config block (`LmStudioConfig` /672`OllamaConfig` with `base_url`, `model`, `load_timeout_ms`) wired into673`Settings`; keychain key + `has_*_token()` + `*_provider()` on `AppState`;674three IPC commands each (`test_*_connection`, `set_*_token`, `clear_*_token`)675registered in `lib.rs::run`; a Settings tab + provider `<option>` in676`Settings.tsx`; a credential row in `Diagnostics.tsx`. Default base URLs:677LM Studio `http://localhost:1234/v1`, Ollama `http://localhost:11434/v1`.678679**Pitfall (this session):** adding a field to `PolishMeta` (a struct with other680literal constructions) breaks `nvoos/types.rs::extract_polish_meta`, which681builds it field-by-field — use `..PolishMeta::default()` there. Also, a682`reqwest::Client` in a struct means it can't trivially `#[derive(Debug)]`, so683`unwrap_err()` on a `Result<Provider, _>` won't compile — match on it instead.684685The design rationale is captured in686`examples/nvoos-dictate/docs/lm-studio-integration-plan.md`.687688## File / media decode seam (Symphonia 0.5)689690File transcription (`Transcribe file…`, `util/decode.rs::decode_file`) decodes691mp3/m4a/wav/flac/ogg to mono f32 PCM via Symphonia and feeds the resident STT692engine through `ipc::transcribe_file`. Key facts:693694- **Symphonia 0695696…(truncated)