# Vrt Live Stream

> vrt-viz live view: H.264 over WebSocket → browser WebCodecs

- Skill: `kornia/vrt-live-stream` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kornia/vrt-live-stream`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kornia/vrt-live-stream/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: kornia (https://skillmd.com/u/kornia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kornia/vrt-live-stream

---


# vrt-viz live view: H.264 over WebSocket → browser WebCodecs

Stream the annotated frame + world-frame BEV to a phone. Pipeline:
**render (CPU) → H.264 encode (worker thread) → WebSocket broadcast → browser
WebCodecs `VideoDecoder` → `<canvas>` with a jitter buffer.** Files: `stream.rs`
(`LiveStream` — owns encoders + worker + server), `serve.rs` (`StreamServer` WS
broadcast + handshake), `h264.rs` (`H264Encoder`, gstreamer), `client.html` (the
WebCodecs page, `include_str!`'d). Behind the `h264` feature (pulls gstreamer).

Use it from the app in two calls: `LiveStream::spawn(port, (w,h), (bw,bh), main_kbps,
bev_kbps, fps)?` then `sink.submit(main_rgb, bev_rgb)` per frame (drop-stale latest-only
slot; encode runs off your capture loop).

## Why H.264, not MJPEG

MJPEG is intra-only — every frame a full JPEG, **~22 Mbit/s** at 720p15, which buffers
on a cellular/remote link. H.264 inter-frame compression → **~4 Mbit/s** for both
streams (~5× measured here; the generic figure is 10–20×). This is *the* fix for remote
"buffering". A jitter buffer only smooths *variance*; it can't cure a bandwidth
shortfall — reach for the codec first.

## No NVENC on Orin Nano

The Orin **Nano has no hardware video encoder** (NVDEC decode only; NVIDIA's spec lists
encode as "1080p30 by 1–2 CPU cores"). Verified: the gstreamer nv plugin exposes
`nvv4l2decoder` **only**, no `/dev/*enc*`, no NVENC in sysfs. So `h264.rs` uses
**software `x264enc`**. On an NVENC Jetson (Orin NX/AGX) swap `x264enc` → `nvv4l2h264enc`
(NVMM path) — the only line that changes. Software x264 at 720p15 is cheap (~20 ms on a
worker thread, off the hot path).

## Encoder pipeline — the profile trap

`appsrc → videoconvert → video/x-raw,format=I420 → x264enc(tune=zerolatency
speed-preset=ultrafast) → video/x-h264,profile=main → h264parse →
video/x-h264,stream-format=avc,alignment=au → appsink`.

- **Force `I420` + a mainstream profile.** Feeding RGB straight to x264enc yields **High
  4:4:4** (profile `0xf4`), which browser WebCodecs **cannot decode**. I420 + zerolatency/
  ultrafast lands at **baseline 4:2:0** (`avc1.42…`) — decodes everywhere incl. iOS Safari.
- **`stream-format=avc`** (not Annex-B) → h264parse puts the **avcC** record in the caps
  `codec_data`; each sample is length-prefixed AVCC. That avcC is the WebCodecs
  `description` — the portable path (Safari won't reliably take raw Annex-B).
- **Zero-copy:** `encode(rgb: Vec<u8>)` uses `Buffer::from_slice` (not
  `from_mut_slice(rgb.to_vec())`) — saves a ~2.7 MB copy/frame; `get_mut` for pts still
  works (metadata, not the readonly memory).
- **GOP = fps** (1 s keyframe interval) so a new viewer joins/re-syncs within ~1 s.

## WebSocket wire + client

One WS carries both views. Binary message = `[kind, tag] + payload`: `kind` ∈
`C`(avcC config) / `K`(keyframe) / `D`(delta), `tag` ∈ `M`/`B` (a `Stream` enum, not
magic bytes). Handshake is hand-rolled SHA-1 + Base64 (dependency-free; no sha1 crate in
the tree). New viewer: cached config first, then skip deltas until the first keyframe.
H.264 is inter-frame → a viewer that falls a full bounded-channel behind is **dropped and
reconnects** (can't drop mid-GOP).

Client (`client.html`): configure `VideoDecoder` with the avcC `description` (codec
string derived from `avcC[1..3]`), decode key/delta chunks, push decoded frames to a
per-stream **jitter buffer** (`MINBUF`/`MAXBUF`), draw on a fixed ~15 fps tick, drop
stale. Auto-reconnect on close.

## The failure modes we actually hit

- **"EncodingError: Decoder failure" after minutes** = **leaked decoders**. The error
  handler must **`close()` the VideoDecoder before discarding it** (the `reset()`
  helper) — iOS Safari allows only a few HW decode sessions, so each un-closed decoder
  leaks one until the pool exhausts and the stream dies permanently. Use `reset()` on
  error, config-change, and reconnect.
- **Choppy, not laggy = cadence, not bandwidth.** A fixed-timer client push (`sleep(66ms)`)
  beats against the producer's jittery cadence → duplicate/dropped frames = judder. Fix:
  **event-driven delivery** (version + condvar), send each frame once when published.
- **Still choppy after that = CPU contention.** Encode competes for cores; a niced
  worker starved by other pegged processes delivers unevenly even though production is
  steady. Check per-core load, not just fps.
- **Still buffering at full res = bandwidth.** Drop bitrate (`RTSP_TRACK_MAIN_KBPS` /
  `_BEV_KBPS`) or resolution; the jitter buffer won't manufacture link capacity.
- **Stale client after an edit** — serve the index with `Cache-Control: no-cache`.

## Serving / viewing

`serve` or `:PORT` on the example → open `http://<jetson-ip>:PORT`. Encode time is
logged on the worker (`RUST_LOG=<bin>=debug`). `/main` `/bev` MJPEG were removed once
H.264 landed — `/` is the WebCodecs page, `/ws` the stream.

## Related skills

- `vrt-pipeline-compose` — produces the frames/tracks you render + stream.
- `vrt-tracking` — the tracks drawn in the main view + BEV.
- `jetson-benchmarking` — encode cost + power discipline.