Vrt Live Stream

vrt-viz live view: H.264 over WebSocket → browser WebCodecs

Stream the annotated frame + world-frame BEV to a phone. Pipeline: render (CPU) → H.264 encode (worker thread) → WebSocket broadcast → browser WebCodecs VideoDecoder → <canvas> with a jitter buffer. Files: stream.rs (LiveStream — owns encoders + worker + server), serve.rs (StreamServer WS broadcast + handshake), h264.rs (H264Encoder, gstreamer), client.html (the WebCodecs page, include_str!'d). Behind the h264 feature (pulls gstreamer).

Use it from the app in two calls: LiveStream::spawn(port, (w,h), (bw,bh), main_kbps, bev_kbps, fps)? then sink.submit(main_rgb, bev_rgb) per frame (drop-stale latest-only slot; encode runs off your capture loop).

Why H.264, not MJPEG

MJPEG is intra-only — every frame a full JPEG, ~22 Mbit/s at 720p15, which buffers on a cellular/remote link. H.264 inter-frame compression → ~4 Mbit/s for both streams (~5× measured here; the generic figure is 10–20×). This is the fix for remote "buffering". A jitter buffer only smooths variance; it can't cure a bandwidth shortfall — reach for the codec first.

No NVENC on Orin Nano

The Orin Nano has no hardware video encoder (NVDEC decode only; NVIDIA's spec lists encode as "1080p30 by 1–2 CPU cores"). Verified: the gstreamer nv plugin exposes nvv4l2decoder only, no /dev/*enc*, no NVENC in sysfs. So h264.rs uses software x264enc. On an NVENC Jetson (Orin NX/AGX) swap x264enc → nvv4l2h264enc (NVMM path) — the only line that changes. Software x264 at 720p15 is cheap (~20 ms on a worker thread, off the hot path).

Encoder pipeline — the profile trap

appsrc → videoconvert → video/x-raw,format=I420 → x264enc(tune=zerolatency speed-preset=ultrafast) → video/x-h264,profile=main → h264parse → video/x-h264,stream-format=avc,alignment=au → appsink.

• Force I420 + a mainstream profile. Feeding RGB straight to x264enc yields High 4:4:4 (profile 0xf4), which browser WebCodecs cannot decode. I420 + zerolatency/ ultrafast lands at baseline 4:2:0 (avc1.42…) — decodes everywhere incl. iOS Safari.
• stream-format=avc (not Annex-B) → h264parse puts the avcC record in the caps codec_data; each sample is length-prefixed AVCC. That avcC is the WebCodecs description — the portable path (Safari won't reliably take raw Annex-B).
• Zero-copy: encode(rgb: Vec<u8>) uses Buffer::from_slice (not from_mut_slice(rgb.to_vec())) — saves a ~2.7 MB copy/frame; get_mut for pts still works (metadata, not the readonly memory).
• GOP = fps (1 s keyframe interval) so a new viewer joins/re-syncs within ~1 s.

WebSocket wire + client

One WS carries both views. Binary message = [kind, tag] + payload: kind ∈ C(avcC config) / K(keyframe) / D(delta), tag ∈ M/B (a Stream enum, not magic bytes). Handshake is hand-rolled SHA-1 + Base64 (dependency-free; no sha1 crate in the tree). New viewer: cached config first, then skip deltas until the first keyframe. H.264 is inter-frame → a viewer that falls a full bounded-channel behind is dropped and reconnects (can't drop mid-GOP).

Client (client.html): configure VideoDecoder with the avcC description (codec string derived from avcC[1..3]), decode key/delta chunks, push decoded frames to a per-stream jitter buffer (MINBUF/MAXBUF), draw on a fixed ~15 fps tick, drop stale. Auto-reconnect on close.

The failure modes we actually hit

• "EncodingError: Decoder failure" after minutes = leaked decoders. The error handler must close() the VideoDecoder before discarding it (the reset() helper) — iOS Safari allows only a few HW decode sessions, so each un-closed decoder leaks one until the pool exhausts and the stream dies permanently. Use reset() on error, config-change, and reconnect.
• Choppy, not laggy = cadence, not bandwidth. A fixed-timer client push (sleep(66ms)) beats against the producer's jittery cadence → duplicate/dropped frames = judder. Fix: event-driven delivery (version + condvar), send each frame once when published.
• Still choppy after that = CPU contention. Encode competes for cores; a niced worker starved by other pegged processes delivers unevenly even though production is steady. Check per-core load, not just fps.
• Still buffering at full res = bandwidth. Drop bitrate (RTSP_TRACK_MAIN_KBPS / _BEV_KBPS) or resolution; the jitter buffer won't manufacture link capacity.
• Stale client after an edit — serve the index with Cache-Control: no-cache.

Serving / viewing

serve or :PORT on the example → open http://<jetson-ip>:PORT. Encode time is logged on the worker (RUST_LOG=<bin>=debug). /main /bev MJPEG were removed once H.264 landed — / is the WebCodecs page, /ws the stream.

Related skills

• vrt-pipeline-compose — produces the frames/tracks you render + stream.
• vrt-tracking — the tracks drawn in the main view + BEV.
• jetson-benchmarking — encode cost + power discipline.