# Ovrtx Timeline Nle

> Build a multi-track, non-linear timeline (NLE) for USD variant + camera changes over time: a pure state-resolution engine (tracks/clips, state_at(t), frame sampling, mixer/slideshow presets), a client-side JS mirror for instant scrubbing, and a render-to-MP4 path on an ovrtx renderer. Use when adding a timeline/sequencer to a variant presenter.

- Skill: `nvidia-omniverse/ovrtx-timeline-nle` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nvidia-omniverse/ovrtx-timeline-nle`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-omniverse/ovrtx-timeline-nle/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: Apache-2.0
- Author: NVIDIA-Omniverse (https://skillmd.com/u/nvidia-omniverse)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia-omniverse/ovrtx-timeline-nle

---


# Timeline (NLE)

A flagship feature: tracks = variant sets (+ one camera track), clips = "this variant for
this duration". Scrubbing drives the live viewport; rendering writes an MP4.

## Pure engine (no ovrtx, no pxr — a test should assert that)

```
Clip  = {value, start_s, duration_s}      # value = variant name, or camera path on the camera track
Track = {kind: "variant_set"|"camera", set_name, prim_path, clips:[...]}
Timeline = {duration_s, fps, tracks:[...]}
```

Core functions:
- `validate(timeline)` — at most one track per variant set; clips have positive duration and
  don't overlap within a track.
- `state_at(timeline, t, base_selection) -> (Selection, camera_path|None)` — compose the
  full selection + active camera at time `t`. Per variant-set track: the covering clip's
  value, else the **previous** clip's value (a gap HOLDS the last value), else base. Untracked
  sets stay pinned to base. Camera track → active camera; **before the first camera clip the
  FIRST clip already governs** (don't fall back to "whatever's live" — that renders surprise
  lead-ins).
- `frame_times(timeline) -> [t...]` — `n = round(duration_s * fps)`, times `i/fps` for
  `i in range(n)`. (So 4 s @ 2 fps → 8 frames.)
- Presets: `make_mixer(sets)` = parallel tracks cycling every set in lockstep (shorter tracks
  hold their last value); `make_slideshow(sets)` = one track stepping through every
  (set, variant) one change at a time.

Keep this module import-pure (only dataclasses/typing + your models). The render thread and
the API both consume `state_at`/`frame_times`; the browser mirrors it in JS.

## Client-side JS mirror (instant scrub, no round-trip)

Mirror `state_at`/`frame_times`/`_value_at` in `timeline-core.js` and PIN IT with a node
test (`timeline-core.test.cjs`) that checks the JS against the same cases as the Python.
**Scrub has NO server endpoint:** on playhead move the frontend computes `state_at`
client-side and POSTs the resulting selection to `/api/variant` (+ `/api/camera/snap` when
the camera changes). This keeps scrubbing smooth and the server authoritative only for
render.

‼ **`renderTimeline()` MUST end by re-applying the state at the playhead** (`postStateAt(playhead)` when a
stage is open on the Timeline tab) so ANY edit — append a clip, change a track's variant, `▾`-edit a clip,
move/resize/clear — updates the viewport IMMEDIATELY. If you only push on scrub (or only for the *selected*
clip), the user sees "variants don't update until I move the time marker."

‼ **Material (fast-path) sets MUST update from the timeline exactly like geometry/environment sets.**
The timeline posts through the same `/api/variant` — if the server's fast path applies silently (writes
never reach the renderer, or `renderer.reset()` missing after them; see `usd-variant-live-switching`),
the user sees Doors/Backdrops clips update the viewport while **Carpaint clips change
nothing**. Verify timeline pixel behavior on a FAST set specifically, AFTER the `classified` event
(pre-classification everything reloads and the bug hides).

‼ **Clip swatch (`.sw`) is added ONLY when a swatch color exists** for that variant: `const sw =
swatches[set]?.[clip.value]; if (sw) { el.style.background = sw; append }` — do NOT always create the swatch
with a placeholder background (e.g. `#ffffff44`); that paints a stray white square on every clip.

> JS-editing trap: do not generate JS containing `${...}` template literals through a bash
> `python -c "..."` heredoc — bash expands `${...}` before Python sees it and silently strips
> them. Write JS with a file tool and `node --check` it.

## ‼ PLAYBACK CONTROL — a real LIVE transport, not just scrub
Pressing **Play** must actually play the assembled timeline in the LIVE viewport, not merely move a
playhead graphic. Required:
- **Real-time transport:** play advances the playhead on a `requestAnimationFrame` WALL-CLOCK
  (`rel_s += (now-last)/1000`), looping at `duration_s` when loop is on; **Pause/Stop** halt it;
  **step ±1 frame**, **to-start/to-end**, **loop toggle**; Space=play/pause, arrows=step, Home/End=ends
  (ignore when typing in an input).
- **The viewport updates LIVE as it plays** — at each tick compute `state_at(playhead)` and drive the
  SAME path as scrub (POST `/api/variant` on a variant change; `/api/camera/snap` on a camera change;
  for an animated/turntable camera evaluate its pose at the looped stage time). Playing must visibly
  switch paints/doors/cameras in the stream in real time — WYSIWYG. Coalesce: only POST when the
  computed selection actually CHANGES (don't spam `/api/variant` every frame).
- **Playhead + readouts move:** the green playhead line tracks the time, with a live `mm:ss / length`
  readout, and the ruler **auto-scroll-follows** under horizontal scroll.
- This is SEPARATE from playing a rendered MP4 in **Results** — that's the dock's `<video controls loop>`
  with its own scrubber/transport BELOW the video (ui skill). Both must work: live timeline playback in
  the viewport, AND rendered-video playback in the Results dock.
**Verify on pixels:** build a 2-clip paint timeline, press Play, and confirm the car color CHANGES in the
stream as the playhead crosses the clip boundary (not just the playhead moving); scrub mid-clip and
confirm the stream jumps to that clip's variant.

## ‼ ANIMATED camera clips ANIMATE under the playhead
A camera-track clip whose camera is ANIMATED (the Turntable rig, or an authored camera move) must
re-pose the rig at the **clip-relative stage time** on every scrub/play tick — not snap frame 0 once:
- client (`applyStateAt`): when the state's camera is animated, send `at_s = playhead − clip.start`
  with the camera snap (`/api/camera/snap {camera_path, at_s}`), EVERY time (not only on camera change);
- server (`_do_snap`): with `at_s` + an animated camera, `tc = loop_stage_time(at_s, fps, start, end)`
  → pxr `ComputeLocalToWorldTransform(tc)` on the live composite → write onto the viewer camera (the
  live-playback mechanism). Snapping camera clips statically makes the turntable clip show ZERO
  rotation while scrubbing/playing the timeline. Verify: scrub to two times inside a Turntable clip —
  the view ANGLE must differ.

## `/api/timeline/render` contract (get `frames` RIGHT — a 0 means a parse bug)

Body = `{ "timeline": {duration_s, fps, tracks:[...]}, "quality": {...}, "out_dir": str }`.
The handler MUST `Timeline.from_dict(body["timeline"])` then return
`{ "frames": len(frame_times(tl)) }` where `frames = round(duration_s * fps)`. A 4 s @ 2 fps
timeline MUST return **8** — if you get 0, you parsed the wrong field (e.g. read `duration`/`fps`
off the top-level body instead of `body["timeline"]`, or defaulted them to 0). Compute frames
from the timeline's own `duration_s`/`fps`, not from the (possibly empty) tracks. **Compute `frames`
SYNCHRONOUSLY in the HTTP handler (pure `frame_times` logic) and return it immediately, then kick the
render FIRE-AND-FORGET on the render thread** (emit `timeline_done` with the MP4 path when finished) —
do NOT block the handler on a render-thread reply for `frames` (that reply-wait flakes/times out and
returns a spurious 0/504). The frames count needs no GPU; only the MP4 assembly does. Verify: post a
2-clip 4 s @ 2 fps timeline → `{frames:8}` instantly → an `.mp4` lands in `out_dir` shortly after.

## Render to MP4 (render thread)

`/api/timeline/render {timeline, quality, out_dir}` validates, computes `frames =
len(frame_times(tl))`, returns `{frames}`, and queues a render-thread job that for each
frame time: resolves `state_at`, applies the selection (fast path or reopen) + camera pose
(for an animated/turntable camera, evaluate its world pose in pxr at the looped stage time —
`loop_stage_time(rel_s, fps, start, end)` — and write it onto the viewer camera via
`ovstage.Stage.write_attribute("omni:xform", ...)` / `session.write_omni_xform` — never the
deprecated `Renderer.write_attribute`), converges, saves a frame, then assembles the MP4 (reuse
the batch `frames_to_video`). Runs LIVE↔BATCH under `USD_LOCK` + heartbeat; cancel via the direct
flag. Pass the per-camera look/framing into each frame's composite (a timeline render that forgets
the camera overrides renders the wrong optics).

**‼ MP4 codec: H.264 (`libx264`) + `pix_fmt yuv420p` via imageio-ffmpeg — NEVER `cv2.VideoWriter`
with `mp4v`.** MPEG-4 Simple Profile writes a valid file that `/api/video` serves fine, but Chrome's
`<video>` cannot decode it → the Results player silently shows NOTHING (ffprobe the output if
unsure). Render frames at the CURRENT DISPLAY RESOLUTION, never a hardcoded 640×360.

## UI MUST let the user ADD clips to tracks (not just render a pre-made timeline)

The timeline is useless if there's no way to put a clip on a track. There is **one track per
variant set + one camera track**, and EACH track must expose a way to add a clip:
- a per-track **picker** of that set's variants (or, on the camera track, the authored cameras) +
  an **Append / Add-clip** button → appends a clip of the chosen value;
- an **append-mode** toggle in the toolbar (Stack after the last clip, or drop At-playhead);
- once a clip exists: **drag** to move, drag the right edge to resize, a **▾** on the clip to
  change its value, and Delete to remove it;
- **presets** (Mixer / Slideshow) auto-fill all tracks, and **Clear** empties them — but presets
  are NOT a substitute for manual per-track add.
Build a clip's value list from the open stage's variant sets + cameras (the camera track lists
`cameras[]`). Verify by clicking a track's picker, choosing a variant, clicking Append, and
seeing a clip appear on that track; do the same on the camera track.

**The per-clip `▾` must be a REAL, appended dropdown that picks ANY value directly** — actually
append a `<select>`/menu element to the clip and wire its change handler; do NOT (a) create the
control and forget to append it, (b) make it `double-click-to-cycle-to-the-next-variant`, or (c)
rely on dblclick at all. The user must be able to jump straight to an arbitrary variant. Selection +
a **dedicated Delete-clip button** (also Delete/Backspace) — not double-click-to-delete. (A `▾` that
is created but never appended, with a dblclick-cycle fallback, does not meet this bar.)

**Track management depth (don't skip):** per-track **hide/show** (a `−` on each track header + a
"show hidden…" menu to bring them back) and a **draggable label-gutter resize** for the track-label
column. These are part of "editing depth," not optional polish.

## UI
A timeline strip in the below-viewport dock (keep the viewport visible for scrub feedback) shown
**ONLY while the Timeline tab is active** (hidden on Configure/Grid/Results even with a stage open —
see the ui skill's tab-contextual dock; it is NOT an always-on fixture). Contents: drag/resize/delete
clips, a ruler scrub (debounce ~90 ms), Mixer/Slideshow/Clear presets, a transport (play/pause/step/loop
+ Space / arrows / Home-End), and Render-to-MP4. Clip fills can reuse the per-variant swatch colors from
the classifier.