# Renderer Gpu Interop > Share renderer output zero-copy across GPU APIs and runtimes (Vulkan↔CUDA, Metal/MPS, DLPack to Warp/PyTorch) and own GPU resource lifetime and teardown across that boundary. Use when Codex wires external-memory/-semaphore interop, hands a render target to a training/compute consumer without a CPU bounce, or debugs interop fd ownership, cross-API sync, double-buffered overlap, ARC/handle lifetime, or teardown-order crashes. - Skill: `nvidia-omniverse/renderer-gpu-interop` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add nvidia-omniverse/renderer-gpu-interop` - Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-omniverse/renderer-gpu-interop/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: NVIDIA-Omniverse (https://skillmd.com/u/nvidia-omniverse) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/nvidia-omniverse/renderer-gpu-interop --- # Renderer GPU Interop ## Purpose The headline value of a headless sensor renderer is often that its pixels never touch the CPU between raygen and the consumer (a Warp/PyTorch training tensor). That payoff lives entirely in the interop seam — and the seam is where the scariest, *silent* bugs live: leaked kernel fds, use-after-free across the API boundary, desynced double buffers, and teardown-order crashes that only fire at process exit. This skill is the knowledge for crossing that boundary safely. Pairs with `renderer-realtime-sensor` (the sensor-output side) and `renderer-simulation-loop` (the per-step update side). ## Ownership is a transfer protocol — name the exact transfer point - An exported memory fd (`VK_KHR_external_memory_fd` → `cuImportExternalMemory`, or the reverse) is *consumed* by exactly one call **on success**; **on failure the producer still owns it and must `close()` it.** Define and document that single transfer point, or you leak a kernel fd every frame (or double-close → `EBADF`). - The consumer's view — a DLPack tensor wrapping interop VRAM — does **not** own the memory: its deleter must be a **no-op**. The producing allocation owns the buffer and outlives the per-frame view; a real deleter frees memory the renderer still owns → use-after-free next frame. - On a **unified-memory backend** (Apple Silicon), the whole external-memory dance collapses: a `Shared` buffer *is* the staging buffer and the consumer reads its `contents` pointer directly. Do not port the fd/external-memory plumbing onto UMA — stub it and record why. (And a host-sync call like `synchronizeResource:` is *invalid* on Shared storage.) ## Sync has modes that are not interchangeable - A CPU-fence wait serializes producer and consumer. An **imported timeline semaphore** waited on the consumer's stream is what lets the GPU wait *overlap* with host work. Choose by whether you need overlap — the semaphore import is cheap; the memory import is the expensive part. ## Double-buffered overlap: one producer index, readers derive theirs - The overlap invariant (consumer reads frame N−1 while the producer writes N) holds only if there is **one** write index owned by the producer and every reader computes `read = 1 − write_idx` — never its own counter (a second counter desyncs the buffers). A frame-skip path must leave that index *untouched* so stale readers re-see the last good slot. - Know and *document* tolerated asymmetries instead of hiding them: e.g. under caller-owned ("inverted") memory the set-A/set-B alternation may drop every other frame's writes — ship single-buffered there and state why it is harmless (the latest writes to slot 0 are always visible) rather than pretending overlap works. ## Inverted ownership dodges a sync slow-path — at the cost of strict teardown order - Holding active consumer-side imports of producer-allocated memory can trip a slow mode (e.g. in `cudaStreamSynchronize`). Inverting it — the consumer allocates and exports, the renderer imports — sidesteps that, **but** the renderer must then be torn down (or re-pointed at fresh fds) *before* the caller frees its allocation. Reversed free order → the GPU dereferences freed memory. ## Resource lifetime and teardown is a dependency graph, not a list - Destroy consumers before producers, and wait-idle first: descriptor sets / pipelines that *reference* a buffer before the buffer; cached command buffers before the command pool; memory pools before the device. Annotate every destroy with its "must come before X because…" reason inline — an out-of-order destroy is a use-after-free that usually only crashes at *exit*. - Make destroys **idempotent** and call them unconditionally — a resource that was uploaded but never built still leaks if its free is gated on the "built" flag. - Any state change that recreates descriptor sets invalidates cached command buffers recorded against the old sets — invalidate the command-buffer cache in the *same* call. - After you drop the CPU mirror (a `finalize` step), defend the GPU buffers against a spurious rebuild request — refuse to re-upload from nothing and nuke correct state. - **Handle-runtime hazards differ by API.** Under ARC (Objective-C++), allocate handle structs with `new`/`delete` (value-init nils the `id` fields; the destructor releases them) and **never `memset` the whole struct** (it clobbers ARC's strong refs) — yet `memset` *is* correct on POD GPU-input structs (instance descriptors, push-constant scratch). The discriminator is "does this struct hold a managed handle," not "is it a struct." Release is assignment-to-nil, not an explicit call. - Split teardown into **persistent** (device / shader library / pipeline — scene-independent and expensive) vs **per-scene** (BLAS/TLAS/scene buffers); the per-scene destroy must not touch persistent state, or you re-pay shader compilation per scene swap. ## Process-global coexistence hazards - The NVIDIA driver may refuse a fresh EGL context **after Vulkan-WSI has loaded in the same process** — create the GL context first and have the renderer reuse it. - OpenGL binds one context per thread *implicitly* (unlike explicit Metal/Vulkan handles): own GL on a dedicated worker thread, serialize all GL calls through it, and never mutate the embedder's main-thread GL state. ## Validate as a boundary - Reproduce with a tiny producer→consumer loop: assert the consumer reads the producer's *latest* frame with zero CPU copies on the hot path; watch the fd table (growth per frame is the leak signature); and test teardown explicitly — construct, hand off, free in the documented order *and* the wrong order, under a sanitizer. ## Anti-Patterns - Leaving fd ownership undefined → leak on failure or double-close. - A DLPack / consumer view with a real (non-no-op) deleter. - A second reader-side buffer counter instead of deriving from the producer's index. - Porting external-memory fd plumbing onto a unified-memory backend. - `memset` over a struct holding ARC/managed GPU handles. - Destroying producers before consumers; gating a free on a "built" flag. - A fresh EGL context after Vulkan in-process; touching the embedder's GL state off the worker thread. ## Handoff Report the interop path (which API exports/imports memory + semaphore), the ownership/transfer points and free order, the buffering scheme and any tolerated asymmetry, what runs on the device vs CPU, and the teardown-order + fd-leak evidence. Defer the physics→render contract to `renderer-simulation-loop`.