Isaac Sim Rendering

Isaac Sim Headless Rendering (Kit 110 / Isaac Sim 6.0+)

Capture pipeline, lighting recipes, ACES calibration, camera math, validation. Host-agnostic; adapt paths to your environment.

Read first

• navigation-primitives: look-at chase camera math (cross-referenced).

Capture: SimulationApp + Replicator RGB Annotator

Standard Kit 110 / Isaac Sim 6.0+ capture pipeline. Works headless including on ARM64 / GB10 Spark (the older 5.1.0 black-frame bug is resolved).

from isaacsim import SimulationApp
app = SimulationApp({"headless": True, "width": 1920, "height": 1080,
                     "renderer": "RayTracedLighting"})

import omni.replicator.core as rep
from pxr import UsdGeom, Gf

# Camera as a USD prim (NOT isaacsim.sensors.camera.Camera)
cam = UsdGeom.Camera.Define(stage, "/World/RenderCam")
cam_xf = UsdGeom.Xformable(cam.GetPrim())
cam_xf.AddTranslateOp().Set(Gf.Vec3d(x, y, z))
cam_xf.AddRotateXYZOp().Set(Gf.Vec3f(rx, ry, rz))
cam.CreateFocalLengthAttr().Set(20.0)

# Render product + RGB annotator
render_product = rep.create.render_product("/World/RenderCam", (1920, 1080))
rgb_annot = rep.AnnotatorRegistry.get_annotator("rgb")
rgb_annot.attach([render_product])

# Settle: 200 frames for RT2 to converge on most scenes,
# 500 frames for deeply-occluded indoor aisles.
for _ in range(200):
    app.update()

# Per-frame capture
rep.orchestrator.step()
data = rgb_annot.get_data()       # (H, W, 4) uint8 RGBA
rgb = data[:, :, :3]

Capture method choice:

• omni.replicator.core RGB annotator -> reliable, supports any resolution.
• RtxCamera + CameraSensor (from isaacsim.sensors.experimental.rtx) for tick-rate control, OpenCV / fisheye lens distortion, ISP, tiled multi-view, or stereo depth (see isaac-camera).
• Swapchain capture -> also works on Kit 110 if you explicitly set window size matching the render resolution.
• Replicator render products may return empty arrays for Gaussian splat scenes; fall back to swapchain capture in that case.

RT2 vs PathTracing

settings.set("/rtx/rendermode", "RayTracedLighting")  # RT2 — real-time
# settings.set("/rtx/rendermode", "PathTracing")      # offline only

Default to RT2. Switch to PathTracing only after RT2 has been calibrated and the user asks for hero quality.

Headless Lighting — Add Explicit Lights

Headless Isaac Sim has NO default lighting. Without explicit lights, frames are black (RGB=0). Always inject at least a DomeLight + DistantLight baseline.

from pxr import UsdLux, UsdGeom, Gf

dome = UsdLux.DomeLight.Define(stage, "/World/DomeLight")
dome.GetIntensityAttr().Set(400.0)

sun = UsdLux.DistantLight.Define(stage, "/World/Sun")
sun.GetIntensityAttr().Set(1500.0)
UsdGeom.Xformable(sun.GetPrim()).AddRotateXYZOp().Set(Gf.Vec3f(-50, 20, 0))

Baseline Intensity Guide

ACES Tone Mapping — The Single Biggest Quality Lever

Without ACES, no amount of intensity tuning produces balanced indoor renders. This is the single most impactful render setting after lighting.

import carb
s = carb.settings.get_settings()

s.set("/rtx/post/tonemap/op", 4)            # ACES
s.set("/rtx/post/tonemap/filmIso", 600.0)   # key parameter (see table)
s.set("/rtx/post/tonemap/whitepoint", 6500.0)
s.set("/rtx/post/tonemap/enabled", True)
s.set("/rtx/post/aa/op", 3)                 # TAA for RT2

filmIso Calibration (validated on warehouse interiors)

Anti-Recipes (don't waste time on these)

• Wide rect lights (width=5+) → flat, no light pools
• High dome intensity (400+) with ACES filmIso 600 → washes out shadows
• Reinhard tonemapping → muddy, low contrast
• PathTracing for iterative work → 5-30 min per frame, kills velocity

Warehouse Lighting Recipe (proven 7/10 → 9/10)

The biggest single quality improvement came from this lighting + fog recipe.

# LOW ambient dome — let spots do the work
dome = UsdLux.DomeLight.Define(stage, "/World/L/Dome")
dome.CreateIntensityAttr(150.0)               # Low, not 400
dome.CreateColorAttr(Gf.Vec3f(0.85, 0.88, 0.95))  # Cool blue-ish

# Focused rect lights for POOLS of light (not wide floods)
for i in range(N):
    rl = UsdLux.RectLight.Define(stage, f"/World/L/HB{i}")
    rl.CreateIntensityAttr(12000.0)           # Bright
    rl.CreateWidthAttr(1.5)                   # Narrow
    rl.CreateHeightAttr(0.2)
    rl.CreateEnableColorTemperatureAttr(True)
    rl.CreateColorTemperatureAttr(4200.0)     # Neutral-warm

# Alternating accent lights (asymmetry = realism)
# Alternate left/right, warm color, mid-height on racks

# Subtle fog — sells scale
s.set("/rtx/fog/enabled", True)
s.set("/rtx/fog/fogDensity", 0.004)
s.set("/rtx/fog/color", (0.85, 0.87, 0.92))   # Cool gray

For 40m warehouse: fog density 0.003 adds depth without murk.

Deep-Aisle Indoor Lighting

Problem

Ground-level camera in narrow aisle = black frame (82KB / mean_RGB < 5). Ceiling rect lights at Z=10m can't illuminate a 3.5m-wide × 8m-tall aisle to ground level — RT2 struggles with deep occlusion.

Solution: Multi-Layer Lighting

# Layer 1: dense ceiling grid (6×12 across facility)
# Rect lights at ceil_z-0.3, pointing down
# intensity=200000, width=4.0, height=3.0  (wide coverage)

# Layer 2: low sphere lights IN each aisle at Z=3.5m (head-height)
# Directly in camera FOV between tier 1 and ground
for aisle_y, lx in aisle_light_positions:
    lt = UsdLux.SphereLight.Define(stage, lp)
    lt.GetRadiusAttr().Set(0.15)
    lt.GetIntensityAttr().Set(100000.0)

• 500 settle frames for indoor aisle scenes (not 200-300)
• Dome at 300 intensity is optional ambient fill — don't go higher or open views wash out

Dome vs Deep-Aisle Tension (fundamental conflict in enclosed scenes)

• High dome → overview/topdown overexpose (mean > 220)
• Low/no dome → deep aisle underexpose (mean < 10)
• Best balance: no dome + sphere lights in aisles + 500K rect grids + 500 settle frames
  • Hero aisle: mean ~60
  • Overview (elevated 3/4): mean ~140-175
  • Cross-aisle: mean ~230

Validated ACES filmIso=600 Light Intensities

• Ceiling rect lights: 70,000 intensity, 2.5×1.5m, warm white (1.0, 0.97, 0.92)
• Aisle sphere lights: 15,000 intensity, radius=0.1, at Z=3.5m
• Grid: 8×14 ceiling panels
• No dome light — ACES handles exposure
• Result: mean 60–155 across all view types

Camera tip: place "hero" camera at cross-aisle intersections, not deep in narrow aisles. The junction has more open space for light to reach.

Frame Quality Validation

Always validate captured frames before delivery. Don't ship black/overexposed frames.

import numpy as np
def validate_frame(rgb_array):
    """Returns (ok: bool, reason: str)."""
    if rgb_array.max() == 0:
        return False, "no light reaches camera — add DomeLight + DistantLight"
    if rgb_array.mean() > 220:
        return False, f"overexposed (mean={rgb_array.mean():.0f}) — reduce intensity"
    if rgb_array.mean() < 10:
        return False, f"underexposed (mean={rgb_array.mean():.0f}) — add aisle lights"
    return True, f"ok (mean={rgb_array.mean():.0f}, max={rgb_array.max()})"

Look-At Camera Math

For chase/POV/overview cameras pointing at a target, always use a look-at matrix. Don't hand-tune Euler angles — they're brittle and you'll waste hours on sign flips.

from pxr import Gf

def look_at_matrix(eye, target, up=Gf.Vec3d(0, 0, 1)):
    """Returns Gf.Matrix4d for a USD camera at `eye` looking at `target`.
    Handles degenerate up-vector (camera looking straight down/up)."""
    eye = Gf.Vec3d(*eye)
    target = Gf.Vec3d(*target)
    fwd = (target - eye).GetNormalized()

    # Degenerate up-vector fallback (top-down or bottom-up)
    if abs(fwd * up) > 0.99:
        up = Gf.Vec3d(0, 1, 0)

    right = (fwd ^ up).GetNormalized()       # cross product
    cam_up = (right ^ fwd).GetNormalized()

    m = Gf.Matrix4d()
    m[0] = [right[0], right[1], right[2], 0]
    m[1] = [cam_up[0], cam_up[1], cam_up[2], 0]
    m[2] = [-fwd[0], -fwd[1], -fwd[2], 0]    # USD camera: -Z is forward
    m[3] = [eye[0], eye[1], eye[2], 1]
    return m

Third-Person Camera Offsets (Z-up, robot facing +X at yaw=0)

import math

behind_dir_x = -math.cos(yaw)
behind_dir_y = -math.sin(yaw)
right_dir_x  = -math.sin(yaw)
right_dir_y  =  math.cos(yaw)

cam_x = robot_x + behind_dist * behind_dir_x + side_offset * right_dir_x
cam_y = robot_y + behind_dist * behind_dir_y + side_offset * right_dir_y
cam_z = height

• side_offset = -2.5 → camera on robot's right
• side_offset = +2.5 → camera on robot's left
• Flip the offset value to change sides, NOT the trig signs.

Dynamic Camera Height (Obstacle Avoidance)

When tracking through cluttered environments, the chase camera will clip into tall geometry. Pre-compute obstacle bboxes, then raise the camera each frame as needed.

# Build obstacle lookup from USD geometry once at startup
obstacles = []
for prim in stage.Traverse():
    if prim.IsA(UsdGeom.Cube):
        # ... extract (xmin, xmax, ymin, ymax, height) ...
        obstacles.append((xmin, xmax, ymin, ymax, height))

def cam_max_height_at(cx, cy, margin=0.5):
    """Highest obstacle near (cx, cy). Camera must clear this."""
    return max((h for xmn, xmx, ymn, ymx, h in obstacles
                if xmn-margin <= cx <= xmx+margin and ymn-margin <= cy <= ymx+margin),
               default=0.0)

# Per-frame:
target_h = max(base_height, cam_max_height_at(cam_x, cam_y) + 1.0)
smooth_h = smooth_h * 0.95 + target_h * 0.05   # smooth transitions

Robot XformOp Discipline

URDF-imported robots (Spot, Carter, etc.) already have authored translate + orient + scale xformOps on the root prim.

• Use xf.ClearXformOpOrder(); xf.MakeMatrixXform() on the root prim only for initial placement.
• Never add ops to child body/link prims — physics drives those.

Video Assembly

ffmpeg -y -framerate 30 -i frames/frame_%05d.png \
  -c:v libx264 -pix_fmt yuv420p -crf 18 output.mp4

Frame numbering must be sequential (frame_0000.png, frame_0001.png, …) — ffmpeg skips gaps.

Session Management

For batch/iterative rendering, keep the Kit app running and switch stages in-place rather than restarting:

• Cold start = 5-7 min wasted
• Persistent session = 10-15s per render
• Use omni.usd.get_context().open_stage(path) to switch scenes
• Only restart Kit if it crashes or hits OOM

Implementation is up to you (REPL, command file, IPC, etc.) — the principle is "don't pay the cold-start cost more than once."

Checklist Before Delivering Renders

1. RT2 enabled (/rtx/rendermode = RayTracedLighting)
2. ACES tone mapping enabled (/rtx/post/tonemap/op = 4)
3. filmIso calibrated for scene type (200 general / 400 aerial-heavy / 600 deep-aisle)
4. Explicit DomeLight + DistantLight (or scene-specific multi-layer setup)
5. Settle frames sufficient (200 standard / 500 deep-aisle)
6. Frame validation passed (rgb.mean() in 30-200 range, file size > 200KB)
7. Frame sequence is gapless for ffmpeg

Integration Points

• RECEIVES from: urdf-mjcf-to-usd-conversion, usd-articulation, mobility-gen, isaac-sim-robot-navigation — populated stages to render
• PRODUCES for: data-collection-sim — validated frame sequences for SDG
• PRODUCES for: isaac-sim-validator — outputs for final QA gate