# Nsys Analyze

> Analyze profiling data from Kit-based apps. Covers Omniverse-specific NVTX zone interpretation, phase detection using sqlite3, Tracy Statistics/Range Limit analysis, csvexport fallback queries, and two-version comparison methodology. Use after capturing profiles with the profiling skill. NOT for capturing traces (use profiling), adding zones to code (use profiling-api), or applying fixes (use perf-tuning).

- Skill: `nvidia/nsys-analyze` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nvidia/nsys-analyze`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/nsys-analyze/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia/nsys-analyze

---


# Profile Analysis for Omniverse / Kit-based Apps

Analyze profiling data from Kit, Isaac Sim, and Isaac Lab using `sqlite3` (for `.nsys-rep`), Tracy Statistics/Range Limit (primary `.tracy` path), and `csvexport` (automated `.tracy` fallback).
For capturing profiles and installing tools, see the `profiling` and `install-profilers` skills.

**Required tools:** `nsys`, `sqlite3`, `csvexport`; Tracy GUI is needed for the primary `.tracy` Statistics workflow. See `install-profilers` skill.

## Omniverse NVTX Zone Reference

| Zone Pattern | Meaning | Phase |
|---|---|---|
| `App Update` / `App Main loop` | Frame boundaries | Runtime |
| `UsdFileOp` / `UsdFileOp::open` / `UsdFileOp::newStage` | Stage operations | Startup/Loading |
| `UsdContext::Impl::render` | USD render context | Runtime |
| `RtxHydraEngine::render*` | RTX render passes | Runtime |
| `Hydra render views*` | Hydra render delegate ops | Runtime |
| `OmniGraph::*` / `ComputeGraphImpl::*` | OmniGraph compute | Runtime |
| `GeoTreeNode::*` / `Fabric::*` | Fabric/scene population | Loading/Runtime |
| `Carbonite::*` / `carb::*` | Low-level framework (**noise — exclude**) | All |
| `Thread waiting...` | Idle thread (**noise — exclude**) | All |
| `Executing task` / `Running fiber` | Task scheduler (**noise — exclude**) | All |

## Phase Detection Rules

Kit apps have phases: **startup** → **loading** → **runtime** → **shutdown**.

- **Startup** = trace start → first `App Update` frame
- **Loading frames** = frames with duration > 5× median (stage loading spikes — can appear at start *or* mid-run)
- **Runtime frames** = frames with duration ≤ 5× median (steady-state)
- **Frame marker** = `App Update` zone (NOT `App::beginUpdate`)

> **Note:** Loading in Kit apps often happens *during* runtime as a long frame, not as a separate phase before the first frame. The 5× median threshold reliably separates loading spikes from runtime frames.

---

## Analysis Path A: nsys SQLite (for .nsys-rep files)

### Step 1: Export to SQLite

```bash
nsys export --type=sqlite -o profile.sqlite profile.nsys-rep --force-overwrite=true
```

### Step 2: Overview + Phases + Frame Analysis

```sql
sqlite3 -header -column profile.sqlite "
WITH frames AS (
  SELECT ROW_NUMBER() OVER (ORDER BY e.start) as n,
         e.start, e.end, (e.end - e.start) as dur_ns
  FROM NVTX_EVENTS e LEFT JOIN StringIds s ON e.textId = s.id
  WHERE COALESCE(e.text, s.value) = 'App Update' AND e.end IS NOT NULL
),
frame_med AS (
  SELECT dur_ns as med FROM frames ORDER BY dur_ns
  LIMIT 1 OFFSET (SELECT COUNT(*)/2 FROM frames)
),
runtime AS (
  SELECT dur_ns FROM frames, frame_med WHERE dur_ns <= med * 5 ORDER BY dur_ns
)
SELECT
  ROUND((SELECT (MIN(start) - (SELECT MIN(start) FROM NVTX_EVENTS)) / 1e9 FROM frames), 2) as startup_sec,
  ROUND((SELECT (MAX(end) - MIN(start)) / 1e9 FROM frames), 2) as total_sec,
  (SELECT COUNT(*) FROM frames) as total_frames,
  (SELECT COUNT(*) FROM frames, frame_med WHERE dur_ns > med * 5) as loading_frames,
  COUNT(*) as runtime_frames,
  ROUND(AVG(dur_ns)/1e6, 2) as mean_ms,
  (SELECT ROUND(dur_ns/1e6,2) FROM runtime LIMIT 1 OFFSET (SELECT COUNT(*)/2 FROM runtime)) as p50_ms,
  (SELECT ROUND(dur_ns/1e6,2) FROM runtime LIMIT 1 OFFSET (SELECT CAST(COUNT(*)*0.95 AS INT) FROM runtime)) as p95_ms,
  ROUND(MIN(dur_ns)/1e6, 2) as min_ms,
  ROUND(MAX(dur_ns)/1e6, 2) as max_ms,
  ROUND(1000.0/(AVG(dur_ns)/1e6), 1) as fps
FROM runtime;
"
```

### Step 3: Top Zones (runtime only, noise excluded)

```sql
sqlite3 -header -column profile.sqlite "
WITH frames AS (
  SELECT ROW_NUMBER() OVER (ORDER BY e.start) as n,
         e.start, e.end, (e.end - e.start) as dur_ns
  FROM NVTX_EVENTS e LEFT JOIN StringIds s ON e.textId = s.id
  WHERE COALESCE(e.text, s.value) = 'App Update' AND e.end IS NOT NULL
),
frame_med AS (
  SELECT dur_ns as med FROM frames ORDER BY dur_ns
  LIMIT 1 OFFSET (SELECT COUNT(*)/2 FROM frames)
),
runtime_frames AS (
  -- Keep only frames classified as steady-state runtime. Do not collapse to
  -- one min/max span, because loading spikes can occur between runtime frames.
  SELECT f.start, f.end
  FROM frames f, frame_med m
  WHERE f.dur_ns <= m.med * 5
)
SELECT
  COALESCE(e.text, s.value) as zone_name,
  COUNT(*) as cnt,
  ROUND(AVG(e.end - e.start)/1e6, 3) as avg_ms,
  ROUND(SUM(e.end - e.start)/1e6, 2) as total_ms,
  ROUND(MAX(e.end - e.start)/1e6, 3) as max_ms
FROM NVTX_EVENTS e
LEFT JOIN StringIds s ON e.textId = s.id
WHERE EXISTS (
    SELECT 1 FROM runtime_frames rf
    WHERE e.start >= rf.start AND e.start < rf.end
  )
  AND e.end IS NOT NULL AND (e.end - e.start) > 0
  AND COALESCE(e.text, s.value) NOT LIKE '%Thread waiting%'
  AND COALESCE(e.text, s.value) NOT LIKE 'Carbonite::%'
  AND COALESCE(e.text, s.value) NOT LIKE 'carb::%'
  AND COALESCE(e.text, s.value) NOT IN ('Executing task','Running fiber')
GROUP BY zone_name
HAVING total_ms > 1
ORDER BY total_ms DESC LIMIT 30;
"
```

### SQLite Schema Quick Reference

| Table | Use |
|-------|-----|
| `NVTX_EVENTS` | NVTX ranges/markers. **No `name` column** — use `text` (inline) or join `textId→StringIds.id`. |
| `StringIds` | String lookup (`id` → `value`) |
| `CUPTI_ACTIVITY_KIND_KERNEL` | CUDA kernel launches (**empty for Kit/RTX apps — normal**) |
| `TARGET_INFO_GPU` | GPU hardware info |
| `TARGET_INFO_SYSTEM_ENV` | System environment |

---

## Analysis Path B: Tracy Statistics (primary path for .tracy files)

Use Tracy GUI Statistics for `.tracy` files when the goal is hotspot ranking, regression analysis, or optimization comparison.

1. Open the `.tracy` file in Tracy Profiler.
2. Open **View -> Statistics**.
3. Drag-select the steady-state interval on the timeline and set **Range Limit**.
4. Record Mean, Median, Min, Max, Std Dev, Count, and Total Time for key zones.
5. For before/after comparisons, use the same hardware, scene, parameters, and equal-length steady-state Range Limits.

Do not compare a single frame unless the issue is known to occur in one frame and is reproduced across multiple runs.

---

## Analysis Path C: Tracy CSV (automated fallback for .tracy files)

```bash
csvexport profile.tracy > zones.csv
```

Inspect the header before scripting against `csvexport` output. Tracy versions and builds can differ:
- Guide examples use `name`, `mean`, `count`, and `total_time`.
- Other builds emit nanosecond-specific names such as `total_ns`, `counts`, and `mean_ns`.

Normalize the column names in scripts instead of assuming one schema.

Data is **pre-aggregated** — one row per unique zone, covering the entire trace (no phase separation).

```bash
head -1 zones.csv
```

> **Tracy CSV limitation:** No per-invocation timestamps — only aggregates. For phase-aware analysis, prefer the nsys SQLite path.

---

## Two-Version Comparison

### With nsys SQLite (recommended)

```bash
nsys export --type=sqlite -o v1.sqlite v1.nsys-rep --force-overwrite=true
nsys export --type=sqlite -o v2.sqlite v2.nsys-rep --force-overwrite=true
```

Run the overview/frames/zones queries (Steps 2-3) on both databases, save outputs, then compare.

### With Tracy CSV

```bash
csvexport v1.tracy > v1_zones.csv
csvexport v2.tracy > v2_zones.csv
```

Compare with Python:

```python
import csv

def number(row, *names):
    for name in names:
        value = row.get(name)
        if value not in (None, ""):
            return float(value)
    return 0.0

def load_zones(path):
    zones = {}
    with open(path) as f:
        for row in csv.DictReader(f):
            name = row.get('name') or row.get('zone_name')
            if not name:
                continue
            zones[name] = {
                'total_ms': number(row, 'total_ns', 'total_time') / 1e6,
                'count': int(number(row, 'counts', 'count')),
                'mean_ms': number(row, 'mean_ns', 'mean') / 1e6,
            }
    return zones

v1, v2 = load_zones('v1_zones.csv'), load_zones('v2_zones.csv')

diffs = []
for name in set(v1) | set(v2):
    t1 = v1.get(name, {}).get('total_ms', 0)
    t2 = v2.get(name, {}).get('total_ms', 0)
    if t1 > 0.1 or t2 > 0.1:  # skip trivial zones
        diffs.append((name, t1, t2, t2 - t1))

print("=== Top Regressions (slower in v2) ===")
for name, t1, t2, d in sorted(diffs, key=lambda x: -x[3])[:15]:
    print(f"  {d:+10.1f}ms  {name}  (v1={t1:.1f}, v2={t2:.1f})")

print("\n=== Top Improvements (faster in v2) ===")
for name, t1, t2, d in sorted(diffs, key=lambda x: x[3])[:15]:
    print(f"  {d:+10.1f}ms  {name}  (v1={t1:.1f}, v2={t2:.1f})")
```

### Report Structure

1. **Overall metrics** — total duration, frame count per version
2. **Phase comparison** — startup time, loading frames count/duration
3. **Frame analysis** — mean frametime, P50, P95, FPS (runtime frames only)
4. **Top regressions** — zones slower in v2, ranked by absolute ms impact
5. **Top improvements** — zones faster in v2
6. **New/removed zones** — zones appearing only in one version
7. **Root cause analysis** — explain *why* the change happened

The goal: not just "FPS dropped 10%" but "FPS dropped 10% because
`rtUpdatePipeline` added 59ms/frame in v2, a new shader pipeline
recompilation step not present in v1."

---