# Nvtx Python

> Profile Python functions with NVTX in non-Kit environments (Isaac Lab 3.0+ standalone, any Python app without Carbonite). Uses a bundled PYTHONPATH-scoped sitecustomize.py with sys.setprofile hook, NVTX push/pop ranges, module include/exclude filtering, and Nsight Systems integration. Use when CARB_PROFILING_PYTHON doesn't work (no Kit/Carbonite runtime), when profiling standalone Isaac Lab scripts, or when you need per-function Python tracing in nsys captures outside Kit.

- Skill: `nvidia/nvtx-python` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add nvidia/nvtx-python`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/nvtx-python/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia/nvtx-python

---


# NVTX Python Function Profiling (Non-Kit Environments)

For Isaac Lab 3.0+ standalone mode and other Python apps without Kit/Carbonite, `CARB_PROFILING_PYTHON` doesn't work. This skill uses the bundled `scripts/sitecustomize.py` helper to install a `sys.setprofile()` hook with NVTX ranges.

Do not write into an environment's existing `sitecustomize.py`. Load the bundled helper with `PYTHONPATH` so disabling it is just unsetting `PYTHONPATH` or `NVTX_PROFILE_PYTHON`.

## Setup

```bash
# From the Isaac Lab directory (or any Python project with uv/pip)
uv pip install nvtx

# Resolve this skill's directory, then put its scripts/ directory on PYTHONPATH.
# From this repository, the default below points at the bundled helper.
NVTX_SKILL_DIR="${NVTX_SKILL_DIR:-$PWD/.agents/skills/nvtx-python}"
test -f "$NVTX_SKILL_DIR/scripts/sitecustomize.py"
export PYTHONPATH="$NVTX_SKILL_DIR/scripts:${PYTHONPATH:-}"
```

## Environment Variables

| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `NVTX_PROFILE_PYTHON` | Set to `1` to enable | (disabled) | `1` |
| `NVTX_PROFILE_INCLUDE` | Module prefixes to trace (comma-separated). Empty = all. | (all) | `isaaclab,skrl,torch` |
| `NVTX_PROFILE_EXCLUDE` | Module prefixes to exclude (comma-separated). | `importlib` | `importlib,threading,nvtx` |

## Usage with Nsight Systems

```bash
# Capture all Python modules
NVTX_SKILL_DIR="${NVTX_SKILL_DIR:-$PWD/.agents/skills/nvtx-python}"
PYTHONPATH="$NVTX_SKILL_DIR/scripts:${PYTHONPATH:-}" \
NVTX_PROFILE_PYTHON=1 \
nsys profile -t nvtx,cuda,osrt \
uv run python scripts/reinforcement_learning/skrl/train.py \
  --task=Isaac-Velocity-Flat-Anymal-C-v0 --num_envs=1024 --max_iterations=10

# Capture specific modules only (recommended — reduces overhead)
NVTX_SKILL_DIR="${NVTX_SKILL_DIR:-$PWD/.agents/skills/nvtx-python}"
PYTHONPATH="$NVTX_SKILL_DIR/scripts:${PYTHONPATH:-}" \
NVTX_PROFILE_PYTHON=1 NVTX_PROFILE_INCLUDE=isaaclab,skrl \
nsys profile -t nvtx,cuda,osrt \
uv run python scripts/reinforcement_learning/skrl/train.py \
  --task=Isaac-Velocity-Flat-Anymal-C-v0 --num_envs=1024 --max_iterations=10
```

## Performance Considerations

- **Every function call/return fires a callback** — significant overhead when tracing all modules
- Use `NVTX_PROFILE_INCLUDE` to limit scope to modules of interest
- To disable: unset `NVTX_PROFILE_PYTHON` or remove the skill's `scripts/` directory from `PYTHONPATH`

## Caveat: Isaac Lab / Kit-based hosts clobber `sys.setprofile`

Isaac Lab 3.0+ standalone *does* boot Kit/Carb under the hood via `AppLauncher`. Just running `from isaaclab.app import AppLauncher` (or any import that transitively pulls `carb` / `isaacsim`) registers Carb's own Python profile callback during module load, **silently overwriting the hook this skill installed at interpreter startup**. The resulting `.nsys-rep` then shows Python NVTX only for the brief window between interpreter startup and the first Carb-pulling import — typically just `argparse`, then nothing.

**Fix:** call `sitecustomize.install()` after the import to re-arm. `install()` is idempotent, re-reads `NVTX_PROFILE_INCLUDE` / `NVTX_PROFILE_EXCLUDE` each call, and re-installs both `sys.setprofile` and `threading.setprofile`.

```python
import sys
print(f"[NVTX] before AppLauncher: {sys.getprofile()!r}", flush=True)  # likely None
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app
print(f"[NVTX] after AppLauncher:  {sys.getprofile()!r}", flush=True)  # still None

import sitecustomize
sitecustomize.install()
print(f"[NVTX] re-installed:       {sys.getprofile()!r}", flush=True)  # callback restored
```

The probes are how you confirm the issue and the fix in one run. Drop them once verified.

### Limitations even after re-install

- `sys.setprofile` is **per-thread**. `threading.setprofile` only catches threads created via Python's `threading` module — Kit / PhysX / Hydra / Fabric / TBB native threads stay invisible. Most of `env.step()` runs in C++ on those threads, so even with the hook re-armed you'll see relatively few Python NVTX zones during the step loop.
- Tracing every module floods the trace and can swamp nsys's NVTX backend; **always set `NVTX_PROFILE_INCLUDE`** to your own packages (e.g. `simulation,isaaclab.envs,isaaclab_tasks`) for Isaac Lab workloads.
- For native-side coverage of Kit/PhysX/Hydra, use `CARB_PROFILING_PYTHON=1` + Carb's NVTX backend (see `profiling` skill) instead of, or alongside, this skill.
- For a clean, low-overhead view of specific hot paths, drop the auto-tracer entirely and decorate the functions you care about with `@nvtx.annotate` (`env.step`, IK solve, observation manager, camera readout).

## Alternative: nsys Built-in Python Tracing

If you know exactly which modules/functions to trace, nsys has a built-in option:
```bash
nsys profile --python-functions-trace=... <command>
```
See the [Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html) for details.