Semantic Labels
When to Use
Use this skill when the user asks to label objects, assign semantic class/label metadata, configure semantic segmentation labels, or author SemanticsAPI overrides.
Inputs
Resolve inputs in this order: existing repository files and referenced snippets, explicit user request, then broader agent context.
- Target API surface: Python, C/C++, USD, or a combination.
- Prims to label, desired semantic class and label values, and whether labels should be authored directly or as override layers.
- Requested ground-truth outputs such as
SemanticSegmentation or SemanticIdMap when the workflow includes rendering.
- Existing composition context: source asset path, inline USDA, sublayer, or runtime-loaded stage.
- Repository source snippets referenced below. Treat these snippets as the API source of truth.
Prerequisites
- Use an ovrtx checkout that contains the referenced examples and docs tests.
- Read the relevant
> **Source:** snippet before writing or explaining API usage.
- Use
camera-outputs-rt2 when the request is about which segmentation outputs to add to a RenderProduct.
- Use
reading-render-output when the request is about mapping the segmentation image or ID map after rendering.
Instructions
- Identify the prims to label, class/label values, and whether the label should be authored directly or as a composed override.
- Read the matching USD or Python source snippet before writing semantic metadata.
- Preserve USD
SemanticsAPI instance names, inheritance behavior, and any source-layer composition pattern.
- Keep label authoring, RenderProduct output selection, and output readback separate unless the user asks for the full segmentation workflow.
- When changing code, run the narrow docs test or example that owns the snippet whenever practical.
Output Format
- For explanations, cite the relevant API names, source snippets, and caveats.
- For code changes, summarize the files changed, snippets affected, and validation run.
Scripts
This skill has no scripts.
Limitations
- The referenced snippets remain the source of truth; update or add tested snippets before documenting new API usage.
Overview
Use USD SemanticsAPI metadata to assign semantic class and label values to scene prims that should appear in semantic segmentation or ground-truth annotation outputs. SemanticsAPI is a multi-apply schema: apply the class and label instances to an ancestor prim, and child prims inherit those semantics unless they author more specific semantics.
For runtime-composed scenes, keep the source asset untouched. Build one inline root layer that sublayers the source scene, then author over prims for the objects you want to label.
USDA
Label existing composed prims
Source: tests/docs/usd/data/semantic_label_overrides.usda snippet doc-semantic-label-overrides
Source: tests/docs/data/ovrtx-test-base-semantic-labels.usda snippet doc-test-base-semantic-class-layer
Python
Use the same inline-root sublayer pattern as the loading skill: build the USDA string with subLayers, add SemanticsAPI over prims for the labeled objects, add SemanticSegmentation and SemanticIdMap outputs to a RenderProduct, then pass that string to renderer.open_usd_from_string().
Source: tests/docs/python/test_semantic_labels.py snippet doc-semantic-class-overrides-python
Interpret rendered semantic outputs
SemanticIdMap maps renderer semantic IDs to semantic strings such as class: logo;. SemanticSegmentation is an image whose pixel values are those renderer semantic IDs.
Source: tests/docs/python/test_semantic_labels.py snippet doc-interpret-semantic-segmentation-python
C
Label existing composed prims
Source: tests/docs/c/test_semantic_labels.cpp snippet doc-semantic-class-overrides-c
Interpret rendered semantic outputs
Source: tests/docs/c/test_semantic_labels.cpp snippet doc-interpret-semantic-segmentation-c
Key USD Fields
| Field |
Purpose |
prepend apiSchemas = ["SemanticsAPI:label", "SemanticsAPI:class"] |
Applies the label and class SemanticsAPI instances to a prim. |
semantic:class:params:semanticType |
Set to class. |
semantic:class:params:semanticData |
The class value, such as robot, vehicle, or person. |
semantic:label:params:semanticType |
Set to label. |
semantic:label:params:semanticData |
The object-specific label value, such as the asset or instance name. |
Troubleshooting
- Label the object root, not the RenderProduct or RenderVar. Semantics are scene metadata consumed by annotation outputs.
- Use
over prims when labeling objects from a sublayered source scene. This avoids modifying the original asset.
- Keep semantic label attributes constant. Sensor RTX semantic label attributes are not time-sampled.
- Apply labels at the right granularity. A top-level asset label is inherited by children; part-level labels should be added only when you need finer segmentation classes.
References
- Use the
> **Source:** directives in this skill to locate tested snippets before reusing API patterns.
- Keep related skills, docs, and snippets synchronized when changing the workflow.
1---2name: semantic-labels3description: Assigning USD SemanticsAPI class and label metadata for semantic segmentation and ground-truth annotation outputs. Use when user asks to label objects, assign semantic class/label metadata, configure semantic segmentation labels, or author SemanticsAPI overrides.4license: LicenseRef-NvidiaProprietary5---67# Semantic Labels89## When to Use1011Use this skill when the user asks to label objects, assign semantic class/label metadata, configure semantic segmentation labels, or author SemanticsAPI overrides.1213## Inputs1415Resolve inputs in this order: existing repository files and referenced snippets, explicit user request, then broader agent context.1617- Target API surface: Python, C/C++, USD, or a combination.18- Prims to label, desired semantic class and label values, and whether labels should be authored directly or as override layers.19- Requested ground-truth outputs such as `SemanticSegmentation` or `SemanticIdMap` when the workflow includes rendering.20- Existing composition context: source asset path, inline USDA, sublayer, or runtime-loaded stage.21- Repository source snippets referenced below. Treat these snippets as the API source of truth.2223## Prerequisites2425- Use an ovrtx checkout that contains the referenced examples and docs tests.26- Read the relevant `> **Source:**` snippet before writing or explaining API usage.27- Use `camera-outputs-rt2` when the request is about which segmentation outputs to add to a RenderProduct.28- Use `reading-render-output` when the request is about mapping the segmentation image or ID map after rendering.2930## Instructions31321. Identify the prims to label, class/label values, and whether the label should be authored directly or as a composed override.332. Read the matching USD or Python source snippet before writing semantic metadata.343. Preserve USD `SemanticsAPI` instance names, inheritance behavior, and any source-layer composition pattern.354. Keep label authoring, RenderProduct output selection, and output readback separate unless the user asks for the full segmentation workflow.365. When changing code, run the narrow docs test or example that owns the snippet whenever practical.3738## Output Format3940- For explanations, cite the relevant API names, source snippets, and caveats.41- For code changes, summarize the files changed, snippets affected, and validation run.4243## Scripts4445This skill has no scripts.4647## Limitations4849- The referenced snippets remain the source of truth; update or add tested snippets before documenting new API usage.5051## Overview5253Use USD `SemanticsAPI` metadata to assign semantic class and label values to scene prims that should appear in semantic segmentation or ground-truth annotation outputs. `SemanticsAPI` is a multi-apply schema: apply the `class` and `label` instances to an ancestor prim, and child prims inherit those semantics unless they author more specific semantics.5455For runtime-composed scenes, keep the source asset untouched. Build one inline root layer that sublayers the source scene, then author `over` prims for the objects you want to label.5657## USDA5859### Label existing composed prims6061> **Source:** `tests/docs/usd/data/semantic_label_overrides.usda` snippet `doc-semantic-label-overrides`62>63> **Source:** `tests/docs/data/ovrtx-test-base-semantic-labels.usda` snippet `doc-test-base-semantic-class-layer`6465## Python6667Use the same inline-root sublayer pattern as the loading skill: build the USDA string with `subLayers`, add `SemanticsAPI` `over` prims for the labeled objects, add `SemanticSegmentation` and `SemanticIdMap` outputs to a RenderProduct, then pass that string to `renderer.open_usd_from_string()`.6869> **Source:** `tests/docs/python/test_semantic_labels.py` snippet `doc-semantic-class-overrides-python`7071### Interpret rendered semantic outputs7273`SemanticIdMap` maps renderer semantic IDs to semantic strings such as `class: logo;`. `SemanticSegmentation` is an image whose pixel values are those renderer semantic IDs.7475> **Source:** `tests/docs/python/test_semantic_labels.py` snippet `doc-interpret-semantic-segmentation-python`7677## C7879### Label existing composed prims8081> **Source:** `tests/docs/c/test_semantic_labels.cpp` snippet `doc-semantic-class-overrides-c`8283### Interpret rendered semantic outputs8485> **Source:** `tests/docs/c/test_semantic_labels.cpp` snippet `doc-interpret-semantic-segmentation-c`8687## Key USD Fields8889| Field | Purpose |90|---|---|91| `prepend apiSchemas = ["SemanticsAPI:label", "SemanticsAPI:class"]` | Applies the `label` and `class` SemanticsAPI instances to a prim. |92| `semantic:class:params:semanticType` | Set to `class`. |93| `semantic:class:params:semanticData` | The class value, such as `robot`, `vehicle`, or `person`. |94| `semantic:label:params:semanticType` | Set to `label`. |95| `semantic:label:params:semanticData` | The object-specific label value, such as the asset or instance name. |9697## Troubleshooting9899- Label the object root, not the RenderProduct or RenderVar. Semantics are scene metadata consumed by annotation outputs.100- Use `over` prims when labeling objects from a sublayered source scene. This avoids modifying the original asset.101- Keep semantic label attributes constant. Sensor RTX semantic label attributes are not time-sampled.102- Apply labels at the right granularity. A top-level asset label is inherited by children; part-level labels should be added only when you need finer segmentation classes.103104## References105106- Use the `> **Source:**` directives in this skill to locate tested snippets before reusing API patterns.107- Keep related skills, docs, and snippets synchronized when changing the workflow.
nvidia-omniverse