Validation Difference GIFs
Generate per-camera GIF animations showing the pixel-wise difference between captured benchmark images and their golden references. Useful for debugging validation tolerance failures.
Prerequisites
- ImageMagick (
composite,convert) must be available onPATH. - A completed validation capture run with matching directory structure to the golden data.
Paths
Captures and golden data live under the Isaac Sim build release directory:
standalone_examples/benchmarks/validation/captures/<run_name>/
standalone_examples/benchmarks/validation/golden_data/<benchmark_name>/
Both share a parallel directory tree (e.g. Robots/Robot_0/.../front_hawk/left/camera_left/rgb/).
Usage
Find the latest capture
ls -td standalone_examples/benchmarks/validation/captures/benchmark_robots_nova_carter_ros2_*/ | head -1
Run the script
The bundled script is resolved via ${CLAUDE_SKILL_DIR}; falls back to the canonical repo path skills/validation-diff-gifs/scripts/:
bash "${CLAUDE_SKILL_DIR:-skills/validation-diff-gifs}/scripts/generate_diff_gifs.sh" \
<captured_run_dir> \
<golden_benchmark_dir> \
[amplify] [fps]
| Argument | Default | Description |
|---|---|---|
captured_run_dir |
(required) | Root of the capture run |
golden_benchmark_dir |
(required) | Root of the golden data for the benchmark |
amplify |
10 |
Multiply pixel differences by this factor for visibility |
fps |
5 |
Frame rate for the output GIF |
Example
LATEST=$(ls -td standalone_examples/benchmarks/validation/captures/benchmark_robots_nova_carter_ros2_*/ | head -1)
GOLDEN="standalone_examples/benchmarks/validation/golden_data/benchmark_robots_nova_carter_ros2"
bash "${CLAUDE_SKILL_DIR:-skills/validation-diff-gifs}/scripts/generate_diff_gifs.sh" "$LATEST" "$GOLDEN"
Output
For each rgb/ directory under the capture, a diff_animation.gif is written alongside the captured PNGs:
<captured_run_dir>/Robots/.../front_hawk/left/camera_left/rgb/diff_animation.gif
- Black pixels = identical between captured and golden
- Brighter pixels = larger difference (amplified by the
amplifyfactor)
Interpreting results
- Uniform low-level noise across the frame → rendering non-determinism (likely acceptable)
- Bright regions concentrated on object edges → sub-pixel movement differences
- Entire frame bright → wrong timestamp match or completely different camera pose
- One camera consistently worse than others → possible per-camera issue (tick rate, initialization)