＋ Add to collection

Install (CLI)

npx skillmds add nvidia/tao-train-depth-anything-v2

Or Copy and Paste in Your Agent chat

Please install the Agent Skill "nvidia/tao-train-depth-anything-v2" for me from skillmd.com. A SkillMD skill is a SKILL.md file — plain markdown plus a few companion files — that you save into my skills directory. Use whichever fetch route fits your setup: (1) if you have a terminal, run npx -y skillmds add nvidia/tao-train-depth-anything-v2; (2) otherwise fetch the file manifest at https://api.skillmd.com/api/skills/nvidia/tao-train-depth-anything-v2/bundle?format=manifest and download each listed file from its "url" into tao-train-depth-anything-v2/ in your skills directory; (3) if the SkillMD MCP tools are connected, call skillmd_download with slug "nvidia/tao-train-depth-anything-v2" and save the returned files under tao-train-depth-anything-v2/. Show me the SKILL.md before saving if you'd like me to review it. If your skills directory is read-only (e.g. the Claude desktop app), save the files to a working folder instead and tell me — I'll add them through my app's skills settings.

README badge

Show this skill's registry status in your repo:

[![SkillMD](https://skillmd.com/badge/nvidia/tao-train-depth-anything-v2.svg)](https://skillmd.com/skills/nvidia/tao-train-depth-anything-v2)

tao-train-depth-anything-v2

Name: tao-train-depth-anything-v2
Rating: 4.2 (5 reviews)
Author: NVIDIA

nvidia/tao-train-depth-anything-v2 · Agent Skill (multi-file)

by NVIDIA · bundle

Published Jun 29, 2026 · Last updated Jul 6, 2026

Train, evaluate, export, and run inference for monocular depth estimation models using Metric Depth Anything v2 or Relative Depth Anything architectures via the TAO toolkit.

SKILL.md

Depth Net Mono

Monocular depth estimation using Metric Depth Anything v2 or Relative Depth Anything architectures. Predicts per-pixel depth from single RGB images.

Pretrained checkpoint loading varies by model variant and use case — see the Pretrained checkpoint loading — use case matrix in references/parameters.md.

The mono and stereo skills both invoke the unified TAO depth_net CLI inside the container; the mono/stereo family is selected via model.model_type (see references/parameters.md).

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-depth-anything-v2.md first. The deploy spec template lives in this skill's references/spec_template_deploy.yaml.

PyT actions packaged by this model skill: train, evaluate, inference, export, and quantize. The PyT depth_net entrypoint does not accept a PyT-side gen_trt_engine action in the current TAO image. The gen_trt_engine action metadata must run with the TAO Deploy container, and the deploy workflow remains the deploy-specific entrypoint.

Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skill_info.yaml and resolve the run override from either an explicit automl_policy value or the user's workflow request. Use automl_policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl_policy: off for this run only. When automl_policy: on, automl_enabled: true, and both schemas/train.schema.json and references/spec_template_train.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skill_dir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl_policy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as evaluate, inference, export, and deploy flows stay in this model skill. The per-run automl_policy override does not change model metadata.

Workflow

Prerequisites — data accessibility

Your dataset (RGB images + GT depth files) must be reachable from inside the container:

SDK runner: place files at the S3 paths the runner resolves (the S3_TRAIN / S3_EVAL placeholders shown in Typical Spec Overrides). The runner handles S3 → container-path mounting transparently.
Direct docker run (e.g. local testing): mount the host dataset root read-only at the same in-container path:

docker run ... -v <host_data_root>:<host_data_root>:ro <container> ...

The same accessibility requirement applies to the <output_dir> written by all actions.

Step 1 — Annotation file

Per-line annotation file referenced by data_sources[*].data_file:

Columns	Format	Use
1	`<image>`	Mono inference (no GT)
2	`<image> <gt_depth>`	Mono with GT

Do not pass stereo annotation rows such as <left_image> <right_image> <gt_depth> directly to mono train/evaluate/inference. If only a stereo depth dataset is available, derive a mono annotation file by keeping the left image and GT depth columns, then mount or stage the image/depth archive at the same container paths referenced by that derived annotation file.

If you already have one, point to it. Otherwise generate via depth_net convert:

depth_net convert -e <convert_spec.yaml>

convert_spec.yaml template:

results_dir: <directory where generated annotation files are written>
data_root: <directory whose immediate children are scene/sample folders that contain your image+depth files; convert walks data_root recursively but expects per-scene subdirectories at one level below>
image_dir_pattern: [<substring matching left/RGB image paths>]
depth_dir_pattern: [<substring matching GT depth paths>]
image_extension: ''     # optional .endswith filter, e.g. '.jpg'
depth_extension: ''     # optional, swapped during depth derivation, e.g. '.png'
split_ratio: 0.0        # 0.0/1.0 = test-only; 0.8 = 80/20 train+val

convert walks data_root recursively, selects paths whose path-string contains all substrings in image_dir_pattern (AND-filter), then derives the depth path by replacing image_dir_pattern[0] with depth_dir_pattern[0] and image_extension with depth_extension. Inspect your dataset's directory layout and identify the substring distinguishing RGB images from depth files (e.g. rgb_ vs sync_depth_).

data_root must point at the parent that contains the per-scene subdirectories (e.g. for NYU eval, use /data/nyu_v2/eval/test, not /data/nyu_v2/eval/test/bathroom — the latter limits the walk to a single scene). Always include the leading dot in image_extension / depth_extension (e.g. '.jpg' not 'jpg'); the substring swap is form-sensitive and a mismatch silently corrupts derived paths.

Step 2 — Pair `model_type` and `dataset_name` based on your data

Default — generic class for each task:

Data category	`model_type`	`dataset_name`
Disparity-encoded data (pixels)	`RelativeDepthAnything`	`RelativeMonoDataset`
Metric depth (meters)	`MetricDepthAnything`	`MetricMonoDataset`
Mono inference (no GT, any image)	matches train choice	`RelativeMonoDataset` or `MetricMonoDataset`

Dataset-specific class — switch when the data needs preprocessing the generic class does not perform:

Special case	`model_type`	`dataset_name`	What the class adds
NYU `sync_depth_*.png` (raw uint16 millimetres) — relative	`RelativeDepthAnything`	`NYUDV2Relative`	mm→m unit conversion + Eigen evaluation crop
NYU `sync_depth_*.png` (raw uint16 millimetres) — metric	`MetricDepthAnything`	`NYUDV2`	same

Using a generic class on data that requires unit conversion (e.g. raw NYU uint16 PNGs) results in an empty valid mask and silent train_loss = NaN. Match the class to your data's encoding.

For relative mono data (RelativeMonoDataset or NYUDV2Relative), leave dataset.min_depth and dataset.max_depth unset or set both to null. Non-null metric depth ranges are passed into the relative dataset constructor and fail with BaseRelativeMonoDataset.__init__() got an unexpected keyword argument 'min_depth'.

Step 3 — Write spec yaml from Typical Spec Overrides

Copy the action block from Typical Spec Overrides (references/spec-overrides.md). Replace:

model.model_type from Step 2
dataset.<...>.data_sources[*].dataset_name from Step 2
data_sources[*].data_file with the path from Step 1 (S3 path under SDK runner, host path for direct docker)
For metric finetune: additionally apply the Metric Variant Finetuning Recipe in references/finetuning-recipes.md.

For mono training set train.precision: fp32 (recommended) or bf16 (Ampere SM80+, alternative).

Step 4 — Run

Create writable home/cache directories inside the mounted output path before using --user. Some TAO containers do not have an /etc/passwd entry for the host UID, and PyTorch / matplotlib need writable cache paths when running as that UID.

mkdir -p <output_dir>/home \
         <output_dir>/.cache/matplotlib \
         <output_dir>/.cache/torchinductor \
         <output_dir>/.cache/xdg

docker run --gpus 'device=0' --shm-size 16G --ipc=host \
  --user "$(id -u):$(id -g)" \
  -e USER="$(id -un)" \
  -e LOGNAME="$(id -un)" \
  -e HOME=<output_dir>/home \
  -e MPLCONFIGDIR=<output_dir>/.cache/matplotlib \
  -e TORCHINDUCTOR_CACHE_DIR=<output_dir>/.cache/torchinductor \
  -e XDG_CACHE_HOME=<output_dir>/.cache/xdg \
  -v <data_root>:<data_root>:ro \
  -v <output_dir>:<output_dir> \
  <container> \
  depth_net <action> -e <spec.yaml>

Without --user "$(id -u):$(id -g)" the container writes outputs as nobody:nogroup, blocking host-side cleanup and retry.

Step 5 — Verify

Container exit code 0
status.json kpi block populated
For train: inspect per-step train_loss directly — the entrypoint reports Execution status: PASS even when train_loss = NaN (see the Metric Variant Finetuning Recipe → Sanity-run PASS criteria in references/finetuning-recipes.md)
For evaluate / inference: artifacts under results_dir

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-depth-anything-v2.md first. Deploy spec templates live in this skill's references/ folder with the spec_template_deploy_*.yaml prefix.

Training Requirements

Valid dataset_name values for mono data_sources (case-insensitive): ThreeDVLM, FSD, NvCLIP, IssacStereo, Crestereo, Middlebury, NYUDV2, NYUDV2Relative, RelativeMonoDataset, MetricMonoDataset. NYUDV2 carries metric depth GT (meters) — pair with MetricDepthAnything; NYUDV2Relative is the same data with relative-depth conventions — pair with RelativeDepthAnything.
Monitoring metric: val/d1, val/loss
For AutoML sanity runs on the packaged relative-depth smoke data, use val/d1 as the primary monitor. val/loss can be emitted as NaN even when the trainer exits successfully and writes a usable checkpoint, so it is not a reliable AutoML objective unless the run's status metrics show a finite value.

Per-Action Dataset Requirements

Action	Spec Key	Source	Files	List?
evaluate	dataset.test_dataset.data_sources	eval_dataset	data_file: annotations.txt + dataset_name	Yes
inference	dataset.infer_dataset.data_sources	inference_dataset	data_file: annotations.txt + dataset_name	Yes
quantize	dataset.train_dataset.data_sources	train_datasets	data_file: annotations.txt + dataset_name	Yes
quantize	dataset.val_dataset.data_sources	eval_dataset	data_file: annotations.txt + dataset_name	Yes
quantize	dataset.quant_calibration_dataset.images_dir	train_datasets	images.tar.gz	No
train	dataset.train_dataset.data_sources	train_datasets	data_file: annotations.txt + dataset_name	Yes
train	dataset.val_dataset.data_sources	eval_dataset	data_file: annotations.txt + dataset_name	Yes

Typical Spec Overrides

Data source overrides are mandatory for every action — construct data source paths from the Per-Action Dataset Requirements table above and include them in spec_overrides. Each data_sources entry is a dict with two mandatory fields: data_file and dataset_name. See references/spec-overrides.md for the full per-action override blocks (train, evaluate, export, inference, quantize), the S3_TRAIN / S3_EVAL placeholders, the relative-variant precision recommendation, and the quantize known-issue note.

Eval Dataset

Optional. Val dataset configured via dataset.val_dataset.data_sources (each entry needs data_file and dataset_name).

Important Parameters

See references/parameters.md for the full parameter glossary (model, train, dataset, export, and inference keys with options, defaults, and sources) and the Pretrained checkpoint loading — use case matrix.

Finetuning Recipes

See references/finetuning-recipes.md for:

Relative Variant Finetuning Recipe — finetune from a TAO-trained RelativeDepthAnything checkpoint (lr 5e-6, LambdaLR, sanity-vs-convergent guidance, deploy LSQ alignment note).
Metric Variant Finetuning Recipe — checkpoint compatibility, required overrides, the dataset normalization block (normalize_depth/min_depth/max_depth) required in train AND export specs, trainer-enforced defaults, precision, the 1-epoch sanity-run override, and the Sanity-run PASS criteria with the NaN-mitigation order.

Multi-GPU / Multi-Node

Launch method: Lightning-managed (single python process, Lightning spawns workers).

Spec Key	Description	Default
`train.num_gpus`	Number of GPUs	1
`train.gpu_ids`	GPU device indices	[0]
`train.num_nodes`	Number of nodes	1
`train.distributed_strategy`	`ddp` or `fsdp`	`ddp`

ddp with activation checkpointing: find_unused_parameters=False
ddp without: find_unused_parameters=True
fsdp forces precision to FP16

Multi-node env vars (set by orchestrator): WORLD_SIZE, NODE_RANK, MASTER_ADDR, MASTER_PORT, NUM_GPU_PER_NODE.

Export / TRT Defaults

TRT data types: FP32, BF16 (Ampere SM80+). FP16 is not supported for the ViT-L mono backbone.
Fresh-install TRT precision: fp32. BF16 is supported on Ampere SM80+ hardware, but keep smoke tests on FP32 unless the user explicitly requests BF16.

Hardware

Minimum 1 GPU(s), recommended 2 GPU(s). 24GB+ VRAM per GPU. ViT-Large encoder is memory intensive. Use fp32 (recommended) or bf16 (Ampere SM80+, alternative) for training. Activation checkpointing is available for larger inputs.

Error Patterns

See references/troubleshooting.md for the full error-pattern catalog (depth range mismatch, relative dataset rejecting min_depth, missing pretrained weights, encoder key location, dataset_name not in struct, depth_net_mono not found, metric variant hyperparameter sourcing, and export ONNX overwrite).

Spec Param / Parent Model Inference

See references/spec-param-inference.md for the model-specific inference mappings (the TAO Core depth_net_mono.config.json action table), checkpoint-file naming under <results_dir>/train/, the dn_model_latest.pth policy, the parent-gen_trt_engine rationale, and the parent_model / parent_job_id resolution rules.

Deployment

tao-deploy-depth-anything-v2

---
name: tao-train-depth-anything-v2
description: Train, evaluate, export, and run inference for monocular depth estimation models using Metric Depth Anything v2 or Relative Depth Anything architectures via the TAO toolkit.
license: Apache-2.0
---

# Depth Net Mono

Monocular depth estimation using Metric Depth Anything v2 or Relative Depth Anything architectures. Predicts per-pixel depth from single RGB images.

Pretrained checkpoint loading varies by model variant and use case — see the **Pretrained checkpoint loading — use case matrix** in `references/parameters.md`.

The mono and stereo skills both invoke the unified TAO `depth_net` CLI inside the container; the mono/stereo family is selected via `model.model_type` (see `references/parameters.md`).

For TAO Deploy TensorRT actions (`gen_trt_engine`, TensorRT `evaluate`, and TensorRT `inference`), read `references/tao-deploy-depth-anything-v2.md` first. The deploy spec template lives in this skill's `references/spec_template_deploy.yaml`.

PyT actions packaged by this model skill: `train`, `evaluate`, `inference`, `export`, and `quantize`. The PyT `depth_net` entrypoint does not accept a PyT-side `gen_trt_engine` action in the current TAO image. The `gen_trt_engine` action metadata must run with the TAO Deploy container, and the deploy workflow remains the deploy-specific entrypoint.

## Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references/skill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Use `automl_policy: on` by default and only expose `on` / `off` in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as `automl_policy: off` for this run only. When `automl_policy: on`, `automl_enabled: true`, and both `schemas/train.schema.json` and `references/spec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.

## Workflow

### Prerequisites — data accessibility

Your dataset (RGB images + GT depth files) must be reachable from inside the container:
- **SDK runner**: place files at the S3 paths the runner resolves (the `S3_TRAIN` / `S3_EVAL` placeholders shown in **Typical Spec Overrides**). The runner handles S3 → container-path mounting transparently.
- **Direct `docker run`** (e.g. local testing): mount the host dataset root read-only at the same in-container path:

```
docker run ... -v <host_data_root>:<host_data_root>:ro <container> ...
```

The same accessibility requirement applies to the `<output_dir>` written by all actions.

### Step 1 — Annotation file

Per-line annotation file referenced by `data_sources[*].data_file`:

| Columns | Format | Use |
|---|---|---|
| 1 | `<image>` | Mono inference (no GT) |
| 2 | `<image> <gt_depth>` | Mono with GT |

Do not pass stereo annotation rows such as `<left_image> <right_image>
<gt_depth>` directly to mono train/evaluate/inference. If only a stereo depth
dataset is available, derive a mono annotation file by keeping the left image
and GT depth columns, then mount or stage the image/depth archive at the same
container paths referenced by that derived annotation file.

If you already have one, point to it. Otherwise generate via `depth_net convert`:

```
depth_net convert -e <convert_spec.yaml>
```

`convert_spec.yaml` template:

```yaml
results_dir: <directory where generated annotation files are written>
data_root: <directory whose immediate children are scene/sample folders that contain your image+depth files; convert walks data_root recursively but expects per-scene subdirectories at one level below>
image_dir_pattern: [<substring matching left/RGB image paths>]
depth_dir_pattern: [<substring matching GT depth paths>]
image_extension: ''     # optional .endswith filter, e.g. '.jpg'
depth_extension: ''     # optional, swapped during depth derivation, e.g. '.png'
split_ratio: 0.0        # 0.0/1.0 = test-only; 0.8 = 80/20 train+val
```

`convert` walks `data_root` recursively, selects paths whose path-string contains *all* substrings in `image_dir_pattern` (AND-filter), then derives the depth path by replacing `image_dir_pattern[0]` with `depth_dir_pattern[0]` and `image_extension` with `depth_extension`. Inspect your dataset's directory layout and identify the substring distinguishing RGB images from depth files (e.g. `rgb_` vs `sync_depth_`).

`data_root` must point at the parent that contains the per-scene subdirectories (e.g. for NYU eval, use `/data/nyu_v2/eval/test`, not `/data/nyu_v2/eval/test/bathroom` — the latter limits the walk to a single scene). Always include the leading dot in `image_extension` / `depth_extension` (e.g. `'.jpg'` not `'jpg'`); the substring swap is form-sensitive and a mismatch silently corrupts derived paths.

### Step 2 — Pair `model_type` and `dataset_name` based on your data

Default — generic class for each task:

| Data category | `model_type` | `dataset_name` |
|---|---|---|
| Disparity-encoded data (pixels) | `RelativeDepthAnything` | `RelativeMonoDataset` |
| Metric depth (meters) | `MetricDepthAnything` | `MetricMonoDataset` |
| Mono inference (no GT, any image) | matches train choice | `RelativeMonoDataset` or `MetricMonoDataset` |

Dataset-specific class — switch when the data needs preprocessing the generic class does not perform:

| Special case | `model_type` | `dataset_name` | What the class adds |
|---|---|---|---|
| NYU `sync_depth_*.png` (raw uint16 millimetres) — relative | `RelativeDepthAnything` | `NYUDV2Relative` | mm→m unit conversion + Eigen evaluation crop |
| NYU `sync_depth_*.png` (raw uint16 millimetres) — metric | `MetricDepthAnything` | `NYUDV2` | same |

Using a generic class on data that requires unit conversion (e.g. raw NYU uint16 PNGs) results in an empty valid mask and silent `train_loss = NaN`. Match the class to your data's encoding.

For relative mono data (`RelativeMonoDataset` or `NYUDV2Relative`), leave `dataset.min_depth` and `dataset.max_depth` unset or set both to `null`. Non-null metric depth ranges are passed into the relative dataset constructor and fail with `BaseRelativeMonoDataset.__init__() got an unexpected keyword argument 'min_depth'`.

### Step 3 — Write spec yaml from Typical Spec Overrides

Copy the action block from **Typical Spec Overrides** (`references/spec-overrides.md`). Replace:
- `model.model_type` from Step 2
- `dataset.<...>.data_sources[*].dataset_name` from Step 2
- `data_sources[*].data_file` with the path from Step 1 (S3 path under SDK runner, host path for direct docker)
- For metric finetune: additionally apply the **Metric Variant Finetuning Recipe** in `references/finetuning-recipes.md`.

For mono training set `train.precision: fp32` (recommended) or `bf16` (Ampere SM80+, alternative).

### Step 4 — Run

Create writable home/cache directories inside the mounted output path before using
`--user`. Some TAO containers do not have an `/etc/passwd` entry for the host UID,
and PyTorch / matplotlib need writable cache paths when running as that UID.

```bash
mkdir -p <output_dir>/home \
         <output_dir>/.cache/matplotlib \
         <output_dir>/.cache/torchinductor \
         <output_dir>/.cache/xdg
```

```
docker run --gpus 'device=0' --shm-size 16G --ipc=host \
  --user "$(id -u):$(id -g)" \
  -e USER="$(id -un)" \
  -e LOGNAME="$(id -un)" \
  -e HOME=<output_dir>/home \
  -e MPLCONFIGDIR=<output_dir>/.cache/matplotlib \
  -e TORCHINDUCTOR_CACHE_DIR=<output_dir>/.cache/torchinductor \
  -e XDG_CACHE_HOME=<output_dir>/.cache/xdg \
  -v <data_root>:<data_root>:ro \
  -v <output_dir>:<output_dir> \
  <container> \
  depth_net <action> -e <spec.yaml>
```

Without `--user "$(id -u):$(id -g)"` the container writes outputs as `nobody:nogroup`, blocking host-side cleanup and retry.

### Step 5 — Verify

- Container exit code 0
- `status.json` `kpi` block populated
- For `train`: inspect per-step `train_loss` directly — the entrypoint reports `Execution status: PASS` even when `train_loss = NaN` (see the Metric Variant Finetuning Recipe → Sanity-run PASS criteria in `references/finetuning-recipes.md`)
- For `evaluate` / `inference`: artifacts under `results_dir`

For TAO Deploy TensorRT actions (`gen_trt_engine`, TensorRT `evaluate`, and TensorRT `inference`), read `references/tao-deploy-depth-anything-v2.md` first. Deploy spec templates live in this skill's `references/` folder with the `spec_template_deploy_*.yaml` prefix.

## Training Requirements

- **Valid `dataset_name` values for mono `data_sources`** (case-insensitive): `ThreeDVLM`, `FSD`, `NvCLIP`, `IssacStereo`, `Crestereo`, `Middlebury`, `NYUDV2`, `NYUDV2Relative`, `RelativeMonoDataset`, `MetricMonoDataset`. `NYUDV2` carries metric depth GT (meters) — pair with `MetricDepthAnything`; `NYUDV2Relative` is the same data with relative-depth conventions — pair with `RelativeDepthAnything`.
- **Monitoring metric:** val/d1, val/loss
- For AutoML sanity runs on the packaged relative-depth smoke data, use `val/d1` as the primary monitor. `val/loss` can be emitted as `NaN` even when the trainer exits successfully and writes a usable checkpoint, so it is not a reliable AutoML objective unless the run's status metrics show a finite value.

### Per-Action Dataset Requirements

| Action | Spec Key | Source | Files | List? |
|---|---|---|---|---|
| evaluate | dataset.test_dataset.data_sources | eval_dataset | data_file: annotations.txt + dataset_name | Yes |
| inference | dataset.infer_dataset.data_sources | inference_dataset | data_file: annotations.txt + dataset_name | Yes |
| quantize | dataset.train_dataset.data_sources | train_datasets | data_file: annotations.txt + dataset_name | Yes |
| quantize | dataset.val_dataset.data_sources | eval_dataset | data_file: annotations.txt + dataset_name | Yes |
| quantize | dataset.quant_calibration_dataset.images_dir | train_datasets | images.tar.gz | No |
| train | dataset.train_dataset.data_sources | train_datasets | data_file: annotations.txt + dataset_name | Yes |
| train | dataset.val_dataset.data_sources | eval_dataset | data_file: annotations.txt + dataset_name | Yes |

### Typical Spec Overrides

Data source overrides are **mandatory for every action** — construct data source paths from the Per-Action Dataset Requirements table above and include them in `spec_overrides`. Each `data_sources` entry is a dict with **two mandatory fields**: `data_file` and `dataset_name`. See `references/spec-overrides.md` for the full per-action override blocks (`train`, `evaluate`, `export`, `inference`, `quantize`), the `S3_TRAIN` / `S3_EVAL` placeholders, the relative-variant precision recommendation, and the `quantize` known-issue note.

## Eval Dataset

Optional. Val dataset configured via `dataset.val_dataset.data_sources` (each entry needs `data_file` and `dataset_name`).

## Important Parameters

See `references/parameters.md` for the full parameter glossary (model, train, dataset, export, and inference keys with options, defaults, and sources) and the **Pretrained checkpoint loading — use case matrix**.

## Finetuning Recipes

See `references/finetuning-recipes.md` for:
- **Relative Variant Finetuning Recipe** — finetune from a TAO-trained `RelativeDepthAnything` checkpoint (lr `5e-6`, `LambdaLR`, sanity-vs-convergent guidance, deploy LSQ alignment note).
- **Metric Variant Finetuning Recipe** — checkpoint compatibility, required overrides, the dataset normalization block (`normalize_depth`/`min_depth`/`max_depth`) required in train AND export specs, trainer-enforced defaults, precision, the 1-epoch sanity-run override, and the Sanity-run PASS criteria with the NaN-mitigation order.

## Multi-GPU / Multi-Node

**Launch method:** Lightning-managed (single `python` process, Lightning spawns workers).

| Spec Key | Description | Default |
|----------|-------------|---------|
| `train.num_gpus` | Number of GPUs | 1 |
| `train.gpu_ids` | GPU device indices | [0] |
| `train.num_nodes` | Number of nodes | 1 |
| `train.distributed_strategy` | `ddp` or `fsdp` | `ddp` |

- `ddp` with activation checkpointing: `find_unused_parameters=False`
- `ddp` without: `find_unused_parameters=True`
- `fsdp` forces precision to FP16

**Multi-node env vars** (set by orchestrator): `WORLD_SIZE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`, `NUM_GPU_PER_NODE`.

## Export / TRT Defaults

- TRT data types: FP32, BF16 (Ampere SM80+). FP16 is not supported for the ViT-L mono backbone.
- Fresh-install TRT precision: `fp32`. BF16 is supported on Ampere SM80+ hardware, but keep smoke tests on FP32 unless the user explicitly requests BF16.

## Hardware

Minimum 1 GPU(s), recommended 2 GPU(s). 24GB+ VRAM per GPU. ViT-Large encoder is memory intensive. Use `fp32` (recommended) or `bf16` (Ampere SM80+, alternative) for training. Activation checkpointing is available for larger inputs.

## Error Patterns

See `references/troubleshooting.md` for the full error-pattern catalog (depth range mismatch, relative dataset rejecting `min_depth`, missing pretrained weights, `encoder` key location, `dataset_name` not in struct, `depth_net_mono` not found, metric variant hyperparameter sourcing, and export ONNX overwrite).

## Spec Param / Parent Model Inference

See `references/spec-param-inference.md` for the model-specific inference mappings (the TAO Core `depth_net_mono.config.json` action table), checkpoint-file naming under `<results_dir>/train/`, the `dn_model_latest.pth` policy, the parent-`gen_trt_engine` rationale, and the `parent_model` / `parent_job_id` resolution rules.

## Deployment

- [tao-deploy-depth-anything-v2](references/tao-deploy-depth-anything-v2.md)

Files

This skill is a package of 27 files. Install with the command above, or download the folder.

📄SKILL.md entry
📁evals
◆evals.json 870 B
📁references
📄finetuning-recipes.md 7.1 KB
📄parameters.md 5.4 KB
◆skill_info.yaml 4.2 KB
◆spec_template_deploy.yaml 2.1 KB
◆spec_template_evaluate.yaml 5.6 KB
◆spec_template_export.yaml 5.6 KB
◆spec_template_gen_trt_engine.yaml 5.6 KB
◆spec_template_inference.yaml 5.6 KB
◆spec_template_quantize.yaml 5.4 KB
◆spec_template_train.yaml 5.4 KB
📄spec-overrides.md 4.6 KB
📄spec-param-inference.md 3.4 KB
📄tao-deploy-depth-anything-v2.md 7.2 KB
◆tao-deploy-depth-anything-v2.skill_info.yaml 2.4 KB
📄troubleshooting.md 2.1 KB
📁schemas
◆evaluate.schema.json GitHub 0 B
◆export.schema.json GitHub 0 B
◆gen_trt_engine.schema.json GitHub 0 B
◆inference.schema.json GitHub 0 B
◆manifest.json 29.9 KB
◆quantize.schema.json GitHub 0 B
◆train.schema.json GitHub 0 B
📄BENCHMARK.md 4.0 KB
📄skill-card.md 3.9 KB
📄skill.oms.sig GitHub 0 B

NVIDIA/skills/tree/main/skills/tao-train-depth-anything-v2 commit · 63c02e122e

▌
tao-train-foundation-stereo · nvidia bundle
Trains, evaluates, exports, and runs inference on FoundationStereo models for stereo depth estimation and 3D reconstruction from stereo image pairs.

2.2k
repo stars
▌
tao-train-visual-changenet · nvidia bundle
Trains, evaluates, exports, and runs inference for Visual ChangeNet models used in AOI defect detection, comparing image pairs for PASS/NO_PASS classification or change-segmentation masks.

2.2k
repo stars
▌
tao-train-nvdinov2 · nvidia bundle
Trains vision transformers via self-distillation without labels for self-supervised visual representation learning, and supports export and inference of NVDINOv2 backbones.

2.2k
repo stars
▌
tao-train-fast-foundation-stereo · nvidia bundle
Trains, evaluates, exports, and runs inference for FastFoundationStereo (FFS) stereo depth estimation models, a distilled variant of FoundationStereo with lower latency.

2.2k
repo stars
▌
nemo-evaluator-sdk · orchestra-research bundle
Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution on local Docker, Slurm HPC, or cloud platforms.

10.4k
repo stars
▌
tao-train-sparse4d · nvidia bundle
Trains, evaluates, exports, quantizes, and runs inference for Sparse4D multi-camera temporal 3D object detection and tracking models using TAO.

2.2k
repo stars

Frequently asked questions

How do I install the tao-train-depth-anything-v2 skill?

Run npx skillmds add nvidia/tao-train-depth-anything-v2 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the tao-train-depth-anything-v2 skill do?

Train, evaluate, export, and run inference for monocular depth estimation models using Metric Depth Anything v2 or Relative Depth Anything architectures via the TAO toolkit. It is listed under AI & ML, Model Training & Fine-tuning on SkillMD.

Is tao-train-depth-anything-v2 safe to use?

SkillMD's automated safety review verdict for this skill is CAUTION. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with tao-train-depth-anything-v2?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is tao-train-depth-anything-v2 free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under Apache-2.

Who published tao-train-depth-anything-v2?

NVIDIA (@nvidia) published this skill as a verified publisher. Their other Agent Skills are listed on their SkillMD profile.