Nemotron Speech ASR Customization — Orchestration Skill
Note: "Nemotron Speech" is the public-facing name for what NVIDIA documents today as Riva / Riva NIM; the acoustic models are trained and fine-tuned with NVIDIA NeMo. Commands, config paths, imports, and doc URLs still use "Riva" / "NeMo" — the rename is brand-only. Do not rename them.
What This Skill Is
This is a high-level orchestration skill, not a step-by-step training manual. Its job, given a goal such as "I want to fine-tune ASR for my domain/language", is to:
- Scope the problem (how much real audio, target eval set, latency/hardware budget, language/domain).
- Choose the cheapest sufficient path — word boosting, n-gram LM fusion, or fine-tuning — and escalate only when quality falls short.
- Delegate each stage to the right sub-skill (data generation, training, evaluation, deployment/optimization).
- Answer cost/time/data questions along the way (how many hours to hit X% WER, synthetic vs real, L40S vs H100, expected cost).
It owns the plan and the routing; the sub-skills own the execution. When a needed sub-skill does not exist yet, this skill names it as a placeholder and gives interim guidance.
When to Use
Use for any request to make a Nemotron Speech / Riva ASR model work better on a specific domain or language — improving accuracy, reducing WER, adding a language, or planning a fine-tune. Start here even when the user names a specific technique, so the cheapest sufficient path is chosen and the right sub-skills are sequenced.
Orchestration Workflow
Run the loop below; each stage names the sub-skill it invokes. Full detail in references/workflow.md.
| # |
Stage |
What happens |
Sub-skill |
| 1 |
State the goal |
Capture the target: domain/language, the errors, the metric. |
Orchestration (this skill) |
| 2 |
Clarify & scope |
Ask the discovery questions: how much real audio? target eval set? latency/HW budget? deployment target? |
Orchestration |
| 3 |
Choose the path |
Pick the cheapest sufficient rung (boosting → n-gram LM → fine-tune). Escalate only if quality is short; experiment while proposing the full plan. |
Orchestration → Research/Training |
| 4 |
Get the data right |
If data is scarce/noisy: synthetic (TTS), TTS-friendly formatting, noise profiling/harvest, blend, score vendor samples; align customer data to training format; flag missing real data. |
SDG / Data |
| 5 |
Train |
Apply the recipe (configs, hyperparameters, replay/curriculum, GPU/OOM preflight) and run. |
Research / Training |
| 6 |
Evaluate |
Normalized WER on the domain set + A/B forgetting check on a general set; error-driven analysis to find the next lever. |
Evaluation |
| 7 |
Loop or ship |
If short of target, loop to 4/5 with targeted data; else select/average checkpoints. Consult the user before more cycles. |
Orchestration |
| 8 |
Deploy |
Export to NIM/HF, hot-swap the checkpoint, serve. |
Deployment / Optimization |
Stages 4–8 are the fine-tune path (data → NeMo train → NeMo eval → Riva deploy). Cheaper rungs (boosting, custom vocab, n-gram LM) take a shorter branch owned by a single sub-skill — don't force them through the full loop. See the branch-by-rung table in references/workflow.md (§3b).
Throughout, answer the "along the way" questions (data volume, synthetic vs real, hours to reach a WER target, cost, GPU choice) — see references/planning-answers.md.
Sub-Skills This Skill Calls
Detailed registry, invocation, and handoff contracts in references/sub-skills.md.
| Role (per the architecture) |
Purpose |
Sub-skill to invoke |
| Research / Training |
NeMo configs, recipes, fine-tuning, checkpoint averaging |
nemo-speech-asr-finetune |
| SDG / Data Designer |
Synthetic transcripts/text, noise profiling, vendor-data impact, blends |
data-designer (synthetic text; audio via TTS in nemotron-speech); placeholder: asr-data-profiling |
| Evaluation |
Normalized WER, A/B forgetting, error analysis |
Offline file WER → nemo-speech-asr-finetune; served-endpoint WER → nemotron-speech |
| Deployment / Optimization |
NIM/Riva export, checkpoint swap, NIM-build optimization, serving |
nemotron-speech |
If a sub-skill is unavailable, say so, give the interim guidance from the reference, and continue the plan.
Choosing The Path (cheapest first)
The scoping in Stage 3 selects the lowest-cost rung that can meet the target. Summary; full docs-grounded ladder in references/path-selection.md.
- Word boosting — a bounded set of known words/names/jargon. Runtime, no training. → Deployment sub-skill.
- Custom vocabulary / pronunciation — OOV or consistently mispronounced terms. Deploy-time. → Deployment sub-skill.
- N-gram (KenLM) LM — domain phrasing/word-sequences when you have text but little audio. Two realizations that are different artifacts: pilot (NeMo) to prove lift offline (
nemo-speech-asr-finetune), or deploy (Riva) to ship it (nemotron-speech). Don't ship the pilot LM — rebuild it in Riva word-level format. See references/path-selection.md.
- Fine-tune — real acoustic gaps (accents, noise, channel) with enough transcribed audio (NIM guide: 100+ h; ~10 h floor only if mixed to avoid catastrophic forgetting). → Research/Training.
- Train from scratch / cross-language transfer — a new language with no suitable checkpoint (last resort). → Research/Training.
Ordering and per-model support follow the NVIDIA Speech NIM ASR customization guide:
https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html.
Key Principles
- Scope before you pick. Don't recommend fine-tuning before the discovery questions and a measured baseline.
- Cheapest sufficient path. Escalate rungs only when the current one provably can't hit the target; you may experiment on a cheap rung while presenting the full fine-tuning plan.
- Measure with a contract. Report normalized WER on the domain set plus an A/B forgetting check on a general set — never in-training logs alone.
- Delegate, don't reimplement. Route execution to the sub-skills; keep this skill focused on the plan, sequencing, and cost/time/data answers.
- Real target-domain audio is the usual bottleneck. Prefer real data; use synthetic to fill measured gaps, kept separately weighted so it can be ablated.
- Consult the user before extra tuning cycles, and when a needed sub-skill is a placeholder.
Source of Truth
Limitations
- Orchestration only — execution happens in the sub-skills. Where a sub-skill is a placeholder, guidance is interim until it exists.
- GPU required for the training rungs; deployment/serving is owned by the
nemotron-speech sub-skill.
- Model names, config paths, flags, and per-model feature support drift across NeMo/Riva releases — verify against the support matrix and the current checkout.
- Public branding is "Nemotron Speech"; commands, imports, config paths, and doc URLs still use "Riva" / "NeMo" — do not rename.
1---2name: nemotron-asr-finetune3description: Orchestration skill for NVIDIA Nemotron Speech (Riva) / NeMo ASR domain and language adaptation. Given a goal like "improve/fine-tune ASR for my domain or language", it scopes the task, picks the cheapest sufficient path (word boosting → n-gram LM → fine-tuning), delegates each stage to the right sub-skill (data generation, training, evaluation, deployment), and answers cost/time/data questions along the way.4license: Apache-2.05---67# Nemotron Speech ASR Customization — Orchestration Skill89> **Note:** "Nemotron Speech" is the public-facing name for what NVIDIA documents today as **Riva** / **Riva NIM**; the acoustic models are trained and fine-tuned with **NVIDIA NeMo**. Commands, config paths, imports, and doc URLs still use **"Riva"** / **"NeMo"** — the rename is brand-only. Do not rename them.1011## What This Skill Is1213This is a **high-level orchestration skill**, not a step-by-step training manual. Its job, given a goal such as *"I want to fine-tune ASR for my domain/language"*, is to:14151. **Scope** the problem (how much real audio, target eval set, latency/hardware budget, language/domain).162. **Choose the cheapest sufficient path** — word boosting, n-gram LM fusion, or fine-tuning — and escalate only when quality falls short.173. **Delegate each stage to the right sub-skill** (data generation, training, evaluation, deployment/optimization).184. **Answer cost/time/data questions along the way** (how many hours to hit X% WER, synthetic vs real, L40S vs H100, expected cost).1920It owns the plan and the routing; the sub-skills own the execution. When a needed sub-skill does not exist yet, this skill names it as a **placeholder** and gives interim guidance.2122## When to Use2324Use for any request to make a Nemotron Speech / Riva ASR model work better on a specific domain or language — improving accuracy, reducing WER, adding a language, or planning a fine-tune. Start here even when the user names a specific technique, so the cheapest sufficient path is chosen and the right sub-skills are sequenced.2526## Orchestration Workflow2728Run the loop below; each stage names the sub-skill it invokes. Full detail in [`references/workflow.md`](references/workflow.md).2930| # | Stage | What happens | Sub-skill |31|---|---|---|---|32| 1 | **State the goal** | Capture the target: domain/language, the errors, the metric. | Orchestration (this skill) |33| 2 | **Clarify & scope** | Ask the discovery questions: how much real audio? target eval set? latency/HW budget? deployment target? | Orchestration |34| 3 | **Choose the path** | Pick the cheapest sufficient rung (boosting → n-gram LM → fine-tune). Escalate only if quality is short; experiment while proposing the full plan. | Orchestration → Research/Training |35| 4 | **Get the data right** | If data is scarce/noisy: synthetic (TTS), TTS-friendly formatting, noise profiling/harvest, blend, score vendor samples; align customer data to training format; flag missing real data. | SDG / Data |36| 5 | **Train** | Apply the recipe (configs, hyperparameters, replay/curriculum, GPU/OOM preflight) and run. | Research / Training |37| 6 | **Evaluate** | Normalized WER on the domain set + A/B forgetting check on a general set; error-driven analysis to find the next lever. | Evaluation |38| 7 | **Loop or ship** | If short of target, loop to 4/5 with targeted data; else select/average checkpoints. Consult the user before more cycles. | Orchestration |39| 8 | **Deploy** | Export to NIM/HF, hot-swap the checkpoint, serve. | Deployment / Optimization |4041**Stages 4–8 are the fine-tune path** (`data → NeMo train → NeMo eval → Riva deploy`). Cheaper rungs (boosting, custom vocab, n-gram LM) take a **shorter branch owned by a single sub-skill** — don't force them through the full loop. See the branch-by-rung table in [`references/workflow.md`](references/workflow.md) (§3b).4243Throughout, answer the **"along the way"** questions (data volume, synthetic vs real, hours to reach a WER target, cost, GPU choice) — see [`references/planning-answers.md`](references/planning-answers.md).4445## Sub-Skills This Skill Calls4647Detailed registry, invocation, and handoff contracts in [`references/sub-skills.md`](references/sub-skills.md).4849| Role (per the architecture) | Purpose | Sub-skill to invoke |50|---|---|---|51| **Research / Training** | NeMo configs, recipes, fine-tuning, checkpoint averaging | `nemo-speech-asr-finetune` |52| **SDG / Data Designer** | Synthetic transcripts/text, noise profiling, vendor-data impact, blends | `data-designer` (synthetic **text**; audio via TTS in `nemotron-speech`); *placeholder:* `asr-data-profiling` |53| **Evaluation** | Normalized WER, A/B forgetting, error analysis | Offline file WER → `nemo-speech-asr-finetune`; **served-endpoint WER → `nemotron-speech`** |54| **Deployment / Optimization** | NIM/Riva export, checkpoint swap, NIM-build optimization, serving | `nemotron-speech` |5556If a sub-skill is unavailable, say so, give the interim guidance from the reference, and continue the plan.5758## Choosing The Path (cheapest first)5960The scoping in Stage 3 selects the lowest-cost rung that can meet the target. Summary; full docs-grounded ladder in [`references/path-selection.md`](references/path-selection.md).6162- **Word boosting** — a bounded set of known words/names/jargon. Runtime, no training. → Deployment sub-skill.63- **Custom vocabulary / pronunciation** — OOV or consistently mispronounced terms. Deploy-time. → Deployment sub-skill.64- **N-gram (KenLM) LM** — domain phrasing/word-sequences when you have text but little audio. **Two realizations that are different artifacts:** *pilot (NeMo)* to prove lift offline (`nemo-speech-asr-finetune`), or *deploy (Riva)* to ship it (`nemotron-speech`). Don't ship the pilot LM — rebuild it in Riva word-level format. See [`references/path-selection.md`](references/path-selection.md).65- **Fine-tune** — real acoustic gaps (accents, noise, channel) with enough transcribed audio (NIM guide: 100+ h; ~10 h floor only if mixed to avoid catastrophic forgetting). → Research/Training.66- **Train from scratch / cross-language transfer** — a new language with no suitable checkpoint (last resort). → Research/Training.6768Ordering and per-model support follow the NVIDIA Speech NIM ASR customization guide:69<https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html>.7071## Key Principles7273- **Scope before you pick.** Don't recommend fine-tuning before the discovery questions and a measured baseline.74- **Cheapest sufficient path.** Escalate rungs only when the current one provably can't hit the target; you may experiment on a cheap rung while presenting the full fine-tuning plan.75- **Measure with a contract.** Report normalized WER on the domain set plus an A/B forgetting check on a general set — never in-training logs alone.76- **Delegate, don't reimplement.** Route execution to the sub-skills; keep this skill focused on the plan, sequencing, and cost/time/data answers.77- **Real target-domain audio is the usual bottleneck.** Prefer real data; use synthetic to fill measured gaps, kept separately weighted so it can be ablated.78- **Consult the user before extra tuning cycles**, and when a needed sub-skill is a placeholder.7980## Source of Truth8182| Topic | Location |83|---|---|84| NIM Speech docs home | https://docs.nvidia.com/nim/speech/latest/index.html |85| ASR customization guide (methods, per-model support) | https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html |86| ASR support matrix (models & features) | https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html |87| NeMo fine-tuning (flags/config) | `docs/source/asr/fine_tuning.rst`, and the `nemo-speech-asr-finetune` sub-skill |88| Riva ASR tutorials (boosting, LM, fine-tune) | https://github.com/nvidia-riva/tutorials |89| Tokenizer extension to new language + acoustic fine-tune | https://github.com/nvidia-riva/tutorials/blob/main/asr-extend-tokenizer-to-newlang-ft-acoustic-model.ipynb |9091## Limitations9293- Orchestration only — execution happens in the sub-skills. Where a sub-skill is a placeholder, guidance is interim until it exists.94- GPU required for the training rungs; deployment/serving is owned by the `nemotron-speech` sub-skill.95- Model names, config paths, flags, and per-model feature support drift across NeMo/Riva releases — verify against the support matrix and the current checkout.96- Public branding is **"Nemotron Speech"**; commands, imports, config paths, and doc URLs still use **"Riva"** / **"NeMo"** — do not rename.
NVIDIA