NVIDIA EAGLE3 New Model Configuration
Create tools/launcher/examples/<Org>/<Model>/hf_offline_eagle3.yaml by copying the
closest existing example and adapting it. Pick a reference with the same shape as the
target (dense vs MoE, similar size) from tools/launcher/examples/ — e.g. the Qwen3-8B
config for a dense model.
The pipeline is a 4-task config (task_0 data synthesis → task_1 hidden-state dump →
task_2 train → task_3 benchmark). The task structure, args, containers, and GPU/node
sizing are all visible in the existing examples — infer them from a reference rather than
hand-rolling. This file documents only the two things that are not obvious from the
examples: which dump backend to pick, and the model-specific gotchas.
Choosing the task_1 hidden-state dump backend
| Backend | Script | When to use |
|---|---|---|
| vLLM | common/eagle3/dump_offline_data_vllm.sh |
Default. Broad coverage via vLLM's native hidden-state extractor. |
| HF | common/eagle3/dump_offline_data_hf.sh |
VLMs / multimodal, custom-code models, sliding-window attention (TRT-LLM can't serve these). |
| TRT-LLM | common/eagle3/dump_offline_data.sh |
Pure-text models with TRT-LLM support; pass --tp <TP> and --moe-ep <EP>. |
Rule of thumb: HF if the model is a VLM or uses sliding-window attention; vLLM otherwise. TRT-LLM only when you specifically want its kernels for a supported plain-text model.
Model-specific adjustments
These are the non-obvious knobs that vary per model:
| Situation | What to change |
|---|---|
Requires --trust-remote-code |
Add to task_0 vLLM args (before the -- separator) and to task_3 benchmark args |
| MoE with large expert hidden dim | Increase intermediate_size in eagle_config.json to match moe_intermediate_size |
| Custom tokenizer (e.g. tiktoken) | Set TIKTOKEN_RS_CACHE_DIR env var in task_0 and task_1 |
After adapting the config, preview it with --dryrun before submitting.