Eagle3 Triage

EAGLE3 Pipeline Triage

Diagnose failures in the 4-step EAGLE3 offline pipeline. This skill walks through each step, identifies the failure point, and provides actionable fixes.

Pipeline Overview

Step 0 — Locate the experiment

Ask the user for one of:

• Experiment directory (e.g., the --job-dir passed to launch.py or slurm.py)
• The model name / YAML they ran

Find recent experiments under the job directory:

ls -td experiments/cicd/cicd_* | head -10
# or wherever --job-dir was pointed

Each experiment directory contains one subdirectory per task (task_0 through task_3), each with a log file whose name varies by launch mode (Slurm: sbatch_*.out, local Docker: *.log).

Step 1 — Fetch logs for the failed task

Match the log files generally and read the tail of each — errors appear at the end:

find experiments/<exp_id>/ -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do
  echo "=== $f ==="; tail -200 "$f"; echo
done

Look for the first task with a non-zero exit code or error message.

Step 2 — Diagnose by step

task_0 failures (Data Synthesis)

How it works: Launches a vLLM OpenAI-compatible server, polls /health until ready, then runs query.py to generate synthetic prompt/response pairs. Output goes to /scratchspace/data/.

task_1 failures (Hidden State Dump)

How it works: Loads the target model and runs a forward pass on each conversation, saving hidden states as .pt files in /scratchspace/offline_hidden_states/.

Three backends are available:

task_2 failures (Training)

How it works: Installs requirements, runs launch_train.sh (Accelerate + FSDP) with the config from modelopt_recipes/general/speculative_decoding/eagle3.yaml, then exports via export_hf_checkpoint.py. Output: /scratchspace/eagle3/ and /scratchspace/export/.

task_3 failures (Benchmark)

How it works: Launches vLLM with the target + draft model, runs acceptance rate and throughput benchmarks. Output: JSON files.

Step 3 — Check for new-model-specific issues

If the user is adding support for a new model, also check:

1. Is the model a VLM? → Use dump_offline_data_hf.sh (text-only path, no vision encoder invoked)
2. Does the model use sliding window attention (SWA)? → TRT-LLM backend won't work; use HF or vLLM
3. Does the model need trust_remote_code? → Add to task_0 args AND task_3 args
4. Is the model MoE? → Check eagle_config.json intermediate_size matches model's moe_intermediate_size
5. Is the model architecture recognized by EAGLE3 training? → may need code changes in modelopt/torch/speculative/
6. Custom tokenizer? → May need additional environment vars (e.g., TIKTOKEN_RS_CACHE_DIR)

Step 4 — Suggest fix and next steps

After diagnosis, provide:

1. Root cause — one-line summary
2. Fix — specific config change, code edit, or command to run
3. How to re-run — skip earlier successful steps by pointing to existing scratchspace artifacts

To skip task_0 and task_1 and re-run from task_2:

uv run launch.py --yaml examples/<Org>/<Model>/hf_offline_eagle3.yaml \
    pipeline.task_0.skip=true \
    pipeline.task_1.skip=true \
    --yes

To run only task_1 standalone (using existing task_0 data):

uv run launch.py --yaml examples/<Org>/<Model>/hf_offline_eagle3.yaml \
    pipeline.task_0.skip=true \
    pipeline.task_2.skip=true \
    pipeline.task_3.skip=true \
    --yes

If the fix requires code changes in ModelOpt (e.g., supporting a new model architecture), note that a separate PR in the modelopt repo is needed.

Step 5 — Record the failure pattern

If you encounter a failure pattern not seen before, capture it in the team's internal triage tracker — the symptom, root cause, and fix — so the next engineer debugging the same issue benefits.