Run Atom Workload

Path convention

All script paths in this skill are project-relative (scripts/foo.sh), not absolute. The skill lives inside the ATOM repo at .claude/skills/, so it travels with the repo wherever it's cloned. CWD when invoking commands should be the repo root (Claude Code's default).

If the user is somewhere else, prefix with bash $(git rev-parse --show-toplevel)/scripts/foo.sh or cd to the repo first.

Why this skill exists

Every blocking ATOM workload follows the same 5-step shape: stop → start → verify-ready → workload-in-bg → wait_infer_drain → stop. The scripts in scripts/ are orchestration-grade — chain them via separate Bash tool calls, not wrappers, not &&.

Past failure modes this skill prevents (collected from many sessions):

1. Writing wrapper scripts in /app/logs_claude/ — start_atom_server.sh etc. ARE the orchestration layer. Wrapping them is pure noise. The dozens of run_*.sh / start_*_safe.sh files in /app/logs_claude/ are session debris — do not mimic them.
2. Chaining all steps with && into one long command — the user has explicitly forbidden this. Each step gets its own Bash tool call so logs are separate, errors abort cleanly, and the user can interrupt at any boundary.
3. Foregrounding start_atom_server.sh — its inline ready-poll caps at 120 iterations × sleep 1 = 120s with no failure exit, so any model that takes >2min to cold-start (V4-Pro takes 5–10min) lets the script return success while the server is still loading. Step 3 then launches against a not-yet-ready server. Fix: launch it with the Bash tool's run_in_background: true, use step 2.5 as the real foreground ready gate.
4. Skipping wait_infer_drain.sh — without it, GPU faults take the whole timeout to surface, and hangs go undetected. wait_infer_drain.sh exits in ~10s on fault and ~1min on hang, with tail-log attached.
5. Using curl /health for liveness — under heavy load it can false-negative. The flow uses /v1/models (start script) and pgrep + Engine Core marker (drain script).
6. Forgetting model-family env vars — V4-Pro silently regresses on accuracy without AITER_BF16_FP8_MOE_BOUND=0 ATOM_MOE_GU_ITLV=1. Pinned in the table below.
7. Skipping drain for offline simple_inference — wait_infer_drain.sh supports offline mode (process-exit detection + fault scan). Without it you lose early fault visibility.
8. Passing the wrong LOG_FILE to drain — wait_infer_drain.sh auto-discovers the server log via /proc/<pid>/fd/1; the user-supplied LOG_FILE is a secondary signal (fault scan + mtime progress for clients with tqdm output). Pass any log you want extra coverage on, or pass nothing — drain still works.
9. Trusting start_atom_server.sh exit alone — same root cause as failure 3. Launch step 2 with run_in_background: true and use step 2.5 wait_server_ready.sh as the mandatory foreground ready gate with a real (15min) timeout. If 2.5 fails, abort to step 5 — do not launch workload.
10. Writing a custom monitor loop to "catch the hang at the right moment" for rocgdb / py-spy attach — wait_infer_drain.sh exit=1 IS that moment: workers are still alive in livelock state, GPU queues still loaded, the next call is your attach. Self-written for i in ...; sleep 10; grep -c "output send" ... loops always misjudge the heuristic (drain's STUCK_POLLS check is tuned across hundreds of runs; yours isn't), waste a turn re-deriving it, and leak past the user-forbidden "no ad-hoc orchestration" rule. Use drain → on exit=1 attach rocgdb (see step 4.5) → step 5.

Backgrounding mechanism — Bash tool run_in_background: true, NOT shell &

Steps 2 and 3 launch long-running processes (server / workload) that must run concurrent with the next foreground gate (step 2.5 ready-poll, step 4 drain). Background them with the Bash tool's run_in_background: true parameter — never append a shell &.

Why run_in_background over shell &:

• Tracked, not orphaned. & detaches the process as an untracked orphan; if a later step kills the parent, the orphan can linger for a long time holding GPU (a real past incident: a &-orphaned server ran 58 min after its launcher was killed). run_in_background registers a task you can TaskStop, and the harness reports its exit.
• Output captured. The task's stdout/stderr go to a file the harness hands you; you're notified on completion and can read it.
• Drain still works. wait_infer_drain.sh / wait_server_ready.sh find the server + workload via pgrep, independent of how they were backgrounded — so nothing about the drain/ready gates changes.

Pattern (each step = one Bash tool call):

# Step 3 — Bash tool call with run_in_background: true (NO trailing &)
bash scripts/run_gsm8k_eval.sh /data/MODEL 30000 3

# Step 4 — Bash tool call, foreground, blocks on drain (finds workload via pgrep)
bash scripts/wait_infer_drain.sh 30000 30 10

The step-3 call returns immediately with a task id (the workload keeps running as a tracked task). Step 4 then runs as the next Bash call and blocks on drain.

Canonical 5-step flow

Run each step as a separate Bash tool call. Never chain with &&.

Step order: 1 stop → 2 start (bg) → 2.5 verify-ready → 3 workload (bg) → 4 drain → 4.5 (optional) hang inspection → 5 stop. Steps 2 and 3 use the Bash tool's run_in_background: true; all others are foreground.

Step 1 — clean GPU (always)

bash scripts/stop_atom_server.sh

Idempotent. SIGTERM → SIGKILL → force-kill GPU PIDs, waits ≤60s for VRAM=0. Always run first even if you believe no server is up — clears orphaned multiprocessing children that hold GPU memory.

Step 2 — start workload host (blocks until ready / completion)

Server-based workloads (GSM8K / benchmark / sweep / fault repro): launch with the Bash tool's run_in_background: true, then use step 2.5 as the real ready gate.

<MODEL_ENV_VARS> bash scripts/start_atom_server.sh <MODEL_PATH> <TP> <PORT> <EXTRA_ARGS...>

(Bash tool call, run_in_background: true, no trailing &.)

• MUST use run_in_background: true so the Bash tool returns immediately and step 2.5 can do the real foreground wait
• The script forks python in background and runs a best-effort inline poll, but that poll caps at 120s and falls through to exit-0 without raising — for any model that takes >2min to load, the inline poll's outcome is meaningless
• Step 2.5 is the source of truth for ready/fail
• Log: hard-coded /app/logs_claude/atom_server.log (LOG_FILE env is NOT respected by this script). Drain auto-discovers it via /proc/<pid>/fd/1 regardless of path

Offline workload (simple_inference): steps 2 and 2.5 are skipped — workload runs offline; jump to step 3.

Model-family env vars (set as VAR=val VAR=val bash ... prefix):

MTP add-on (any supporting model): append --method mtp --num-speculative-tokens N to EXTRA_ARGS. V4-Pro: keep --level 0.

Step 2.5 — verify server ready (MANDATORY for server-based workloads)

bash scripts/wait_server_ready.sh <PORT> <MAX_MIN> <POLL_SEC> /app/logs_claude/atom_server.log

Typical: bash scripts/wait_server_ready.sh 30000 15 5 /app/logs_claude/atom_server.log.

• Polls /v1/models and grep-watches the server log for startup errors (cluster_dims, InductorError, SHUTDOWN, proc died, AssertionError)
• Exit 0 → server ready, proceed to step 3
• Exit non-zero → abort to step 5 (stop_atom_server.sh). Do NOT launch step 3
• MAX_MIN: V4-Pro cold start ~5–10min; use 15 to be safe. Smaller models: 5–8
• Set Bash tool timeout to ≥ MAX_MIN × 60 × 1000 ms (e.g. 900000 for 15min) so the tool doesn't kill the poll prematurely

Why mandatory: step 2 ran in the background (run_in_background: true), so its exit code is meaningless to us — we deliberately ignored it. Step 2.5 is the one and only clean gate — its exit code, blocking behavior, and tail-on-fail are all visible to the operator. Without it, step 3 may launch against a dead or not-yet-ready server (lm_eval just sits in its own retry loop and hides the failure).

Skip step 2.5 for: offline simple_inference (no server), debug-agent fault repro (the fault IS the goal).

Step 3 — launch workload in the background (run_in_background: true)

Launch the workload as a Bash tool call with run_in_background: true (no trailing &) so the tool returns immediately and step 4 can start monitoring in parallel. Drain finds the workload via pgrep.

Server-based workloads (PORT is needed):

No foreground exception. Even a single perf benchmark runs with run_in_background: true — never foreground (it would block the tool for minutes and leave the run untracked). Read the metrics from the task output file after step 4's drain returns.

Offline simple_inference (no PORT; step 2 is skipped since this script IS the workload host):

<MODEL_ENV_VARS> bash scripts/start_simple_inference.sh MODEL TP <EXTRA_ARGS...>

(Bash tool call, run_in_background: true, no trailing &.)

Optional client log for drain: /app/logs_claude/simple_inference.log (drain auto-discovers via /proc anyway; this only helps fault grep redundancy).

Common workload knobs:

• GSM8K shots: 3 for fast/CI parity, 5 for thorough. Set LIMIT=50 env for first-50-sample sanity.
• Benchmark PROMPT_MULTIPLIER default 10. Profiling: use 2 (CONC × 2 requests).
• MTP benchmark MUST add --use-chat-template via EXTRA_ARGS (tokenizer mismatch otherwise).
• Benchmark throughput metric: report Total Token throughput (tok/s), NOT Output throughput. Total = input+output, which users care about.
• Never add --mark-trace or ENABLE_TORCH_PROFILER=1 (handled by capture-trace skill).

Step 4 — wait_infer_drain (blocks, with early fault/hang detection)

bash scripts/wait_infer_drain.sh PORT MAX_MIN POLL_SEC [LOG_FILE] [STUCK_POLLS]

Defaults: PORT=8000, MAX_MIN=30, POLL_SEC=10, LOG_FILE=empty (server log auto-discovered via /proc/<pid>/fd/1), STUCK_POLLS=6.

LOG_FILE is optional. The drain script discovers the server log itself from the running atom.entrypoints process. Pass an additional client/workload log only if you want:

• Extra fault grep coverage (drain scans both)
• Mtime-based progress detection for client tools that write tqdm to a file (benchmark, simple_inference)

PORT is unused in offline mode but kept positional for API symmetry.

How drain decides (auto-detects server vs offline by SERVER_PATTERN pgrep):

• Server mode: client gone (lm_eval / curl / benchmark process exited) + no new "Engine Core: output send" since last poll → exit 0
• Offline mode: simple_inference process exited cleanly (no fault grep) → exit 0
• Either mode: fault grep on auto-discovered server log + optional caller LOG_FILE → exit 2 in ≤10s
• Server only: no progress (engine output count flat + caller LOG_FILE mtime flat) AND client still running for STUCK_POLLS × POLL_SEC ≈ 1min → exit 1 (hang)
• Either mode: MAX_MIN elapsed without resolution → exit 4

If exit ≠ 0: read the printed tail. If exit=1 (hang) and you want to inspect the stuck GPU state, do step 4.5 BEFORE step 5 (step 5 kills the workers and destroys the evidence). Otherwise run step 5 regardless.

Typical wait windows:

• GSM8K (1319 samples): MAX_MIN=30 plenty for V4-Pro
• Single benchmark: MAX_MIN=30
• Sweep (8 conc points): MAX_MIN=60
• Simple_inference (default ~10 prompts): MAX_MIN=15 plenty
• Fault repro: MAX_MIN=10 (fault should land within first request)

Step 4.5 — hang inspection (optional, only when drain exit=1)

Trigger: step 4 returned exit=1 (HANG detected) AND you want to know which kernel / Python frame is stuck.

wait_infer_drain.sh exit=1 means the engine stopped emitting "output send" for STUCK_POLLS×POLL_SEC seconds while the benchmark client is still alive. At that instant:

• Server process is alive (pgrep atom.entrypoints returns)
• Worker spawn_main processes are alive (ps -ef | grep spawn_main)
• GPU queues still hold the stuck dispatches
• HSA debugger hook is FREE (you launched with plain start_atom_server.sh, not run_debug_agent.sh)

This is the unique window to attach rocgdb or py-spy before step 5 destroys the evidence. After step 5 all workers are SIGKILL'd and info dispatches returns "No dispatches" — too late.

Use the [[debug-agent-locate-kernel]] skill for the rocgdb workflow:

• Step R2 in that skill: pick a worker PID (ps -ef | grep spawn_main, PPID = dispatcher, NOT the openai_server)
• Step R3: rocgdb -p $WORKER_PID -x cmdfile -batch with detach before quit
• Step R4-R6: read info dispatches / info queues from the dump

Quick py-spy companion (for the Python-side stack on the same worker):

py-spy dump --pid $WORKER_PID 2>&1 | tee /app/logs_claude/pyspy_${WORKER_PID}.txt

Hard rule: do NOT replace step 4 with a self-written for/while polling loop just to "time the attach better". Drain's exit=1 IS the attach moment. See failure mode 10.

After inspection completes, proceed to step 5.

Step 5 — teardown (always)

bash scripts/stop_atom_server.sh

Same script as step 1. ALWAYS run, even on fault or for offline workloads — releases GPU for next attempt and kills any lingering multiprocessing children.

Reading results

After step 4 returns 0:

# GSM8K
grep -E "flexible-extract|strict-match" /app/logs_claude/gsm8k_eval.log | head -2
# Benchmark
grep -E "Total Token throughput|Mean TPOT|Mean TTFT" /app/logs_claude/benchmark.log
# Simple_inference
grep -E "^Generated|^Output|tokens/s" /app/logs_claude/simple_inference.log

GSM8K format: |gsm8k|3|flexible-extract|3|exact_match|↑|0.XXXX|±|0.00XX| — flexible-extract is the headline number, ±value is the noise band. Anything within 1σ of baseline = no regression.

Reading baselines

Per-model accuracy baselines and thresholds live in .github/benchmark/models_accuracy.json (CI is the source of truth). For one-off comparison, run the same (model, fewshot) pair locally and diff against the CI threshold in that file.

Hard rules (do not violate)

1. One Bash tool call per script. No && chains. User has explicitly forbidden chaining.
2. No wrapper scripts in /app/logs_claude/. Call scripts/* directly.
3. Bash tool run_in_background: true, NOT shell & for steps 2 and 3. & orphans the process untracked (past incident: 58-min lingering server); run_in_background gives a tracked, stoppable task with captured output. Drain finds the workload via pgrep either way.
4. Drain auto-discovers server log via /proc/<pid>/fd/1 — you no longer need to know or pass the canonical server log path. Pass a client log only as supplementary signal.
5. Always step 5 (stop_atom_server.sh), even after a fault, even for offline workloads.
6. Never use curl /health to verify ready — only /v1/models (already inlined in start script).
7. No LOG_FILE= env on start_atom_server.sh — log path is hard-coded.
8. Step 2 (start_atom_server.sh) MUST run with run_in_background: true. Foregrounding it is a trap: its inline ready-poll caps at 120s and falls through to exit-0 without raising — for V4-Pro and other slow-starting models the script silently returns success while the server is still loading. See failure mode 3.
9. Step 2.5 (wait_server_ready.sh) is MANDATORY for server-based workloads. Run it as a separate foreground Bash tool call after step 2. If it exits non-zero, abort to step 5; do not launch step 3.
10. Step 4 (wait_infer_drain.sh) is the ONLY hang detector. Never replace it with a custom for/sleep/grep monitor loop, even if your goal is to attach rocgdb / py-spy at hang time — drain's exit=1 IS the attach moment (workers still alive, GPU queues still loaded). Step 4.5 covers the inspection workflow. See failure mode 10.

Reference: each script in one line

Worked example: V4-Pro MTP3 GSM8K accuracy

# Step 1
bash scripts/stop_atom_server.sh

# Step 2 — Bash tool run_in_background: true (REQUIRED — see Hard rule 8), NO trailing &
AITER_BF16_FP8_MOE_BOUND=0 ATOM_MOE_GU_ITLV=1 AITER_LOG_LEVEL=WARNING \
  bash scripts/start_atom_server.sh /data/DeepSeek-V4-Pro 8 30000 \
  --kv_cache_dtype fp8 --method mtp --num-speculative-tokens 3 --level 0

# Step 2.5 — MANDATORY foreground ready gate. Bash tool timeout ≥ 900000ms.
# Abort to step 5 on non-zero exit.
bash scripts/wait_server_ready.sh 30000 15 5 /app/logs_claude/atom_server.log

# Step 3 — Bash tool run_in_background: true, NO trailing &
bash scripts/run_gsm8k_eval.sh /data/DeepSeek-V4-Pro 30000 3

# Step 4 — drain auto-discovers server log; no LOG_FILE needed
bash scripts/wait_infer_drain.sh 30000 30 10

# Step 5
bash scripts/stop_atom_server.sh

# Read result
grep -E "flexible-extract|strict-match" /app/logs_claude/gsm8k_eval.log | head -2

Worked example: V4-Pro offline simple_inference

# Step 1
bash scripts/stop_atom_server.sh

# Step 2+3 fused (simple_inference IS the workload host)
# Bash tool run_in_background: true, NO trailing &
AITER_BF16_FP8_MOE_BOUND=0 ATOM_MOE_GU_ITLV=1 AITER_LOG_LEVEL=WARNING \
  bash scripts/start_simple_inference.sh /data/DeepSeek-V4-Pro 8 \
  --kv_cache_dtype fp8 --level 0

# Step 4 — drain auto-discovers via /proc; PORT unused
bash scripts/wait_infer_drain.sh 0 15 10

# Step 5
bash scripts/stop_atom_server.sh