helixml Troubleshoot a ramjet node
Node06's 2026-08-14 cooling/AC moratorium was retired on 2026-08-25 after the
operator confirmed the repair. Synthetic tests and requested fixes are now
admitted through the ordinary intake-air thermal, duration, deployment-lock,
identity, and rollback controls in AGENTS.md.
Diagnose first. Do not restart, recreate, reconfigure, or delete anything unless the user also asks for a fix. Keep all output bounded and redact credentials, prompts, generated text, upstream addresses, and cache identifiers.
Establish scope and time
Record the reported symptom, first observed time, affected endpoint/model, and the Compose project. Capture image digests and container start times so a silent restart or mixed version cannot be mistaken for a performance issue.
Check from hardware upward
Run read-only checks in this order:
nvidia-smi -L
nvidia-smi --query-gpu=index,uuid,temperature.gpu,power.draw,memory.used,memory.total,utilization.gpu,ecc.errors.uncorrected.volatile --format=csv,noheader
nvidia-smi topo -m
df -h / /tmp
free -h
docker compose -f COMPOSE_FILE ps
docker stats --no-stream
Look for missing GPUs, Xid/ECC faults, thermal or power limits, exhausted VRAM,
host memory/disk pressure, restart loops, unhealthy containers, and unexpected
GPU assignments. Use docker inspect for exact image, start time, health, and
device bindings; do not dump the full environment because it may contain
secrets.
Check the serving path
curl --silent --show-error --fail http://127.0.0.1:API_PORT/health
curl --silent --show-error --fail http://127.0.0.1:METRICS_PORT/metrics \
| grep -E '^(ramjet_upstream_up|ramjet_route_decisions_total|ramjet_cache_requests_total|ramjet_snapshot_route_ready)'
curl --silent --show-error --fail http://127.0.0.1:ENGINE_PORT/health
Interpret /health as ok, degraded, or unhealthy; use opaque replica
ordinals in reports. Compare router upstream health, inflight/load, route split,
cache outcomes, TTFT, native queue/prefill gauges, preemptions, and GPU
utilization. A healthy HTTP endpoint with a growing queue is a capacity or
scheduler symptom, not a network outage.
Inspect only bounded recent logs around the incident:
docker compose -f COMPOSE_FILE logs --since 15m --tail 300 ramjet
docker compose -f COMPOSE_FILE logs --since 15m --tail 300 ENGINE_SERVICE \
| grep -Ei 'error|fatal|panic|traceback|CUDA|NCCL|OOM|Xid|JIT'
Run basic synthetic tests
Use a synthetic prompt and obtain the bearer token without printing it. First
query /v1/models, then send a deterministic request with at most eight output
tokens. Confirm HTTP success, response usage, and
X-Ramjet-Upstream. Run one request at a time before any concurrency
test.
If basic requests pass, choose only the focused test that matches the symptom:
- cache/locality:
python3 bench/cachebench.py ... --require-reconciled - route distribution:
bench/concurrent_sameapp.sh - agent/tool protocol:
python3 bench/agentbench.py run ... - direct engine decode:
python3 bench/codebench.py ... - long prefill:
python3 bench/mixed_bench.py ...
Use a fresh salt and point collectors at the same engine under test. Stop if native/client token counts do not reconcile or live traffic contaminates the interval.
Report the diagnosis
Separate observed facts from inference. Report the failing layer, first bad timestamp, affected replica ordinal, image/process identity, GPU and container state, decisive metrics/log markers, and the smallest next action. If the root cause is not proven, give ranked hypotheses and the read-only check that would distinguish them; do not present a restart as a diagnosis.