# Troubleshoot Ramjet Node

> Diagnose a ramjet GPU inference node by checking NVIDIA GPUs, Docker Compose services, OpenAI endpoints, router health, Prometheus metrics, logs, networking, and basic synthetic requests. Use for outages, degraded health, low throughput, cache misses, stuck requests, GPU errors, or post-deploy verification.

- Skill: `helixml/troubleshoot-ramjet-node` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add helixml/troubleshoot-ramjet-node`
- Raw SKILL.md: https://api.skillmd.com/api/skills/helixml/troubleshoot-ramjet-node/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: helixml (https://skillmd.com/u/helixml)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/helixml/troubleshoot-ramjet-node

---


# Troubleshoot a ramjet node

Node06's 2026-08-14 cooling/AC moratorium was retired on 2026-08-25 after the
operator confirmed the repair. Synthetic tests and requested fixes are now
admitted through the ordinary intake-air thermal, duration, deployment-lock,
identity, and rollback controls in `AGENTS.md`.

Diagnose first. Do not restart, recreate, reconfigure, or delete anything unless
the user also asks for a fix. Keep all output bounded and redact credentials,
prompts, generated text, upstream addresses, and cache identifiers.

## Establish scope and time

Record the reported symptom, first observed time, affected endpoint/model, and
the Compose project. Capture image digests and container start times so a
silent restart or mixed version cannot be mistaken for a performance issue.

## Check from hardware upward

Run read-only checks in this order:

```bash
nvidia-smi -L
nvidia-smi --query-gpu=index,uuid,temperature.gpu,power.draw,memory.used,memory.total,utilization.gpu,ecc.errors.uncorrected.volatile --format=csv,noheader
nvidia-smi topo -m
df -h / /tmp
free -h
docker compose -f COMPOSE_FILE ps
docker stats --no-stream
```

Look for missing GPUs, Xid/ECC faults, thermal or power limits, exhausted VRAM,
host memory/disk pressure, restart loops, unhealthy containers, and unexpected
GPU assignments. Use `docker inspect` for exact image, start time, health, and
device bindings; do not dump the full environment because it may contain
secrets.

## Check the serving path

```bash
curl --silent --show-error --fail http://127.0.0.1:API_PORT/health
curl --silent --show-error --fail http://127.0.0.1:METRICS_PORT/metrics \
  | grep -E '^(ramjet_upstream_up|ramjet_route_decisions_total|ramjet_cache_requests_total|ramjet_snapshot_route_ready)'
curl --silent --show-error --fail http://127.0.0.1:ENGINE_PORT/health
```

Interpret `/health` as `ok`, `degraded`, or `unhealthy`; use opaque replica
ordinals in reports. Compare router upstream health, inflight/load, route split,
cache outcomes, TTFT, native queue/prefill gauges, preemptions, and GPU
utilization. A healthy HTTP endpoint with a growing queue is a capacity or
scheduler symptom, not a network outage.

Inspect only bounded recent logs around the incident:

```bash
docker compose -f COMPOSE_FILE logs --since 15m --tail 300 ramjet
docker compose -f COMPOSE_FILE logs --since 15m --tail 300 ENGINE_SERVICE \
  | grep -Ei 'error|fatal|panic|traceback|CUDA|NCCL|OOM|Xid|JIT'
```

## Run basic synthetic tests

Use a synthetic prompt and obtain the bearer token without printing it. First
query `/v1/models`, then send a deterministic request with at most eight output
tokens. Confirm HTTP success, response usage, and
`X-Ramjet-Upstream`. Run one request at a time before any concurrency
test.

If basic requests pass, choose only the focused test that matches the symptom:

- cache/locality: `python3 bench/cachebench.py ... --require-reconciled`
- route distribution: `bench/concurrent_sameapp.sh`
- agent/tool protocol: `python3 bench/agentbench.py run ...`
- direct engine decode: `python3 bench/codebench.py ...`
- long prefill: `python3 bench/mixed_bench.py ...`

Use a fresh salt and point collectors at the same engine under test. Stop if
native/client token counts do not reconcile or live traffic contaminates the
interval.

## Report the diagnosis

Separate observed facts from inference. Report the failing layer, first bad
timestamp, affected replica ordinal, image/process identity, GPU and container
state, decisive metrics/log markers, and the smallest next action. If the root
cause is not proven, give ranked hypotheses and the read-only check that would
distinguish them; do not present a restart as a diagnosis.