Optimize Ramjet Node

Optimize a ramjet node

Node06 access

The 2026-08-14 cooling/AC moratorium was retired on 2026-08-25 after the operator confirmed the repair. Optimization work is admitted through the intake-air thermal guard, bounded runtime, isolated candidate traffic, common deployment lock, and exact rollback authority described in AGENTS.md.

Optimize from evidence, with one attributable change at a time. Read AGENTS.md for the current benchmark contract and RESULTS.md for the metric definitions before running GPU work.

Capture a clean baseline

1. Record the immutable engine and LB image digests, process/container start times, effective argv, model/tokenizer revision, GPU topology, CPU/NUMA placement, and current router configuration.
2. Check nvidia-smi, engine health, ramjet /health, queue/load gauges, and recent CUDA/NCCL/OOM/Xid/JIT markers. Do not benchmark an unhealthy or identity-ambiguous stack.
3. Keep unrelated production traffic off the engine whose native metrics will be interpreted. Reject intervals whose request/token counters do not reconcile.

Choose the smallest useful experiment

• Routing policy: use bench/route_replay.py before a live A/B.
• Cache working sets: use bench/cachebench.py with a fresh salt for every cell; keep cells serial.
• Concurrent shared prompts: use bench/concurrent_sameapp.sh.
• Direct engine throughput: use bench/codebench.py or bench/engine_matrix.sh with the same engine's metrics endpoint.
• Long-prefill interference: use bench/mixed_bench.py in both request orders.
• Infernal r11 on B: use bench/candidate_gate.py --profile infernal-r11-b; stop at the first failed correctness, runtime, or scout gate. Add and review a pinned admission profile before using the gate for another engine image.

Start with a correctness smoke and one representative scout. Run a full matrix only when the candidate passes. Always use fresh synthetic salts; never compare one candidate's cold state with another candidate's warm state.

Change one variable

State the hypothesis, primary success metric, regression guardrails, and rollback value before mutation. Validate any new vLLM argument inside the pinned image before restarting an engine. Roll one replica at a time and keep production single-homed on its healthy peer.

Matched direct-engine work may use a two-round A/B crossover on independent GPU pairs. Keep cache-locality tests, LB policy tests, aggregate box-capacity tests, and exact-placement tests serial because concurrent work changes the state being measured.

Interpret the result

Compare end-to-end throughput, TTFT, request success, token reconciliation, queue/prefill time, preemptions, and effective speculative tokens per step. Never promote on cache-hit percentage or draft acceptance alone; either can improve because its denominator shrank.

Record commands, fresh input identity, wall time, image/process identity, metric deltas, variance, and contamination checks in EXPERIMENTS.md. Promote only when the result beats the declared threshold without a correctness or latency regression; otherwise restore the baseline and keep the negative result as evidence.