Deploy Linux Gpu

deploy-linux-gpu

Use this when standing up mesh-llm on a remote Linux GPU box (rented GPU like Vast.ai / RunPod, or your own server) to serve a specific model and join the mesh.

This is the Linux/CUDA counterpart to the deploy-macos skill. It does NOT use the old llama-server/rpc-server lane — the current binary embeds the staged runtime. There are no .dylib/codesign/quarantine steps on Linux.

note --auto flag tells it to join the public mesh. serve command with --model tells it to run a specific model. Examples here are for solo serving — don't read this in isolation:

• deploy-macos / deploy-windows — other platforms
• mesh-join — creating/joining private and public meshes (tokens, NAT, multi-node)
• connect-agents — pointing Goose/Claude Code/OpenCode/Pi at a running mesh
• docs/USAGE.md — install details, service mode, model storage
• docs/CLI.md — full command and model-ref reference
• docs/SKIPPY_SPLITS.md — splitting big models across nodes

The one rule that matters most

mesh-llm resolves and downloads the model itself. When you pass --model, it fetches the GGUF into the standard Hugging Face cache on first use and serves it. Do NOT pre-download with hf / huggingface-cli, do NOT scp a GGUF, do NOT hunt for where the file lives. Just pass --model <ref> and let mesh-llm do it.

Install

SSH in and run the official installer. It auto-detects the GPU and CUDA major version and pulls the matching CUDA build (e.g. ...-cuda-13.tar.gz). RTX 50xx (Blackwell / sm_120) is detected and handled by the installer.

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | sh

The binary lands at ~/.local/bin/mesh-llm (not always on a non-interactive SSH PATH — use the full path or a login shell). Verify:

~/.local/bin/mesh-llm --version
nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv,noheader

Confirm the driver/CUDA is new enough for your GPU (Blackwell/RTX 50xx needs CUDA 13 + a recent driver, which the installer's -cuda-13 asset targets).

The launch command

all you need:

mesh-llm serve --model unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL --auto

• mesh-llm downloads/resolves the model itself (HF cache, first use only).
• It serves the model locally on the GPU.
• --auto also discovers and joins the community mesh, so /v1/models shows the union of local + peer models and routing works across nodes.
• Both serving locally AND joining the mesh happen together — --auto does not suppress local serving when --model is set.

Notes / gotchas:

• Do NOT use --headless to "go quiet". It only disables the embedded web console; it does nothing useful for backgrounding and is a recurring mistake.
• Model load takes time. After it joins the mesh, the GPU load + server bring-up can take a few minutes. Do not conclude "it's not serving" from an early check — poll until the GPU shows VRAM used and ports are bound.
• --model accepts catalog names, repo/file.gguf, repo:QUANT, or a full HF URL. For models not in the bundled catalog (e.g. Qwen3.6), use the HF ref form org/Repo-GGUF:QUANT.

Choosing a quant for a context target

Context length is auto-scaled to VRAM (KV cache defaults to Q8_0). With --auto the planner targets up to 4 concurrent slots, which divides the KV budget. If you need a guaranteed deep context per request, pin it:

mesh-llm serve --model <ref> --auto --ctx-size 65536

Pick the quant so model_bytes + KV fits VRAM at your target context. KV cost scales with layers × kv_heads × head_dim. Heavy-KV models (e.g. 64 layers, head_dim 256) cost ~130 KB/token at Q8_0 → ~8.5 GB at 64K per slot, so prefer a smaller weight quant (e.g. UD-Q4_K_XL) on a 32 GB card to leave KV headroom.

Keep it running (survives SSH disconnect)

A plain cmd & over SSH dies when the session closes, and tmux dies if the tmux server is killed/reaped. On managed GPU images that ship supervisor (common on Vast.ai), use supervisor — it restarts on crash and persists.

# Run this as the user who installed mesh-llm, so HOME, the model cache, and
# the persistent node identity all resolve under that user's account.
install_user="$(id -un)"
install_home="$(getent passwd "$install_user" | cut -d: -f6)"
if ! test -n "$install_home" || ! test -x "$install_home/.local/bin/mesh-llm"; then
  echo "mesh-llm binary is missing from $install_home/.local/bin" >&2
  exit 1
fi

sudo tee /etc/supervisor/conf.d/mesh-llm.conf >/dev/null <<EOF
[program:mesh-llm]
command=$install_home/.local/bin/mesh-llm serve --model unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL --auto
directory=$install_home
user=$install_user
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=30
stdout_logfile=/var/log/mesh-llm.log
stderr_logfile=/var/log/mesh-llm.log
environment=HOME="$install_home"
EOF
sudo supervisorctl reread && sudo supervisorctl update && sudo supervisorctl start mesh-llm
sudo supervisorctl status mesh-llm

If there is no supervisor, the installer can set up a systemd --user service for you (curl -fsSL .../install.sh | sh -s -- --service installs ~/.config/systemd/user/mesh-llm.service; startup models go in ~/.mesh-llm/config.toml, and sudo loginctl enable-linger "$USER" makes it survive reboot before login). Otherwise use tmux new -d, or a foreground process in a held SSH session for first-run debugging (allocate a TTY with ssh -tt host 'bash -lc "..."').

Verify it's actually serving

Poll until VRAM is used and the OpenAI port is bound, then test inference.

# GPU should show real VRAM used (not ~1 MiB) once the model loads
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv,noheader

# OpenAI port 9337 and console 3131 should be bound
ss -lntp | grep -E '9337|3131'

# Models (union of local + mesh peers)
curl -s http://localhost:9337/v1/models | python3 -m json.tool

# Inference — confirm the returned "model" is YOUR model id
curl -s http://localhost:9337/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'

The response "model" field tells you which node/model answered. To force your local model specifically, pass its exact id from /v1/models instead of auto.

Logs and state

• /var/log/mesh-llm.log — process output when run under the supervisor config above.
• ~/.mesh-llm/runtime/<pid>/logs/skippy-native.log — embedded llama.cpp/skippy native logs (redirected away from the TUI). Check here if the model fails to load onto the GPU.
• ~/.mesh-llm/key — persistent node identity.
• HF cache (~/.cache/huggingface/...) — where mesh-llm puts downloaded GGUFs. You generally never need to touch this.

Stop / clean up

# Tracked foreground/background run:
mesh-llm stop
# For a Supervisor-managed instance, stop Supervisor before any emergency kill:
sudo supervisorctl stop mesh-llm
# Emergency only if it remains alive: verify this is the intended instance/PID.
sudo supervisorctl status mesh-llm
kill -9 <mesh-llm-pid>

A clean stop removes the instance runtime dir under ~/.mesh-llm/runtime/.

Vast.ai specifics

• First SSH after boot can fail auth with a "try again after a few seconds" banner. Retry the connection (a short loop that re-attempts on exit code 255 works).
• Vast images often run their own services (portal, jupyter, syncthing, supervisor) — a non-zero load average at idle is normal and not your process.
• Vast pre-allocates external ports and fronts web apps with Caddy; if you want the console/API reachable externally, map it through the box's documented external port + proxy rather than assuming 9337/3131 are public.
• Use the proxy or direct SSH connect string Vast gives you; -L 8080:... port-forwards are handy for poking at local services from your workstation.

Troubleshooting