Serve a model on one node
Use a DeepOps checkout on the allocated serving node (or inside the workload
container) with scripts/validation/validate_model_cache.py and
scripts/validation/validate_vllm.py readable and their parent directories
traversable by the consumer UID; run from that checkout's root. Read the
vLLM quickstart for the exact
installation and foreground launch commands. This skill does not allocate a GPU
or authorize a workload.
Preconditions
- Confirm permission for the node, one isolated GPU, package installation,
server startup and the validator's inference request. Record the allocation
and the process you own. Never use a login shell outside that allocation or
override scheduler device isolation.
- Confirm model license, immutable revision, complete file list, reader identity,
cache mount and served alias. Use manage-model-cache
for missing files; do not make consumers download them.
- Require Python 3.12 and a fresh private serving environment using
examples/vllm/requirements.txt. Check wheel/driver compatibility against the
quickstart; do not change drivers or the cache writer's environment.
- Keep the whole Hub cache read-only, including blobs. Keep authentication,
HF_HOME, Xet and framework caches private. Pause snapshot writes during use.
Procedure and success gate
- Follow quickstart steps 1–2. Require cache validator exit
0 and ok: true
as the actual reader before startup. Config alone is not a complete model;
name every required weight shard and tokenizer file. The three files in the
example are the set it was validated with; finetune-model also requires
tokenizer_config.json. Add it for any model whose tokenizer needs it, or
when one cache snapshot serves both workflows.
- Use quickstart step 3 unchanged for the public example: local snapshot and
tokenizer, offline loading, one GPU, foreground process, loopback only. Never
enable remote model code.
HF_HUB_OFFLINE=1 is not a firewall and belongs only
on the loading process, not globally on clients.
- Wait for startup; use another shell in the same allocation and namespace to
validate. This submits inference, not just a read-only health check:
python3 scripts/validation/validate_vllm.py --json \
--base-url http://127.0.0.1:8000 --model deepops-gpt2 \
--cache-dir /shared/hf-cache/hub \
--repo-id openai-community/gpt2 \
--revision 607a30d783dfa663caf39e06633721c8d4cfcd7e \
--require-file config.json --require-file tokenizer.json --require-file model.safetensors
- Require exit
0 and ok: true. Schema version 1 checks cache, health,
advertised model and bounded completion in order. Exit 1 fails validation;
exit 2 is a command syntax error without guaranteed JSON. A false check can
mean skipped; inspect the first error stage. Preserve the invocation, package
versions and private startup log. Success does not prove loaded revision,
GPU use, quality or throughput; an alias alone is not revision attestation.
- Stop only your own foreground server with Ctrl-C. Confirm exit before releasing
the allocation. Do not kill another listener or prune the shared cache.
Stop and recovery
| Observation |
Action |
| Missing files or unreadable cache |
Stop; designated writer repairs the pinned snapshot, then reader revalidates. Never loosen permissions globally. |
| Full cache |
Stop downloads; request operator-reviewed retention, not automatic deletion. |
| Port occupied or endpoint ownership unclear |
Stop; identify the owner. Do not validate or kill it without permission. |
| Startup/health failure |
Inspect private logs for compatibility, memory and files; rerun validation only after startup is confirmed. |
| Authentication required / HTTP 401 |
Stop; use the site's approved authenticated workflow. Never disable controls for this example. |
| Model/completion failure |
Compare alias, full file list and server log. Health alone is not success. |
| Multi-node, remote access or production service requested |
Stop; unsupported here. Do not widen the bind address or improvise distributed flags. |
Offline fixtures test the validator, not real cache reuse, GPU inference or
agent decision-making. Those require separate verification.
1---2name: serve-model3description: Use when serving a cached model on one GPU with vLLM, checking a local completion endpoint, or diagnosing startup and inference failures in the single-node quickstart.4metadata:5 source: https://skillmd.com/skills/nvidia/serve-model6---78# Serve a model on one node910Use a DeepOps checkout on the allocated serving node (or inside the workload11container) with `scripts/validation/validate_model_cache.py` and12`scripts/validation/validate_vllm.py` readable and their parent directories13traversable by the consumer UID; run from that checkout's root. Read the14[vLLM quickstart](../../docs/model-workflows/vllm-quickstart.md) for the exact15installation and foreground launch commands. This skill does not allocate a GPU16or authorize a workload.1718## Preconditions1920- Confirm permission for the node, one isolated GPU, package installation,21 server startup and the validator's inference request. Record the allocation22 and the process you own. Never use a login shell outside that allocation or23 override scheduler device isolation.24- Confirm model license, immutable revision, complete file list, reader identity,25 cache mount and served alias. Use [manage-model-cache](../manage-model-cache/SKILL.md)26 for missing files; do not make consumers download them.27- Require Python 3.12 and a fresh private serving environment using28 `examples/vllm/requirements.txt`. Check wheel/driver compatibility against the29 quickstart; do not change drivers or the cache writer's environment.30- Keep the whole Hub cache read-only, including blobs. Keep authentication,31 `HF_HOME`, Xet and framework caches private. Pause snapshot writes during use.3233## Procedure and success gate34351. Follow quickstart steps 1–2. Require cache validator exit `0` **and** `ok: true`36 as the actual reader before startup. Config alone is not a complete model;37 name every required weight shard and tokenizer file. The three files in the38 example are the set it was validated with; `finetune-model` also requires39 `tokenizer_config.json`. Add it for any model whose tokenizer needs it, or40 when one cache snapshot serves both workflows.412. Use quickstart step 3 unchanged for the public example: local snapshot and42 tokenizer, offline loading, one GPU, foreground process, loopback only. Never43 enable remote model code. `HF_HUB_OFFLINE=1` is not a firewall and belongs only44 on the loading process, not globally on clients.453. Wait for startup; use another shell in the same allocation and namespace to46 validate. This submits inference, not just a read-only health check:4748```bash49python3 scripts/validation/validate_vllm.py --json \50 --base-url http://127.0.0.1:8000 --model deepops-gpt2 \51 --cache-dir /shared/hf-cache/hub \52 --repo-id openai-community/gpt2 \53 --revision 607a30d783dfa663caf39e06633721c8d4cfcd7e \54 --require-file config.json --require-file tokenizer.json --require-file model.safetensors55```56574. Require exit `0` **and** `ok: true`. Schema version 1 checks cache, health,58 advertised model and bounded completion in order. Exit `1` fails validation;59 exit `2` is a command syntax error without guaranteed JSON. A false check can60 mean skipped; inspect the first error stage. Preserve the invocation, package61 versions and private startup log. Success does not prove loaded revision,62 GPU use, quality or throughput; an alias alone is not revision attestation.635. Stop only your own foreground server with Ctrl-C. Confirm exit before releasing64 the allocation. Do not kill another listener or prune the shared cache.6566## Stop and recovery6768| Observation | Action |69|---|---|70| Missing files or unreadable cache | Stop; designated writer repairs the pinned snapshot, then reader revalidates. Never loosen permissions globally. |71| Full cache | Stop downloads; request operator-reviewed retention, not automatic deletion. |72| Port occupied or endpoint ownership unclear | Stop; identify the owner. Do not validate or kill it without permission. |73| Startup/health failure | Inspect private logs for compatibility, memory and files; rerun validation only after startup is confirmed. |74| Authentication required / HTTP 401 | Stop; use the site's approved authenticated workflow. Never disable controls for this example. |75| Model/completion failure | Compare alias, full file list and server log. Health alone is not success. |76| Multi-node, remote access or production service requested | Stop; unsupported here. Do not widen the bind address or improvise distributed flags. |7778Offline fixtures test the validator, not real cache reuse, GPU inference or79agent decision-making. Those require separate verification.