majiayu000 Deploy Model
Deploy or manage inference models on Kaizen. Argument should be one of: reasoning, embedding, heavy, or a manifest path.
Pre-flight Checks
- Verify target node is Ready:
kubectl get nodes - Check GPU availability:
kubectl describe node <node> | grep -A3 nvidia.com/gpu - Check existing deployments:
kubectl get pods -n inference
Available Manifests
k8s/apps/inference/— All inference deploymentsk8s/apps/inference/sglang-core.yaml— Heavy model (Qwen2.5-72B, TP=4, CORE)
GPU Constraints
- INTERFACE: RTX 5090 (32GB) + RTX 4090 (24GB) — NO tensor parallelism
- CORE: 4x RTX 5070 Ti (16GB each) — TP=4 supported
- Deployment order determines GPU assignment on INTERFACE
Deploy Steps
- Read the manifest file
- Validate:
kubectl apply --dry-run=client -f <manifest> - Show the user what will be deployed (model, GPU, node, port)
- Ask for confirmation before applying
- Apply:
kubectl apply -f <manifest> - Watch:
kubectl rollout status -n inference deploy/<name> --timeout=300s - Test:
curl http://10.10.10.10:<port>/v1/models
Restart
kubectl rollout restart -n inference deploy/<name>
kubectl rollout status -n inference deploy/<name>