Vllm

Use when self-hosting an open-weight LLM for high-throughput concurrent serving with vLLM — running an OpenAI-compatible endpoint, splitting a model across GPUs with tensor or pipeline parallelism, loading quantized weights, serving one or many LoRA adapters, and debugging KV-cache OOM from memory-utilisation and context-length flags. NOT renting or provisioning the GPU box (that is `runpod` or `modal`), NOT single-user laptop inference (that is `ollama`), NOT a hosted inference API you do not operate (that is `together-fireworks` or `huggingface`).

ericrisco Updated 610 repo stars
rsc-harness: you build the product, rsc-harness builds your agent's harness, following strict best practices instead of improvising one.

File contents

ericrisco/rsc-harness commit c840aa6e97

Frequently asked questions

npx get-skill add ericrisco/vllm