Results for “vllm”
20 skills
orchestra-research Serving Llms Vllm
Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.
10.4k · bundle
nvidia Jetson LLM Benchmark
Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.
2.2k · bundle
qhjqhj00 Hqq Quantization
Quantize LLMs to 8/4/3/2/1-bit precision without calibration data, using multiple backends and HuggingFace/vLLM integration.
3 · bundle
oyi77 LLM Deployment
Deploy and serve LLMs in production with vLLM, Ollama, TGI, and llama.cpp, including quantization and GPU optimization.
10
orchestra-research Llamaguard
Deploy Meta's LlamaGuard moderation model to filter LLM inputs and outputs across 6 safety categories using HuggingFace, vLLM, or FastAPI.
10.4k
huggingface Huggingface Community Evals
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware, with backend selection between vLLM, Transformers, and accelerate.
10.8k · bundle
More results
orchestra-research Openrlhf Training
Train large language models (7B-70B+) with RLHF using PPO, GRPO, DPO, and other algorithms, accelerated by Ray and vLLM for distributed multi-GPU setups.
10.4k · bundle
nvidia Jetson Speculative Decoding
Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.
2.2k · bundle
orchestra-research Evaluating Llms Harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.
10.4k · bundle
majiayu000 Hqq Quantization
Quantize large language models to 8/4/3/2/1-bit precision without calibration data, using multiple optimized backends and integrations with HuggingFace Transformers, vLLM, and PEFT/LoRA.
567 · bundle
nvidia Tao Analyze Gaps Vlm Bcq
Extract false-positive and false-negative gaps from VLM binary-classification-question predictions by comparing model responses against ground truth, producing a structured JSONL file and summary report for downstream root-cause analysis.
2.2k · bundle
nvidia Jetson Inference Mem Tune
Recommends an inference runtime and memory-related launch flags for LLM/VLM workloads on NVIDIA Jetson devices, based on a live memory audit snapshot.
2.2k · bundle
orchestra-research Hqq Quantization
Quantize large language models to 8/4/3/2/1-bit precision without calibration data, using multiple optimized backends for deployment with vLLM or HuggingFace Transformers.
10.4k · bundle
orchestra-research Tensorrt LLM
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).
10.4k · bundle
orchestra-research Verl Rl Training
Train LLMs with reinforcement learning using verl (Volcano Engine RL), supporting RLHF, GRPO, PPO, and other algorithms for scalable post-training with flexible infrastructure backends.
10.4k · bundle
majiayu000 Ray
Scales Python ML workloads across clusters using Ray's distributed tasks, actors, data, and serving capabilities.
567 · bundle
majiayu000 Awq Quantization
Quantize large language models to 4-bit precision using activation-aware weight quantization, reducing memory footprint and speeding up inference with minimal accuracy loss.
567 · bundle
orchestra-research Speculative Decoding
Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques for 1.5-3.6× speedup without quality loss.
10.4k · bundle
orchestra-research Awq Quantization
Quantize large language models to 4-bit using activation-aware weight quantization, achieving ~3x speedup with minimal accuracy loss for deployment on limited GPU memory.
10.4k · bundle
orchestra-research Torchforge Rl Training
Train reinforcement learning models using torchforge, Meta's PyTorch-native RL library for scalable, algorithm-focused experimentation with GRPO, DAPO, and custom loss functions.
10.4k · bundle