Results for “vllm”

Deploy Meta's LlamaGuard moderation model to filter LLM inputs and outputs across 6 safety categories using HuggingFace, vLLM, or FastAPI.

Serve LLMs and VLMs on NVIDIA Jetson devices using vLLM or SGLang with optimized Docker containers and quantization presets.

Set up GPU profiling, tracing, and monitoring for inference workloads using vLLM, SGLang, PyTorch, and dstack services via the Graphsignal Profiler sidecar.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware, with backend selection between vLLM, Transformers, and accelerate.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Train large language models (7B-70B+) with RLHF using PPO, GRPO, DPO, and other algorithms, accelerated by Ray and vLLM for distributed multi-GPU setups.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Quantize large language models to 8/4/3/2/1-bit precision without calibration data, using multiple optimized backends and integrations with HuggingFace Transformers, vLLM, and PEFT/LoRA.

Quantize large language models to 8/4/3/2/1-bit precision without calibration data, using multiple optimized backends for deployment with vLLM or HuggingFace Transformers.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Guarantee valid JSON, XML, or code structure during text generation using Pydantic models for type-safe outputs, supporting local models (Transformers, vLLM, llama.cpp) and maximizing inference speed with structured generation.

Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers.

Summarize recorded video clips using the LVS microservice with a VLM fallback, producing a narrative summary with timestamped events.

KV cache compression for LLM inference — 4.4x compression, 2x context capacity, near-lossless quality. ICLR 2026 paper implementation with vLLM integration.

MMBench: Is Your Multi-modal Model an All-around Player?

DocVQA: A Dataset for VQA on Document Images

Extract false-positive and false-negative gaps from VLM binary-classification-question predictions by comparing model responses against ground truth, producing a structured JSONL file and summary report for downstream root-cause analysis.

Deploy a standalone RT-VLM dense-captioning microservice and exercise its REST API endpoints for file upload, caption generation, streaming, chat completions, and Kafka integration.

Recommends an inference runtime and memory-related launch flags for LLM/VLM workloads on NVIDIA Jetson devices, based on a live memory audit snapshot.

Selects, configures, deploys, verifies, debugs, or tears down a VSS profile (base, search, lvs, warehouse, edge) for NVIDIA's video search and summarization stack.

Ask visual questions about video clips using a VSS agent's video_understanding tool, requiring a fresh look at frames rather than prior metadata or search results.

Scales Python ML workloads across clusters using Ray's distributed tasks, actors, data, and serving capabilities.

SNLI-VE: Visual Entailment Dataset

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).

Train LLMs with reinforcement learning using verl (Volcano Engine RL), supporting RLHF, GRPO, PPO, and other algorithms for scalable post-training with flexible infrastructure backends.

Dolphins: Multimodal Language Model for Driving