468 Index Eba8d182

vLLM-MLX Documentation

Apple Silicon MLX Backend for vLLM - GPU-accelerated Text, Image, Video & Audio on Mac

What is vLLM-MLX?

vllm-mlx brings native Apple Silicon GPU acceleration to vLLM by integrating:

• MLX: Apple's ML framework with unified memory and Metal kernels
• mlx-lm: Optimized LLM inference with KV cache and quantization
• mlx-vlm: Vision-language models for multimodal inference
• mlx-audio: Text-to-Speech and Speech-to-Text with native voices
• mlx-embeddings: Text embeddings for semantic search and RAG

Key Features

• Multimodal - Text, Image, Video & Audio in one platform
• Native GPU acceleration on Apple Silicon (M1, M2, M3, M4)
• Native TTS voices - Spanish, French, Chinese, Japanese + 5 more languages
• OpenAI API compatible - drop-in replacement for OpenAI client
• Embeddings - OpenAI-compatible /v1/embeddings endpoint
• MCP Tool Calling - integrate external tools via Model Context Protocol
• Paged KV Cache - memory-efficient caching with prefix sharing
• Continuous Batching - high throughput for multiple concurrent users

Quick Links

Getting Started

• Installation
• Quick Start

User Guides

• OpenAI-Compatible Server
• Python API
• Multimodal (Images & Video)
• Audio (STT/TTS)
• Embeddings
• Reasoning Models
• Tool Calling
• MCP & Tool Calling
• Continuous Batching

Reference

• CLI Commands
• Supported Models
• Configuration

Benchmarks

• LLM Benchmarks
• Image Benchmarks
• Video Benchmarks
• Audio Benchmarks

Development

• Architecture
• Contributing

Requirements

• macOS on Apple Silicon (M1/M2/M3/M4)
• Python 3.10+
• 8GB+ RAM recommended

License

Apache 2.0 - See LICENSE for details.