Results for “llama-server”

Run LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

Provides expert guidance for fine-tuning LLMs with LLaMA-Factory, covering WebUI no-code, 100+ models, 2/3/4/5/6/8-bit QLoRA, and multimodal support.

Run GGUF models locally with llama.cpp, including finding the right file on the Hugging Face Hub, installing, quantizing, serving, and using Python bindings.

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

Deploy Meta's LlamaGuard moderation model to filter LLM inputs and outputs across 6 safety categories using HuggingFace, vLLM, or FastAPI.

Deploy and serve LLMs in production with vLLM, Ollama, TGI, and llama.cpp, including quantization and GPU optimization.

Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.

Convert and quantize models to GGUF format for efficient CPU/GPU inference with llama.cpp, supporting 2-8 bit quantization and Apple Silicon acceleration.

Runs the open-source LLaVA vision-language model for image understanding, captioning, visual question answering, and multi-turn image conversations, including setup, inference, and training guidance.