Llama Cpp

LLM inference in C/C++ with Python bindings. GPU acceleration via CUDA/Metal/Vulkan, 2-8 bit quantization (GGUF), KV cache, and grammar-based sampling. Run Llama, Mistral, Gemma, Phi locally.

mkurman Updated 327 repo stars
Picture from the README

File contents

mkurman/zorai commit ddd9a90d39

Frequently asked questions

npx get-skill add mkurman/llama-cpp