# Llama Cpp > Run GGUF models locally with llama.cpp, including finding the right file on the Hugging Face Hub, installing, quantizing, serving, and using Python bindings. - Skill: `infinition/llama-cpp` (Agent Skill, multi-file: 7 files) - Install (CLI): `npx skillmds@latest add infinition/llama-cpp` - Raw SKILL.md: https://api.skillmd.com/api/skills/infinition/llama-cpp/raw - Safety review: CAUTION (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML, Model Training & Fine-tuning - Tags: Gguf, Huggingface, Llama Cpp Python, Llama Server, Local Inference, Quantization - Author: infinition (https://skillmd.com/u/infinition) - Updated: 2026-08-22 - Page: https://skillmd.com/skills/infinition/llama-cpp --- # llama.cpp and GGUF Run a model on this machine: no account, no per-token cost, nothing leaving the disk. llama.cpp does the running; GGUF is the single-file format it reads; the quantisation level is the dial that trades quality for memory. Two jobs live here. Finding the right file on the Hub, which is mostly a research task, and running it, which is mostly an arithmetic one about how much memory the user has. ## Finding the file The Hub's web pages are the map, but **the tree API is the territory.** A model card can list quants that were renamed, split or never uploaded. Confirm against the API before handing anyone a command. 1. **Find candidate repositories**, with `web_fetch`: ``` https://huggingface.co/models?apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending ``` `apps=llama.cpp` is the filter that matters: it excludes everything with no GGUF. 2. **Open the repository's local-app view**, which carries the maintainer's own command: ``` https://huggingface.co/?local-app=llama.cpp ``` If the snippet is readable, take the command and the recommended quant verbatim. The `Hardware compatibility` block is better than any general table here, because it was written against these specific files. 3. **Confirm against the tree API**, which lists what actually exists: ``` https://huggingface.co/api/models//tree/main?recursive=true ``` It returns a JSON array; keep entries whose `type` is `file` and whose `path` ends in `.gguf`. `path` and `size` are the truth for filenames and bytes. Three kinds of file show up together and must not be confused: - the quantised checkpoints, which is what you want; - `mmproj-*.gguf`, the vision projector for a multimodal model, which is loaded ALONGSIDE the main file and is useless alone; - shards such as `BF16/` or `*-00001-of-0000N.gguf`, an unquantised model split across files. 4. **Build the command.** Shorthand when the quant tag is standard, exact file when the repository names things its own way: ```bash llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M llama-server --hf-repo --hf-file -c 4096 ``` 5. Suggest converting from Transformers weights only when the repository publishes no GGUF at all. It is a long job and someone has usually already done it: search for ` GGUF` first. **Report the label exactly as the repository writes it.** `UD-Q4_K_M` and `IQ4_NL_XL` are real, specific names. Normalising one to `Q4_K_M` produces a command that downloads nothing. ## Install ```bash brew install llama.cpp # macOS, Linux winget install llama.cpp # Windows ``` From source, when you need a specific backend compiled in: ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release ``` Verify with `llama-cli --version` before anything else. A missing binary and a failed model load produce different errors, and confusing them costs an hour. ## The arithmetic that decides everything Before recommending a quant, work out what fits. Roughly: - **The file size is the floor**, not the total. Add context on top. - **Context costs memory too**, and it grows with `-c`. A large context window on a small machine is what turns a working setup into an out-of-memory crash halfway through a conversation. - **Leave headroom.** A model whose file is 90% of available RAM will swap, and a swapping model is slower than the CPU-only path it was supposed to beat. Then pick: | Situation | Quant | |---|---| | General chat, the default worth starting from | `Q4_K_M` | | Code or anything technical, if memory allows | `Q5_K_M`, `Q6_K` | | It must fit, quality second | `Q3_K_M`, or an `IQ` variant | | Memory is not a constraint | `Q8_0` | `IQ` quants are smaller at equal quality but need more compute to unpack, so on a slow CPU they can be the wrong trade. Below `Q3` the model degrades in ways that look like stupidity rather than compression. If the local-app view named a quant for the user's hardware, prefer it over this table. ## Serving ```bash llama-server -hf : -c 4096 --port 8080 ``` It exposes an OpenAI-compatible API, which is what lets existing clients talk to it unchanged: ```bash curl -s http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages":[{"role":"user","content":"say hello"}]}' ``` A JSON reply with a `choices` array means the model is loaded and serving. Anything else, read stderr: llama-server reports the load there, including the layer count it offloaded and the memory it took. ## Python bindings ```bash pip install llama-cpp-python CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python --force-reinstall --no-cache-dir ``` The plain install is CPU-only. GPU support is compiled in, so switching backends means reinstalling with `--force-reinstall --no-cache-dir`: without those flags pip serves the cached CPU wheel and the GPU is silently never used. ```python from llama_cpp import Llama llm = Llama(model_path="C:/models/model-q4_k_m.gguf", n_ctx=4096, n_gpu_layers=35, n_threads=8) reply = llm.create_chat_completion( messages=[{"role": "user", "content": "what is a GGUF file"}], max_tokens=256, ) print(reply["choices"][0]["message"]["content"]) for chunk in llm("explain quantisation:", max_tokens=256, stream=True): print(chunk["choices"][0]["text"], end="", flush=True) hub = Llama.from_pretrained(repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF", filename="*Q4_K_M.gguf", n_gpu_layers=35) ``` `n_gpu_layers` is how many transformer layers move to the GPU. `0` is CPU-only, `-1` is all of them, and a number too high for the available VRAM fails at load rather than falling back. ## Reporting a discovery ```text Repo: Recommended: