local-models
Quick access to local LLMs through llama.cpp, reusing the GGUF models already
pulled by Ollama (no re-download for text and embeddings). Everything runs on the
machine — no API key, no network, no per-token cost.
When to use this skill
Reach for local models instead of a cloud API when the task is:
- Privacy-sensitive — redacting PII, processing personal notes, health data, secrets-adjacent text. The data never leaves the machine.
- Offline — no network available, or the user explicitly wants local-only.
- High-volume / low-stakes — classifying or tagging hundreds of items, where a small model is good enough and cloud cost/latency would add up.
- A fast throwaway — a quick summary, translation, or "what is this" where round-tripping to a frontier model is overkill.
Prefer a frontier (Claude) model when the task needs strong reasoning, long
context, careful code, or high accuracy — these local models are small (0.6–4B).
The core trick: reuse Ollama's models
Ollama stores model weights as extension-less GGUF blobs under
~/.ollama/models/blobs/. These are ordinary GGUF files — llama.cpp loads
them directly. scripts/ollama_blob.py reads Ollama's manifests and resolves a
friendly name (e.g. qwen2.5:3b) to its weights blob path. No conversion, no
duplicate downloads.
Usage
The entry point is scripts/lm. Run scripts/lm help for the full list. Invoke
it with an absolute path, e.g. ~/ai_projects/claude-skills/local-models/scripts/lm.
lm models # list local models (text / vision / embed)
lm ask [MODEL] "PROMPT" # one-shot prompt (default qwen2.5:3b)
lm chat [MODEL] # interactive REPL
# Text presets — accept a file path, inline text, OR stdin:
lm summarize report.md
cat notes.txt | lm tldr
lm keywords article.txt
lm anonymize transcript.txt # → [NAME] [EMAIL] [PHONE] [ADDRESS] ...
lm proofread draft.md
lm translate German "Good morning"
lm classify "praise,complaint,question" feedback.txt # → one label
lm extract "invoice_number, total, due_date" invoice.txt # → JSON
# Vision (downloads model+projector once via HuggingFace — see note below):
lm describe-image photo.jpg
lm tag-image screenshot.png
lm vision photo.jpg "What brand is the shoe?"
# Embeddings & serving:
lm embed "text to embed" # → OpenAI-style JSON vector
lm serve qwen2.5:3b 8080 # OpenAI-compatible server on :8080
Output is clean (just the answer) — the wrapper drives llama-completion in
single-turn mode and strips the chat-template scaffolding and llama.cpp logs.
Choosing a model
Defaults are tuned for clean, fast output and can be overridden per call:
- General text presets →
qwen2.5:3b (LM_TEXT_MODEL)
- Classify / extract →
qwen2.5:3b (LM_REASON_MODEL), run at temperature 0
- Embeddings →
jeffh/intfloat-multilingual-e5-large:f16 (LM_EMBED_MODEL)
- Other envs:
LM_NTOK (max tokens), LM_VISION_HF (vision repo), LM_DEBUG=1 (show llama.cpp logs)
Pass an explicit model as the first argument to ask/chat/embed/serve
(e.g. lm ask qwen3:4b "...").
Critical gotchas
- Ollama's
gemma3 GGUF does NOT load in stock llama.cpp. It fails with
key not found in model: gemma3.attention.layer_norm_rms_epsilon because
Ollama writes custom metadata keys mainline llama.cpp doesn't read. Use a
qwen* model instead, or pull a community gemma3 GGUF via -hf. This is why
the defaults are qwen, not gemma3.
qwen3:4b emits <think>…</think> reasoning blocks before its answer.
Fine for ask/chat, but it pollutes preset output (JSON, labels) — the
presets default to qwen2.5:3b to avoid this.
- Vision has no Ollama blob to reuse. Ollama did not store an
mmproj
(vision projector) for qwen2.5vl, and llama.cpp needs one. So the vision
commands use llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF, which
downloads model+projector (2–3 GB) into `/.cache/llama.cpp` on first use,
then runs offline. Warn the user before the first vision call.
- Each one-shot call reloads the model (a few seconds for these small
models). For many sequential calls, start a server once with
lm serve and
hit http://localhost:8080/v1/chat/completions — see
references/serving-and-embeddings.md.
Reference material
- references/serving-and-embeddings.md —
running
llama-server as an OpenAI-compatible endpoint (and pointing the llm
CLI or any OpenAI client at it), plus local embeddings / RAG patterns with
llama-embedding.
Requirements
llama.cpp installed (brew install llama.cpp) — provides llama-completion,
llama-mtmd-cli, llama-embedding, llama-server.
- Ollama with at least one pulled model (for the blob-reuse path).
python3 for
the resolver. No API keys.
1---
2name: local-models
3description: Run quick, offline, private LLM tasks on local models via llama.cpp, reusing models already downloaded by Ollama. Use for cheap/bulk text work (summarize, classify, extract JSON, anonymize PII, translate, proofread, keywords), local embeddings, and offline image description — and prefer it over a cloud API whenever a task is privacy-sensitive, must run offline, is high-volume/low-stakes, or just needs a fast throwaway answer. Provides an `lm` CLI wrapper plus an OpenAI-compatible local server.
4metadata:
5 source: https://skillmd.com/skills/glebis/local-models
6---
7
8# local-models
9
10Quick access to local LLMs through **llama.cpp**, reusing the GGUF models already
11pulled by Ollama (no re-download for text and embeddings). Everything runs on the
12machine — no API key, no network, no per-token cost.
13
14## When to use this skill
15
16Reach for local models instead of a cloud API when the task is:
17- **Privacy-sensitive** — redacting PII, processing personal notes, health data, secrets-adjacent text. The data never leaves the machine.
18- **Offline** — no network available, or the user explicitly wants local-only.
19- **High-volume / low-stakes** — classifying or tagging hundreds of items, where a small model is good enough and cloud cost/latency would add up.
20- **A fast throwaway** — a quick summary, translation, or "what is this" where round-tripping to a frontier model is overkill.
21
22Prefer a frontier (Claude) model when the task needs strong reasoning, long
23context, careful code, or high accuracy — these local models are small (0.6–4B).
24
25## The core trick: reuse Ollama's models
26
27Ollama stores model weights as extension-less GGUF blobs under
28`~/.ollama/models/blobs/`. **These are ordinary GGUF files** — llama.cpp loads
29them directly. `scripts/ollama_blob.py` reads Ollama's manifests and resolves a
30friendly name (e.g. `qwen2.5:3b`) to its weights blob path. No conversion, no
31duplicate downloads.
32
33## Usage
34
35The entry point is `scripts/lm`. Run `scripts/lm help` for the full list. Invoke
36it with an absolute path, e.g. `~/ai_projects/claude-skills/local-models/scripts/lm`.
37
38```bash
39lm models # list local models (text / vision / embed)
40lm ask [MODEL] "PROMPT" # one-shot prompt (default qwen2.5:3b)
41lm chat [MODEL] # interactive REPL
42
43# Text presets — accept a file path, inline text, OR stdin:
44lm summarize report.md
45cat notes.txt | lm tldr
46lm keywords article.txt
47lm anonymize transcript.txt # → [NAME] [EMAIL] [PHONE] [ADDRESS] ...
48lm proofread draft.md
49lm translate German "Good morning"
50lm classify "praise,complaint,question" feedback.txt # → one label
51lm extract "invoice_number, total, due_date" invoice.txt # → JSON
52
53# Vision (downloads model+projector once via HuggingFace — see note below):
54lm describe-image photo.jpg
55lm tag-image screenshot.png
56lm vision photo.jpg "What brand is the shoe?"
57
58# Embeddings & serving:
59lm embed "text to embed" # → OpenAI-style JSON vector
60lm serve qwen2.5:3b 8080 # OpenAI-compatible server on :8080
61```
62
63Output is clean (just the answer) — the wrapper drives `llama-completion` in
64single-turn mode and strips the chat-template scaffolding and llama.cpp logs.
65
66### Choosing a model
67
68Defaults are tuned for clean, fast output and can be overridden per call:
69- General text presets → `qwen2.5:3b` (`LM_TEXT_MODEL`)
70- Classify / extract → `qwen2.5:3b` (`LM_REASON_MODEL`), run at temperature 0
71- Embeddings → `jeffh/intfloat-multilingual-e5-large:f16` (`LM_EMBED_MODEL`)
72- Other envs: `LM_NTOK` (max tokens), `LM_VISION_HF` (vision repo), `LM_DEBUG=1` (show llama.cpp logs)
73
74Pass an explicit model as the first argument to `ask`/`chat`/`embed`/`serve`
75(e.g. `lm ask qwen3:4b "..."`).
76
77## Critical gotchas
78
79- **Ollama's `gemma3` GGUF does NOT load in stock llama.cpp.** It fails with
80 `key not found in model: gemma3.attention.layer_norm_rms_epsilon` because
81 Ollama writes custom metadata keys mainline llama.cpp doesn't read. Use a
82 `qwen*` model instead, or pull a community gemma3 GGUF via `-hf`. This is why
83 the defaults are qwen, not gemma3.
84- **`qwen3:4b` emits `<think>…</think>` reasoning blocks** before its answer.
85 Fine for `ask`/`chat`, but it pollutes preset output (JSON, labels) — the
86 presets default to `qwen2.5:3b` to avoid this.
87- **Vision has no Ollama blob to reuse.** Ollama did not store an `mmproj`
88 (vision projector) for `qwen2.5vl`, and llama.cpp needs one. So the vision
89 commands use `llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF`, which
90 downloads model+projector (~2–3 GB) into `~/.cache/llama.cpp` on first use,
91 then runs offline. Warn the user before the first vision call.
92- **Each one-shot call reloads the model** (a few seconds for these small
93 models). For many sequential calls, start a server once with `lm serve` and
94 hit `http://localhost:8080/v1/chat/completions` — see
95 [references/serving-and-embeddings.md](references/serving-and-embeddings.md).
96
97## Reference material
98
99- [references/serving-and-embeddings.md](references/serving-and-embeddings.md) —
100 running `llama-server` as an OpenAI-compatible endpoint (and pointing the `llm`
101 CLI or any OpenAI client at it), plus local embeddings / RAG patterns with
102 `llama-embedding`.
103
104## Requirements
105
106- `llama.cpp` installed (`brew install llama.cpp`) — provides `llama-completion`,
107 `llama-mtmd-cli`, `llama-embedding`, `llama-server`.
108- Ollama with at least one pulled model (for the blob-reuse path). `python3` for
109 the resolver. No API keys.