Manage Local Models

Manage Local Models

Get open-weight models onto the machine, keep them organized, and teach the user enough to choose well. This skill is acquisition + curation + education; to score a local model against a workload, use ../run-local-model-lab/SKILL.md.

The habit this skill installs: use a small verified model to prove the local loop, then choose the workload candidate from task fit, evidence, hardware, and time. The bootstrap model is not a route recommendation. Skip directly to a stronger rung when a weak-model sequence would delay the answer; big quality lives one understudy route away when you need it.

Safety Gates

• No download without explicit approval + a size cap. Always state model, quantization, and GB on disk first, then confirm. Weights are large; a wrong pull can fill a disk.
• Background big pulls. Announce the ETA, start the download in the background, and keep working — do not block the user on a progress bar.
• Gated weights need consent. Gemma (and some others) require accepting a license and using a Hugging Face token. Walk the user through acceptance; never print, log, or commit the token. The Ollama path serves Gemma without an HF token.
• Local-first, no upload. Pulling weights is a download only; nothing about the user's data leaves the machine.
• Make size/spec/price claims from fresh official sources (HF model cards, the Ollama library, vendor pages), never from memory — label anything indicative.

Intake

Read ~/.understudy/profile.json for hardware, installed runtimes, and the user's experience tier (set tone accordingly). Inventory what is already cached before proposing a download — the best pull is often one they already have. Disk locations and registry links are in reference.md.

If the Understudy desktop app is running, prefer its daemon. Check ~/.understudy/agent-card.json and trust the app block only after a pid check on app.pid plus a health probe of <app.base_url>/health (understudy daemon status does exactly this; then run understudy desktop capabilities; schema in ../onboard/reference.md). A running app can start/poll/cancel verified snapshot downloads into the same ~/.understudy/models cache and already serves warm slots (app.warm_models) and exposes them through understudy desktop chat with canonical runtime evidence — reuse it instead of spawning your own MLX servers or a second download of the same weights. The concrete verbs: understudy desktop model list / model catalog (cached snapshots and the bundled certified catalog), slot list|add|assign (residency), download list|start (managed downloads), and chat --slot <id>. If the app is not installed, it ships as GitHub Releases on understudylabs/understudy-agent-tools (macOS Apple Silicon .dmg, tags desktop-v*); the headless CLI covers everything in this skill without it.

Flow

1. Inventory. List installed runtimes and already-cached models, and report free disk. (ollama list; Hugging Face cache scan; MLX/LM Studio dirs — see reference.md.) Surface total disk used by weights.

2. Separate the bootstrap model from the workload candidate. For onboarding on Apple Silicon, be prescriptive: start with Understudy's verified google/gemma-4-e2b-it MLX-VLM 4-bit snapshot as the fast product smoke. It is not the default workload recommendation. Pick the workload model from task fit, evidence, hardware, and time; skip directly to a stronger rung when a small model is unlikely to answer the question. Match goals and hardware to a tier (full ladder + hardware rule-of-thumb in reference.md and ../../docs/open-model-spotlight.md):

   • Gemma 4 (Google) — verified E2B first; E4B/12B to climb; 26B-MoE / 31B dense for workstation or remote routes. Strong small-to-mid, multimodal.
   • Nemotron 3 (NVIDIA) — Nano 4B as an alternate edge rung; Nano 30B-A3B (MoE, ~4B-active speed) or Super on big-RAM boxes. Agentic-reasoning, long context.

3. Choose source + format for the runtime. Ollama library (simplest, GGUF, no HF token for Gemma); Hugging Face GGUF (llama.cpp / LM Studio); MLX builds (Apple Silicon). Quantization/format primer in reference.md.

4. Confirm the size, then use the CLI pull command. For the Understudy verified MLX ladder, use the product command after approval:

   understudy models pull gemma-4-e2b-it-qat-mlx-vlm-understudy
   

   Use --dry-run first when you need to show destination/log paths without downloading:

   understudy models pull gemma-4-e2b-it-qat-mlx-vlm-understudy --dry-run
   

   To cache every verified snapshot currently listed in the CLI catalog, use:

   understudy models pull --all
   

   The command downloads signed per-file URLs from models.understudylabs.com, writes into ~/.understudy/models, verifies sizes and hashes when present, and logs progress/ETA to ~/.understudy/agent-tools/logs/model-pull-*.log. For non-Understudy sources, use the native runtime pull (ollama pull, hf download, LM Studio) and keep the same approval boundary.

5. Serve from the manifest, not from memory. Each verified artifact ships an understudy.serving.json (see references/serving-manifest.md) encoding the exact launcher, required flags, and prescribed decode. Emit the correct serve command with the helper rather than hand-specifying flags:

   node scripts/serve-understudy-snapshot.mjs --model gemma-4-e2b-it-qat-mlx-vlm-understudy        # print the exact command
   node scripts/serve-understudy-snapshot.mjs --model gemma-4-e2b-it-qat-mlx-vlm-understudy --exec  # spawn it
   

   This is what prevents the recurring config mistakes: a forgotten --top-logprobs-k 20 (mlx_vlm defaults it to 0 and silently gates top_logprobs), the wrong sampling (greedy is off-spec for do_sample: true models), or a mis-wired MTP draft.

6. Verify + record. Once cached, run a one-line generation to confirm it loads and does tool calls if the workload needs them. The models directory plus each snapshot's catalog/serving manifest is the source of truth for what is installed; you may append an interview-time note to local_models in the profile (id, runtime, quant, size, date), but never treat that field as the library index — list the models dir to answer "what do I have?".

   Also pre-research and record the recommended serving settings, at pull time, not at bench time. Read the snapshot's understudy.serving.json (preferred — it is the machine-readable source of truth) or, if absent, generation_config.json (sampling: do_sample, temperature, top_p/top_k, schedules) and the model card's serving guidance, and record them alongside the model entry in the profile (e.g. serving: {temperature: 1.0, top_k: 64, top_p: 0.95}). Local servers are not neutral: MLX servers map an omitted temperature to 0 (greedy), which is off-spec for every do_sample: true model and breaks diffusion LMs outright. The pre-researched ladder settings live in reference.md — check there before serving anything from the verified ladder.

7. Curate. Offer to remove superseded or oversized weights to reclaim disk; show how to relocate the cache to another volume if space is tight (reference.md).

8. Point at graduation. When a local model is good but not quite enough, the path is same family, larger, remote via ../use-understudy-gateway/SKILL.md — prompts and behavior carry over. Evaluate the gap with ../run-local-model-lab/SKILL.md.

For first-timers, teach as you go: what an open-weight model is, why local is free and private, what quantization trades away, and why MoE "30B but 3B active" runs fast. For practitioners, skip it and just name the pick. If the user is hitting tool-call fidelity problems after quantization (broken JSON, model stops calling tools), route to ../optimize-local-model-compression/SKILL.md for the layer-aware compression method and the QAT group-size fix.

Output Standard

End with: runtimes + models already cached and disk used; the recommended pull (model, quant, GB, source link) and why that tier; download status (backgrounded

• ETA, or cached); profile updated; any approval still pending (gated-weight license/token, large download); and one recommended next skill/command.

References

• reference.md — download locations, registry links, format & quantization primer, gated-weights/token, disk budgeting & relocation.
• ../../docs/open-model-spotlight.md — Gemma 4 & Nemotron 3 variants, benchmarks, and hardware fit.
• ../optimize-local-model-compression/SKILL.md — layer-aware compression for tool-calling workloads (the QAT group-size fix, outcome-optimized calibration, and the stacked method).