# Domain Specific Slms

> Design, adapt, optimize, and ship small, domain-specific language models (SLMs) — the practitioner's playbook from Guglielmo Iozzia's *Domain-Specific Small Language Models* (Manning). Use whenever someone is working with SLMs or trying to make an LLM run cheaply, privately, or on constrained hardware. Triggers: "should I fine-tune or use RAG?", "which quantization should I use?", "run an LLM on a laptop / CPU / phone / edge device", "shrink / speed up / reduce the cost of a model", LoRA / PEFT, ONNX / ONNX Runtime, INT8 / INT4 / GPTQ / ggml / gguf / AWQ / SmoothQuant / BitNet / FlexGen, vLLM, Ollama / LM Studio / Jan / Cortex / llama.cpp, MLC LLM, DeepSpeed, model profiling, "deploy an LLM with FastAPI / vLLM", domain-specific fine-tuning (code, chemistry, proteins, legal, medical, finance), RAG pipelines, vector databases, GraphRAG, agentic RAG, LLM agents, long/short-term memory, test-time compute, reasoning models, OptiLLM, or building a private/on-prem generative AI system in a regulated industry. Also t

- Skill: `jordangaston/domain-specific-slms` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add jordangaston/domain-specific-slms`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jordangaston/domain-specific-slms/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: jordangaston (https://skillmd.com/u/jordangaston)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jordangaston/domain-specific-slms

---


# Domain-Specific Small Language Models

## What this skill does

This skill is a practitioner's decision-and-technique reference for building **small language
models (SLMs)** that are **adapted to one domain**, **optimized to run cheaply**, and
**deployed close to the data** — including on-prem clusters, laptops, CPUs, and mobile/edge
devices. It distills Guglielmo Iozzia's *Domain-Specific Small Language Models*.

Use it to answer questions like: *Do I fine-tune or use RAG? Which quantization? How do I get
this onto a CPU or a phone? How do I serve it? How do I turn a small model into an accurate,
reasoning, RAG-grounded agent for my domain?*

The book's thesis: **you rarely need a frontier model.** A carefully chosen, well-tuned model in
the **~100M–10B parameter** range, pushed as close as possible to where data is generated and
decisions are made, delivers outsized business value — lower latency, lower cost, data privacy,
regulatory fit, and no vendor lock-in.

## The mental model: the SLM lifecycle

Every SLM project moves through four stages. Most questions map to one of them.

```
1. CHOOSE & ADAPT      Pick a base model in your domain → adapt it
                       (prompting → RAG → PEFT/LoRA → full fine-tune)
        │
2. OPTIMIZE            Make it small and fast enough for your target
                       (precision/quantization → ONNX → profiling → advanced quant)
        │
3. DEPLOY & SERVE      Put it where the data is
                       (vLLM / FastAPI server → laptop runner → Android/edge)
        │
4. COMPOSE             Make it capable and grounded
                       (RAG → vector DB → agents → GraphRAG → memory → test-time compute)
```

The stages are a pipeline, not a waterfall — optimization choices (e.g. INT4 for a phone)
constrain adaptation choices, and composition (RAG) can substitute for adaptation (fine-tuning).

## Reference routing

Load the reference file that matches the task. Read only what you need.

| Topic | Reference file | Read when |
|-------|----------------|-----------|
| What SLMs are, generalist-vs-domain decision, open-source ecosystem, data prep, fine-tuning, LoRA/PEFT, RAG-vs-fine-tune, end-to-end tuning + evaluation | `references/01-fundamentals-and-adaptation.md` | Choosing a base model; deciding how to adapt; preparing data; running a fine-tune; evaluating a tuned model |
| Generation/decoding params, inference cost math, GPU utilization, batching, DeepSpeed, ONNX format + Runtime + providers, exporting to ONNX, precision formats, INT8/INT4, LLM.int8(), GPTQ, ggml/gguf | `references/02-inference-onnx-quantization.md` | Tuning generation quality; cutting inference cost; converting to ONNX; picking a quantization method |
| Domain playbook: code generation, chemistry/proteins/materials; advanced quantization (FlexGen, SmoothQuant, BitNet 1.58-bit) | `references/03-domain-usecases-advanced-quant.md` | Adapting an SLM to a specialized/scientific domain; squeezing a model past INT4 |
| Profiling ONNX graphs, deployment & serving (vLLM, FastAPI, MLC LLM, Android), local runners (Ollama, LM Studio, Jan, Cortex) | `references/04-profiling-deployment-serving.md` | Finding bottlenecks; choosing how/where to serve; running locally or on-device |
| RAG pipelines, vector DBs, agents, GraphRAG, agentic RAG, memory, test-time compute, reasoning SLMs, OptiLLM | `references/05-llm-applications-and-test-time-compute.md` | Building a real application around the model; grounding, agents, or reasoning |

## Master decision frameworks

These are the high-frequency calls. Deeper criteria live in the reference files.

### 1. Domain-specific SLM vs. generalist LLM

Prefer a **domain-specific SLM** when any of these hold:

- **Data is sensitive / regulated** (health, finance, legal, defense) — you need on-prem or
  air-gapped inference, no data leaving your boundary.
- **Cost at scale matters** — per-token API costs of a frontier model dwarf a self-hosted SLM.
- **Latency / offline** — edge, mobile, or real-time needs rule out a round-trip to a hosted API.
- **The task is narrow and repeatable** — a tuned small model matches or beats a generalist on
  *your* task while being 10–100× cheaper to run.
- **You need control** — reproducibility, versioning, no silent model swaps, no vendor lock-in.

Prefer a **generalist (often hosted) LLM** when: the task is broad/open-ended, volume is low,
you have no training data, or time-to-first-prototype dominates cost. Common pattern: prototype
on a generalist, then distill/fine-tune a small model for production.

### 2. RAG vs. fine-tuning (vs. both)

| Use **RAG** when… | Use **fine-tuning / PEFT** when… |
|---|---|
| Knowledge changes often (docs, tickets, policies) | The *behavior/format/style* must change (tone, schema, code dialect) |
| You need source citations / auditability | You need a new *skill* or domain *language*, not just facts |
| Facts are large and enumerable | Latency/prompt-length budget is tight (no room for retrieved context) |
| You must add/remove knowledge without retraining | The domain vocabulary/tokens are alien to the base model |
| Hallucination on facts is the main risk | You need the capability offline with no retrieval store |

**They compose.** The book's headline application pattern is a **fine-tuned (or PEFT-adapted)
domain SLM + RAG**: fine-tune for the domain's *language and behavior*, use RAG for *current,
citable facts*. Default to RAG first (cheaper, faster to iterate); add PEFT/LoRA when RAG can't
fix behavior; reserve full fine-tuning for when PEFT plateaus.

### 3. Quantization / precision picker

Order of preference — stop at the first that meets your accuracy + hardware budget:

1. **FP16 / BF16** (2 bytes) — near-lossless, ~2× smaller than FP32. First step for any GPU.
2. **INT8** (LLM.int8() / bitsandbytes, or ONNX dynamic/static) — ~4× smaller than FP32, small
   accuracy loss; great default for GPU serving and many CPUs.
3. **INT4** — GPTQ (GPU, calibration-based) or ggml/gguf (llama.cpp, CPU/laptop-first). ~8×
   smaller; the workhorse for laptops and commodity hardware. Expect some accuracy cost.
4. **Advanced** — SmoothQuant (better INT8 by moving outliers weight-ward), AWQ, FlexGen
   (offloading for throughput on one GPU), BitNet (1.58-bit ternary, needs a BitNet-native model).

Rule of thumb: **quantize as little as you can get away with.** More bits = more accuracy. Pick
the coarsest precision that still passes your domain evaluation, not the coarsest that runs.

### 4. Deployment / runner picker

| Target | Reach for | Why |
|---|---|---|
| High-throughput GPU serving | **vLLM** (offline batch or OpenAI-compatible online server) | PagedAttention + continuous batching → best tokens/sec |
| Simple custom API / microservice | **FastAPI** wrapping the model | Full control, easy to embed in an app |
| Developer laptop, quick local use | **Ollama** (Modelfile), **LM Studio** (GUI + Python SDK), **Jan**, **Cortex** | Zero-infra, private, gguf-based |
| CPU-only / pure C++ | **llama.cpp** / **ggml/gguf** | No GPU, portable |
| Phone / Android / cross-platform edge | **MLC LLM** (compiles models per-target) | On-device, offline, private |
| CPU/GPU-portable optimized runtime | **ONNX Runtime** | One graph, many execution providers |

## Quick workflow for a new SLM project

1. **Frame the decision** — domain SLM vs generalist (framework 1). If SLM, pick a base model
   from a family already strong in your domain (see fundamentals reference).
2. **Adapt cheaply first** — prompting → RAG → LoRA/PEFT → full fine-tune, in that order. Stop
   when domain evaluation is good enough. (fundamentals + applications references)
3. **Define domain evaluation early** — generic benchmarks lie; build task-specific metrics and a
   validation set before you optimize. (fundamentals reference)
4. **Optimize to the target** — precision/quantization + ONNX; profile to find the real
   bottleneck before optimizing blindly. (inference-onnx-quantization + profiling references)
5. **Deploy where the data is** — pick a runner (framework 4). (deployment reference)
6. **Compose** — add RAG for facts, a vector DB, agents/tools, and (if you need reasoning)
   test-time compute. (applications reference)

## Guardrails the book insists on

- **Measure, don't guess.** Profile before optimizing; benchmark models against *your* task, not
  a leaderboard. Accuracy claims from generic benchmarks rarely survive contact with a domain.
- **Every optimization is a tradeoff.** Quantization and offloading trade accuracy/latency for
  size/throughput. Re-run domain evaluation after every optimization step.
- **Privacy is a feature, not an afterthought.** For regulated data, on-prem/local/on-device
  inference (Ollama, MLC LLM, air-gapped ONNX) is often the *reason* to use an SLM at all.
- **Leave the calibration knob.** Hardware and quantization behave differently than the spec
  sheet — keep tunable params (batch size, precision, decoding params) exposed and re-tune per
  deployment target.