Domain Specific Slms

Domain-Specific Small Language Models

What this skill does

This skill is a practitioner's decision-and-technique reference for building small language models (SLMs) that are adapted to one domain, optimized to run cheaply, and deployed close to the data — including on-prem clusters, laptops, CPUs, and mobile/edge devices. It distills Guglielmo Iozzia's Domain-Specific Small Language Models.

Use it to answer questions like: Do I fine-tune or use RAG? Which quantization? How do I get this onto a CPU or a phone? How do I serve it? How do I turn a small model into an accurate, reasoning, RAG-grounded agent for my domain?

The book's thesis: you rarely need a frontier model. A carefully chosen, well-tuned model in the ~100M–10B parameter range, pushed as close as possible to where data is generated and decisions are made, delivers outsized business value — lower latency, lower cost, data privacy, regulatory fit, and no vendor lock-in.

The mental model: the SLM lifecycle

Every SLM project moves through four stages. Most questions map to one of them.

1. CHOOSE & ADAPT      Pick a base model in your domain → adapt it
                       (prompting → RAG → PEFT/LoRA → full fine-tune)
        │
2. OPTIMIZE            Make it small and fast enough for your target
                       (precision/quantization → ONNX → profiling → advanced quant)
        │
3. DEPLOY & SERVE      Put it where the data is
                       (vLLM / FastAPI server → laptop runner → Android/edge)
        │
4. COMPOSE             Make it capable and grounded
                       (RAG → vector DB → agents → GraphRAG → memory → test-time compute)

The stages are a pipeline, not a waterfall — optimization choices (e.g. INT4 for a phone) constrain adaptation choices, and composition (RAG) can substitute for adaptation (fine-tuning).

Reference routing

Load the reference file that matches the task. Read only what you need.

Master decision frameworks

These are the high-frequency calls. Deeper criteria live in the reference files.

1. Domain-specific SLM vs. generalist LLM

Prefer a domain-specific SLM when any of these hold:

• Data is sensitive / regulated (health, finance, legal, defense) — you need on-prem or air-gapped inference, no data leaving your boundary.
• Cost at scale matters — per-token API costs of a frontier model dwarf a self-hosted SLM.
• Latency / offline — edge, mobile, or real-time needs rule out a round-trip to a hosted API.
• The task is narrow and repeatable — a tuned small model matches or beats a generalist on your task while being 10–100× cheaper to run.
• You need control — reproducibility, versioning, no silent model swaps, no vendor lock-in.

Prefer a generalist (often hosted) LLM when: the task is broad/open-ended, volume is low, you have no training data, or time-to-first-prototype dominates cost. Common pattern: prototype on a generalist, then distill/fine-tune a small model for production.

2. RAG vs. fine-tuning (vs. both)

They compose. The book's headline application pattern is a fine-tuned (or PEFT-adapted) domain SLM + RAG: fine-tune for the domain's language and behavior, use RAG for current, citable facts. Default to RAG first (cheaper, faster to iterate); add PEFT/LoRA when RAG can't fix behavior; reserve full fine-tuning for when PEFT plateaus.

3. Quantization / precision picker

Order of preference — stop at the first that meets your accuracy + hardware budget:

1. FP16 / BF16 (2 bytes) — near-lossless, ~2× smaller than FP32. First step for any GPU.
2. INT8 (LLM.int8() / bitsandbytes, or ONNX dynamic/static) — ~4× smaller than FP32, small accuracy loss; great default for GPU serving and many CPUs.
3. INT4 — GPTQ (GPU, calibration-based) or ggml/gguf (llama.cpp, CPU/laptop-first). ~8× smaller; the workhorse for laptops and commodity hardware. Expect some accuracy cost.
4. Advanced — SmoothQuant (better INT8 by moving outliers weight-ward), AWQ, FlexGen (offloading for throughput on one GPU), BitNet (1.58-bit ternary, needs a BitNet-native model).

Rule of thumb: quantize as little as you can get away with. More bits = more accuracy. Pick the coarsest precision that still passes your domain evaluation, not the coarsest that runs.

4. Deployment / runner picker

Quick workflow for a new SLM project

1. Frame the decision — domain SLM vs generalist (framework 1). If SLM, pick a base model from a family already strong in your domain (see fundamentals reference).
2. Adapt cheaply first — prompting → RAG → LoRA/PEFT → full fine-tune, in that order. Stop when domain evaluation is good enough. (fundamentals + applications references)
3. Define domain evaluation early — generic benchmarks lie; build task-specific metrics and a validation set before you optimize. (fundamentals reference)
4. Optimize to the target — precision/quantization + ONNX; profile to find the real bottleneck before optimizing blindly. (inference-onnx-quantization + profiling references)
5. Deploy where the data is — pick a runner (framework 4). (deployment reference)
6. Compose — add RAG for facts, a vector DB, agents/tools, and (if you need reasoning) test-time compute. (applications reference)

Guardrails the book insists on

• Measure, don't guess. Profile before optimizing; benchmark models against your task, not a leaderboard. Accuracy claims from generic benchmarks rarely survive contact with a domain.
• Every optimization is a tradeoff. Quantization and offloading trade accuracy/latency for size/throughput. Re-run domain evaluation after every optimization step.
• Privacy is a feature, not an afterthought. For regulated data, on-prem/local/on-device inference (Ollama, MLC LLM, air-gapped ONNX) is often the reason to use an SLM at all.
• Leave the calibration knob. Hardware and quantization behave differently than the spec sheet — keep tunable params (batch size, precision, decoding params) exposed and re-tune per deployment target.