Adaptation Strategies

Adaptation Strategies — Fine-tune vs ICL vs RAG vs Distillation

You have a base model. It doesn't do the task well enough. You have four levers to change that, and they are not interchangeable — each changes a different thing. Picking the wrong one is the most expensive recoverable mistake in applied AI: weeks of labeling, a training run, and an eval suite, all to solve a problem a 200-token prompt would have fixed.

The whole discipline reduces to one question: what kind of gap do you have?

Memorize that table. Everything below is detail on each lever and the failure modes of using one in the wrong row.

1. The four levers — what each actually changes

In-context learning (ICL)

Prompt engineering, few-shot examples, instructions, in-prompt context. Changes behavior at inference time only — zero weight updates.

• Mechanism: the model conditions on tokens you supply. No training, no artifact, nothing persisted.
• Iteration speed: instant. Edit a string, rerun. This is its superpower — you can try 50 variants in an hour.
• Cost shape: pay in tokens on every single call, forever. A 2,000-token few-shot block on 10M calls/month is a real line item.
• Ceiling: bounded by context window and by the fact that long prompts dilute attention and raise latency. Few-shot helps format and style enormously; it does not reliably add deep capability the base model lacks.
• Reach for it: always first. Every other lever should be benchmarked against a seriously-tried prompt, not a lazy one.

RAG (retrieval-augmented generation)

Retrieve relevant chunks from a knowledge source and inject them into the prompt. For KNOWLEDGE, freshness, and attribution — not for behavior. (Full retrieval/chunking/eval detail lives in the rag skill; here we only place it among the levers.)

• Mechanism: index → retrieve → stuff into context → generate. Weights untouched.
• What it fixes: "the model doesn't know our docs / today's prices / this customer's history," and "I need a citation for every claim."
• What it does NOT fix: how the model writes, formats, or reasons. Retrieving better text does not change tone or output structure.
• Freshness: update the index, not the model. This is why current/changing facts belong in RAG, never in weights.

Fine-tuning

Update the model's weights on task data. For BEHAVIOR, format, style, domain register, and latency-by-shrinking-the-prompt. Variants:

• Mechanism: gradient descent bakes the pattern into weights. The behavior becomes free at inference (no prompt tokens spent on it).
• What it's great at: consistent output format, house style/tone, terse domain register, classification, function-calling shape, and shrinking a giant prompt into the weights so each call is cheaper/faster.
• What it's bad at: reliably injecting facts (see §3), and anything you haven't built an eval for.
• Cost shape: pay once to train (+ retrains on drift), cheap per call. The inverse of ICL.

Capability gap — the lever nobody wants to hear

A capability gap is when the base model can't do the task at any prompt — the reasoning is too deep, the chain too long, the judgment too hard. This is the one row where adaptation can't save you.

• Fine-tuning does not add capability. It moves format/behavior, not the reasoning ceiling. You can fine-tune a model to look confident on a task it fundamentally can't do — that's how you get fluent, well-formatted wrong answers.
• RAG does not add capability. It moves knowledge into context; it doesn't make the model reason better over that context.
• The only real fixes: a bigger/stronger base model, or ICL that decomposes the task — chain-of-thought, step-by-step, breaking one hard call into several easy ones. CoT/decomposition is steering the capability the model already has into reach; it is not adding capability. If decomposition + the strongest model still fails, the task is out of reach and no adaptation lever changes that.

Distillation

Train a small model on a big model's outputs for a specific task. For cost/latency at a fixed quality bar on a narrow task.

• Mechanism: generate a dataset with the teacher (or use its logits/rationales), fine-tune the student on it. The student learns the task, not the teacher's whole generality.
• What it buys: 10–50× cheaper/faster inference at near-teacher quality — on that one task. The student does not inherit the teacher's broad ability.
• Precondition: a stable, well-defined task and an eval that proves the student matches the teacher where it counts. No stable task → nothing to distill to.

2. The decision — gap → lever

                          ┌─────────────────────────────────────┐
                          │  Start: model underperforms on task  │
                          └──────────────────┬──────────────────┘
                                             ▼
                 ┌───────────────────────────────────────────────┐
                 │ Have you SERIOUSLY tried prompt + few-shot?    │
                 └───────────────┬───────────────────────┬───────┘
                              no │                    yes │
                                 ▼                        ▼
                    Do that first. Cheapest    ┌──────────────────────────┐
                    fix, instant iteration.    │ What kind of gap remains? │
                                               └─────────────┬────────────┘
              ┌──────────────────┬─────────────────────┬─────┴───────────────┐
              ▼                  ▼                     ▼                      ▼
        KNOWLEDGE          BEHAVIOR/FORMAT        CAPABILITY            COST/LATENCY
     (missing facts,    (won't follow style,   (can't do the task   (quality OK, bill
      freshness, cites)  format, tone)          at any prompt)        or p95 too high)
              │                  │                     │                      │
              ▼                  ▼                     ▼                      ▼
            RAG          Fine-tune (LoRA/DPO)    Bigger/better model    Distill teacher→
        (update index,   on 100s–1000s of        OR more ICL/CoT.       student, OR fine-
         not weights)    curated examples.       Fine-tuning will      tune a small model
                         Need an eval set.       NOT add capability    on the stable task.
                                                 the base lacks.

Read it as a priority order, not a menu. Cheapest, most reversible levers first: prompt → RAG → fine-tune small → distill → full fine-tune. Climb only when the cheaper rung is genuinely exhausted against a real eval, not against a hunch.

A blunter heuristic:

• knowledge gap → RAG
• behavior/format gap → fine-tune
• capability gap → bigger model (then ICL to steer it)
• cost/latency at scale → distill, or fine-tune a small model

3. When each is the WRONG tool (this is where money dies)

The single most common and most expensive error: fine-tuning to make the model "know" something. It is the wrong tool by mechanism, not by degree. Knowledge → RAG. Always.

4. Fine-tuning mechanics you actually need

PEFT / LoRA / QLoRA

• PEFT (parameter-efficient fine-tuning) = train a tiny fraction of params, freeze the rest. LoRA is the dominant member.
• LoRA: inserts low-rank matrices (rank r, typ. 8–64) into attention/MLP layers; only those train. ~0.1–1% of params updated. Cheap, fast, and you can swap adapters per task without storing full model copies.
• QLoRA: base weights quantized to 4-bit (NF4) + LoRA adapters in higher precision. Lets you fine-tune a 70B model on a single 48–80GB GPU. Near-LoRA quality at a fraction of the memory.
• Default choice: LoRA (or QLoRA if VRAM-bound). Full SFT only when you have lots of data, budget, ops maturity, and a measured reason adapters aren't enough.

Preference tuning (DPO and kin)

• SFT teaches imitation ("produce outputs like these"). DPO teaches judgment ("prefer this over that") from (prompt, chosen, rejected) triples. No separate reward model, simpler than RLHF/PPO.
• Use DPO/ORPO/KTO when "good" is about ranking — conciseness, safety refusals, helpfulness — that's hard to specify by example alone. Common pattern: SFT first to set the format, then DPO to refine the judgment.

Data: quality ≫ quantity

• Quality dominates. A few hundred clean, consistent, on-distribution examples beat tens of thousands of noisy ones. The model imitates your data faithfully — including its mistakes, its inconsistencies, and its format drift.
• Rough scale: LoRA behavior/format/classification often lands in the low hundreds to low thousands of examples. More data with worse consistency is a downgrade, not an upgrade.
• Consistency is the asset: every example formatted the same way, labeled by the same rubric, reflecting the exact behavior you want at inference. One inconsistent labeler poisons the run.
• Match the inference distribution: train on inputs that look like production. Train on clean prose, deploy on messy user text → it breaks.

Eval-driven, always

• No held-out eval set = do not fine-tune. Without it you cannot tell improvement from regression, and you will regress something. Build the eval before the training data.
• Hard split: train / eval examples must never overlap. Leakage (§5) makes a broken model look perfect.
• Eval the whole behavior, not just the target task — including a regression check on general capability to catch forgetting.

5. Failure modes — name them, build guards against them

Catastrophic forgetting deserves its own emphasis: weights are shared, so teaching one thing can erase another. It is the hidden cost that makes fine-tuning-for-facts doubly wrong — you pay to not learn the fact and to forget something useful. Mitigate with PEFT, conservative learning rates, mixing in general-purpose data, and a broad regression eval after every run.

6. Combining levers (the real-world answer is usually "and," not "or")

These are not exclusive. Production systems stack them:

• RAG + fine-tuned small model — the most common mature pattern. RAG supplies fresh, attributable knowledge; the small fine-tune nails format/tone and makes each call cheap. Knowledge stays swappable in the index; behavior stays baked in the weights. Best of both, and neither does the other's job.
• ICL + RAG — retrieved chunks plus a few-shot template and instructions. The default starting architecture before any training exists.
• SFT → DPO — imitate the format, then refine the judgment.
• Distill (teacher→student) + RAG — student handles the narrow task cheaply; RAG keeps it factually current without retraining.
• Fine-tune to shrink the prompt, then keep ICL for the variable part — bake the stable 3K-token instruction block into weights; keep a small dynamic few-shot slot for per-request specifics.

Guiding split when combining: knowledge → retrieval (mutable, external); behavior → weights (stable, internal). Put each thing where it belongs and they stop fighting.

7. Worked example — adjudicating a real decision

Scenario: a customer-support assistant. It must (a) answer using the current product docs and this customer's plan/usage, and (b) reply in the company's terse, no-fluff house voice with a fixed structure (answer → next step → doc link).

The lazy instinct is "fine-tune it on our support transcripts." Walk the gaps instead:

1. Knowledge gap? Yes — current docs, per-customer state. That is a knowledge row → RAG. Index the docs + pull the customer record at query time.
2. The trap, named: fine-tuning on transcripts to "teach the product" is the §3 error in full. The facts go stale the day docs change; the model will hallucinate plausible-but-wrong details in perfect house format; and the run will forget unrelated ability. You'd pay to make the model worse and more confident.
3. Behavior gap? Yes — house voice + fixed structure won't hold from prompting alone across millions of varied tickets. That is a behavior row → LoRA fine-tune on a few hundred clean, consistently-formatted exemplar replies. This also shrinks the prompt: the style instructions move into weights, so every call is cheaper and faster.
4. Capability gap? No. A strong base model can do support reasoning given the right context. No bigger model needed.

Resolution: RAG + LoRA together (§6's canonical pair). Knowledge lives in the swappable index and stays fresh; behavior lives in the adapter and stays consistent. Each lever does its own job and they stop fighting.

Mini-case (cost only): a field-extraction step where a good prompt already hits the quality bar — but it's 10M calls/month on a frontier model. Quality gap = zero; the entire problem is cost/latency. → Distill the frontier model's outputs into a small student (or fine-tune a small model directly) once the schema is frozen. No new quality is needed, so no bigger model and no RAG — just move the same quality onto a cheaper engine.

8. Comparison table — the one-screen summary

9. Operating rules (do / don't)

Do

• Try a serious prompt + few-shot first, and benchmark every other lever against it on the same eval.
• Build the held-out eval set before you collect training data or stand up retrieval.
• Put facts/knowledge in RAG; put behavior/format/tone in weights.
• Prefer LoRA/QLoRA over full SFT; prefer fine-tuning a small model over reaching for a bigger one when the gap is format, not capability.
• Treat data quality and consistency as the primary lever — a few hundred clean examples beat thousands of noisy ones.
• Regression-test broad capability after every fine-tune to catch forgetting.
• Budget the whole lifecycle of any weight change: data refresh, retrain, re-eval, redeploy.
• Stabilize the task + eval before distilling.

Don't

• Don't fine-tune to add facts. (Use RAG.)
• Don't use RAG to fix tone/format. (Use prompt or fine-tune.)
• Don't fine-tune before prompt + RAG are genuinely exhausted against a real eval.
• Don't fine-tune without a held-out eval set — you can't tell improvement from regression.
• Don't let train and eval data overlap. Verify zero leakage.
• Don't distill an unstable task — the student is obsolete on arrival.
• Don't keep paying a giant repeated prompt every call when fine-tuning would bake it in.
• Don't assume "one training run." Assume a retrain treadmill and price it in.

Cross-references: retrieval/chunking/index/retrieval-eval detail → rag skill. Prompt structure, few-shot design, output schemas → prompt-engineer skill. Eval-suite construction → your evaluation skill of record.