Model Selection

Model Selection

The Core Problem

Choosing a model is not just a technical decision — it's a cost, latency, privacy, and capability tradeoff. The most common mistakes are: using the most powerful model everywhere (expensive and slow), not considering local models when privacy matters, and jumping to fine-tuning when better prompting would have worked.

The discipline: Understand your constraints first. The right model is the weakest model that reliably solves your problem.

Step 1: Understand Your Constraints

Before looking at any specific model, answer these questions:

What is the task?
(text generation, classification, summarization, code, reasoning, etc.)

What is the acceptable latency?
(< 1s for real-time UX, 1-10s is acceptable, > 10s is background/batch)

What is the cost sensitivity?
(prototype: cost doesn't matter, production: minimize cost per query)

Does the data contain sensitive or private information?
(PII, proprietary data, medical records, legal documents)

Do you need the model to run offline or on-device?
(air-gapped environment, no internet, edge deployment)

What is the deployment environment?
(cloud server, local laptop, edge device, mobile)

What context window size do you need?
(short queries, long documents, entire codebases)

Step 2: API vs Local Model

This is the first and most important decision.

Use API (cloud-hosted) when:

• You need the best possible capability
• Latency of 1-5s is acceptable
• Data is not sensitive or you trust the provider's privacy policy
• You don't want to manage infrastructure
• You're prototyping or in early stages
• Usage is unpredictable (API scales automatically)

Use local model when:

• Data is sensitive, private, or regulated (PII, medical, legal, financial)
• You need offline capability or air-gapped environment
• You want zero per-query cost after setup
• Latency requirements are strict and you have good hardware
• You want full control over the model and its outputs
• You're building for edge or on-device deployment

The hybrid pattern

Many production systems use both:

• Local model for sensitive or high-volume operations
• API model for complex reasoning that requires maximum capability
• Local model as fallback when API is unavailable

Step 3A: If Using API Models

Provider comparison

Model tier selection

Use the smallest model that works. Start with the medium tier as your baseline, then try smaller if it works or larger if it doesn't.

Practical approach:

1. Start with a medium-tier model
2. If it works reliably → try the small tier (cheaper)
3. If it doesn't work reliably → try the large tier (more capable)
4. If large tier still fails → the problem may need fine-tuning or a different approach

API configuration to record

Provider: [Anthropic / OpenAI / Google / Mistral / other]
Model: [exact model name and version]
Temperature: [0 for deterministic, 0.7 for creative]
Max tokens: [set explicitly — never leave unlimited]
Top-p: [usually leave at default]
System prompt: [yes/no — stored where?]
API key storage: [environment variable name]
Rate limits: [requests per minute, tokens per minute]
Estimated cost per query: [input tokens × rate + output tokens × rate]

Step 3B: If Using Local Models

Hardware requirements by model size

GPU vs CPU: GPU is 5-20x faster for inference. CPU-only is fine for development and low-latency-tolerant production use.

Local model options by use case

General purpose / instruction following:

• Llama 3.2 3B — fast, good for simple tasks, fits on any hardware
• Llama 3.1 8B — good balance, recommended starting point
• Llama 3.3 70B — near-API quality, needs good hardware
• Mistral 7B / Mixtral 8x7B — strong for its size, good instruction following

Code:

• Qwen2.5-Coder 7B / 14B — best open-source code model at its size
• DeepSeek-Coder V2 — strong code capabilities
• CodeLlama 13B — solid, well-tested

Reasoning:

• Qwen3 8B / 14B — strong reasoning, supports thinking mode
• DeepSeek-R1 (distilled) — reasoning-focused, various sizes

Note: Local model options evolve rapidly. Check Ollama's library or Hugging Face's Open LLM Leaderboard for current rankings before committing to a model.

Embeddings:

• nomic-embed-text — good general embeddings, small and fast
• snowflake-arctic-embed — strong retrieval performance
• mxbai-embed-large — high quality, larger

Multimodal (vision + text):

• Llama 3.2 Vision 11B — good vision capability, reasonable size
• Qwen2.5-VL 7B — strong multimodal performance

Local deployment options

Recommended for most cases: Ollama for development, vLLM for production GPU serving.

Local model configuration to record

Model: [model name and version]
Model file: [path or Ollama tag]
Quantization: [Q4_K_M / Q5_K_M / Q8_0 / fp16 — higher = better quality, more memory]
Context window: [num_ctx in Ollama]
Deployment tool: [Ollama / llama.cpp / vLLM / other]
API endpoint: [http://localhost:11434 for Ollama]
Temperature: [0 for deterministic, higher for creative]
Hardware: [CPU only / GPU model / RAM available]
Inference speed: [tokens per second — measure this]

Step 4: Prompt Engineering vs Fine-Tuning

Always try prompt engineering first. Fine-tuning is expensive, time-consuming, and introduces a maintenance burden. Most problems that seem to need fine-tuning can be solved with better prompting.

Try prompt engineering first when:

• The base model knows how to do the task but produces inconsistent output format
• You need the model to follow specific style or tone guidelines
• The task requires few-shot examples to guide behavior
• You haven't yet tried chain-of-thought, system prompts, or structured output

Consider fine-tuning when:

• Prompt engineering has been thoroughly tried and still fails
• You need highly consistent structured output and the model keeps deviating
• The task requires domain-specific knowledge not in the base model
• You have 100+ high-quality labeled examples (see minimum data requirements below)
• Latency is critical and a smaller fine-tuned model can replace a larger base model
• You need to reduce prompt token usage significantly (system prompt baked into weights)

Fine-tuning options

Minimum data requirements:

• Classification / structured output: 50-200 examples
• Style / tone: 100-500 examples
• Domain knowledge: 500-2000 examples
• Complex reasoning: 1000+ examples

Step 5: Record Your Setup

Every model configuration decision should be recorded. This prevents "I forgot which model we were using" and makes it easy to reproduce results.

## Model Configuration — [Project Name]

**Date:** YYYY-MM-DD
**Task:** [what this model is doing]

### Deployment
- Type: API / Local
- Provider/Tool: [name]
- Model: [exact name and version]

### Configuration
- Temperature: [value]
- Max tokens: [value]
- Context window used: [value]
- System prompt: [yes/no — location if yes]

### Hardware (if local)
- CPU/GPU: [model]
- RAM: [amount]
- VRAM: [amount if GPU]
- Quantization: [value]
- Inference speed: [tokens/second]

### Cost (if API)
- Input cost: [$X per 1M tokens]
- Output cost: [$X per 1M tokens]
- Estimated cost per query: [$X]
- Estimated monthly cost at [N] queries/day: [$X]

### Decision Log
- Why this model over alternatives: [reason]
- What was tried before: [if anything]
- Known limitations: [what it struggles with]

### Fine-tuning (if applicable)
- Base model: [name]
- Method: [LoRA / full / API]
- Training data: [size, source]
- Evaluation metric: [what improved and by how much]

Quick Decision Tree

Do you have sensitive/private data, or need offline?
├── Yes → Local model
│   ├── Have GPU with 8GB+ VRAM? → Llama 3.1 8B via Ollama (start here)
│   ├── CPU only? → Llama 3.2 3B via Ollama (fastest on CPU)
│   └── Need best quality? → Llama 3.3 70B (needs 32GB+ RAM)
└── No → API model
    ├── Prototyping / cost doesn't matter → Claude Sonnet or GPT-4o
    ├── Production, cost-sensitive, simple task → Claude Haiku or GPT-4o-mini
    ├── Production, quality matters → Claude Sonnet or GPT-4o
    └── Complex reasoning, quality critical → Claude Opus or o3

Is output format inconsistent or task-specific?
├── Haven't tried few-shot examples yet → Try prompt engineering first (see prompt-design skill)
├── Tried prompt engineering, still inconsistent → Consider fine-tuning
└── Need domain knowledge not in base model → Fine-tuning with domain data