Model Selection

Model Selection Framework

When to Use This Skill

Use this skill when:

• Model Selection tasks - Working on select appropriate ai/ml models based on capability matching, benchmarks, cost-performance tradeoffs, and deployment constraints
• Planning or design - Need guidance on Model Selection approaches
• Best practices - Want to follow established patterns and standards

Overview

Model selection is the systematic process of choosing the right AI/ML model based on task requirements, performance characteristics, cost constraints, and deployment considerations. Poor model selection leads to suboptimal performance, excessive costs, or deployment failures.

Model Selection Decision Tree

┌─────────────────────────────────────────────────────────────────┐
│                   MODEL SELECTION FRAMEWORK                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. TASK ANALYSIS                                               │
│     What are the core capabilities needed?                      │
│     ├── Text Generation → LLM                                   │
│     ├── Classification → Traditional ML / Small LM              │
│     ├── Code Generation → Code-specialized LLM                  │
│     ├── Vision → Multimodal / Vision Model                      │
│     ├── Embeddings → Embedding Model                            │
│     └── Structured Output → Instruction-tuned LLM               │
│                                                                  │
│  2. REQUIREMENTS MAPPING                                         │
│     ├── Quality: Accuracy, coherence, factuality                │
│     ├── Latency: Real-time vs batch                             │
│     ├── Cost: Per-token, per-request budgets                    │
│     ├── Privacy: Data residency, local deployment               │
│     └── Scale: Requests per second, concurrent users            │
│                                                                  │
│  3. MODEL EVALUATION                                             │
│     ├── Benchmark analysis                                       │
│     ├── Task-specific testing                                   │
│     └── Cost-performance optimization                           │
│                                                                  │
│  4. DEPLOYMENT PLANNING                                          │
│     ├── Cloud API vs self-hosted                                │
│     ├── Hardware requirements                                   │
│     └── Scaling strategy                                        │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

LLM Capability Matrix

General Purpose Models (December 2025)

Specialized Models

Local/Open Models

Model Comparison Framework

Benchmark Interpretation

Task-Specific Evaluation

public class ModelEvaluator
{
    public async Task<EvaluationReport> EvaluateModels(
        List<ModelConfig> candidates,
        EvaluationDataset dataset,
        CancellationToken ct)
    {
        var results = new Dictionary<string, ModelMetrics>();

        foreach (var model in candidates)
        {
            var metrics = new ModelMetrics
            {
                ModelId = model.Id,
                Provider = model.Provider
            };

            // Run task-specific tests
            foreach (var testCase in dataset.TestCases)
            {
                var startTime = Stopwatch.StartNew();

                var response = await CallModel(model, testCase.Prompt, ct);

                startTime.Stop();

                metrics.AddResult(new TestResult
                {
                    TestId = testCase.Id,
                    LatencyMs = startTime.ElapsedMilliseconds,
                    InputTokens = CountTokens(testCase.Prompt),
                    OutputTokens = CountTokens(response),
                    Score = await EvaluateResponse(response, testCase.Expected),
                    Cost = CalculateCost(model, testCase.Prompt, response)
                });
            }

            results[model.Id] = metrics;
        }

        return new EvaluationReport
        {
            Results = results,
            Recommendation = SelectBestModel(results, dataset.Requirements)
        };
    }

    private ModelRecommendation SelectBestModel(
        Dictionary<string, ModelMetrics> results,
        Requirements requirements)
    {
        // Score based on requirements weights
        var scores = results.Select(r => new
        {
            Model = r.Key,
            Score = CalculateWeightedScore(r.Value, requirements)
        }).OrderByDescending(s => s.Score);

        return new ModelRecommendation
        {
            Primary = scores.First().Model,
            Fallback = scores.Skip(1).FirstOrDefault()?.Model,
            Reasoning = GenerateReasoning(scores, requirements)
        };
    }
}

Cost-Performance Analysis

Pricing Comparison (December 2025, per 1M tokens)

Cost Optimization Strategies

ROI Calculator

public class ModelCostCalculator
{
    public CostProjection CalculateMonthlyCost(
        ModelConfig model,
        UsageEstimate usage)
    {
        var inputCost = usage.MonthlyInputTokens / 1_000_000m
            * model.InputPricePerMillion;

        var outputCost = usage.MonthlyOutputTokens / 1_000_000m
            * model.OutputPricePerMillion;

        var cachedSavings = usage.CacheHitRate * inputCost
            * model.CacheDiscount;

        return new CostProjection
        {
            GrossInputCost = inputCost,
            GrossOutputCost = outputCost,
            CacheSavings = cachedSavings,
            NetMonthlyCost = inputCost + outputCost - cachedSavings,
            CostPerRequest = (inputCost + outputCost - cachedSavings)
                / usage.MonthlyRequests
        };
    }

    public ModelComparison CompareModels(
        List<ModelConfig> models,
        UsageEstimate usage,
        QualityRequirements requirements)
    {
        var comparisons = models.Select(m => new
        {
            Model = m,
            Cost = CalculateMonthlyCost(m, usage),
            MeetsRequirements = EvaluateQuality(m, requirements)
        }).Where(c => c.MeetsRequirements)
          .OrderBy(c => c.Cost.NetMonthlyCost)
          .ToList();

        return new ModelComparison
        {
            CheapestQualified = comparisons.FirstOrDefault()?.Model,
            AllOptions = comparisons,
            Savings = CalculateSavings(comparisons)
        };
    }
}

Fine-Tuning Decision Framework

When to Fine-Tune

Fine-Tuning ROI Analysis

## Fine-Tuning Decision: [Use Case]

### Current State (Prompting)
- Prompt tokens: [X] tokens
- Output quality: [Score]
- Cost per request: $[X]
- Monthly cost: $[X]

### Projected State (Fine-Tuned)
- Prompt tokens: [Y] tokens (reduced)
- Output quality: [Score] (maintained/improved)
- Fine-tuning cost: $[X] (one-time)
- Inference cost per request: $[Y]
- Monthly cost: $[Y]

### Break-Even Analysis
- Monthly savings: $[X]
- Break-even: [N] months
- 12-month ROI: [X]%

### Recommendation
[Fine-tune / Continue prompting / Hybrid approach]

Deployment Considerations

Cloud vs Self-Hosted Decision

Hardware Requirements (Self-Hosted)

Model Selection Template

# Model Selection: [Project Name]

## Task Requirements
- **Primary Use Case**: [Description]
- **Quality Requirements**: [Accuracy/coherence targets]
- **Latency Requirements**: [P95 target]
- **Volume**: [Requests/day]
- **Budget**: [Monthly budget]

## Evaluation Results

| Model | Quality Score | P95 Latency | Monthly Cost | Notes |
|-------|--------------|-------------|--------------|-------|
| [Model A] | [Score] | [ms] | $[X] | [Notes] |
| [Model B] | [Score] | [ms] | $[X] | [Notes] |
| [Model C] | [Score] | [ms] | $[X] | [Notes] |

## Recommendation

**Primary Model**: [Model]
- Rationale: [Why this model]

**Fallback Model**: [Model]
- Use when: [Conditions]

**Cost Projection**: $[X]/month

## Implementation Notes
- [Deployment approach]
- [Monitoring strategy]
- [Scaling considerations]

Validation Checklist

• Task requirements clearly defined
• Capability mapping completed
• Candidate models identified
• Benchmarks reviewed
• Task-specific evaluation conducted
• Cost analysis completed
• Latency requirements validated
• Deployment constraints considered
• Fine-tuning decision made
• Fallback strategy defined

Integration Points

Inputs from:

• Business requirements → Task definition
• ml-project-lifecycle skill → Project constraints

Outputs to:

• token-budgeting skill → Cost estimation
• rag-architecture skill → Embedding model selection
• Application code → Model configuration

Converted and distributed by TomeVault — claim your Tome and manage your conversions.