LLM Cost Optimization

LLM Cost Optimization

Systematic framework for auditing, routing, caching, and governing LLM API spend.

Core principle: Measure first. Most teams can cut LLM costs 40-75% without quality regression by routing tasks to appropriate models and caching repeated computations.

When to Use

• LLM API bills are growing faster than usage
• Launching a new AI feature and need to estimate/control costs
• Spending more than $1K/month on LLM APIs
• Users are hitting rate limits due to inefficient token usage
• Need to implement per-user or per-feature cost controls

When NOT to Use

• Initial prototype with <$100/month spend (premature optimization)
• Quality is failing (fix quality first, then optimize cost)
• Prompt engineering improvements are still untapped

Phase 1: Cost Audit

Before optimizing, understand where money goes.

Instrument Every API Call

async function callLLM(params: LLMParams): Promise<LLMResponse> {
  const response = await llm.complete(params);
  
  // Log cost metadata
  logger.info({
    model: params.model,
    feature: params.metadata.feature,       // "search", "chat", "summarize"
    userId: params.metadata.userId,
    inputTokens: response.usage.input_tokens,
    outputTokens: response.usage.output_tokens,
    estimatedCost: calculateCost(response.usage, params.model),
  });
  
  return response;
}

Cost Breakdown Query

SELECT
  feature,
  model,
  SUM(input_tokens) AS total_input,
  SUM(output_tokens) AS total_output,
  SUM(estimated_cost) AS total_cost,
  COUNT(*) AS request_count,
  AVG(estimated_cost) AS cost_per_request
FROM llm_calls
WHERE created_at >= NOW() - INTERVAL '30 days'
GROUP BY feature, model
ORDER BY total_cost DESC;

Find: Top 20% of features consuming 80% of spend. Optimize those first.

Phase 2: Model Routing

Model Tier Reference (Anthropic, April 2026)

Routing Rules

function selectModel(task: string, qualityRequired: 'high' | 'standard' | 'low'): string {
  // Complex reasoning always uses best model
  if (['research', 'strategy', 'novel_problem'].includes(task)) {
    return 'claude-opus-4-6';
  }
  
  // Standard quality: balanced model
  if (qualityRequired === 'high' || ['code_review', 'analysis', 'writing'].includes(task)) {
    return 'claude-sonnet-4-6';
  }
  
  // Simple tasks: fast/cheap model
  // classification, extraction, summarization, tagging
  return 'claude-haiku-4-5';
}

Fallback Chain (Quality-Gated)

Request → Haiku → Quality check passes? → Return
                → Quality check fails? → Sonnet → Quality check passes? → Return
                                                 → Quality check fails? → Opus → Return

Track fallback rate per task type. If Haiku fails >10%, update routing rules.

Phase 3: Token Optimization

Input Token Reduction

Output Token Reduction

Sliding Context Window

For chat/conversation:

Full context (all history) → most accurate, highest cost
Sliding window (last N turns) → good accuracy, moderate cost  
Compressed history (summary + recent) → acceptable, low cost

Choose based on task criticality and budget.

Phase 4: Caching

Prompt Prefix Caching

Cache stable portions of the prompt (system prompt, knowledge base) to avoid re-tokenizing:

Anthropic: cached input tokens cost ~10% of normal input tokens.

Semantic Caching

Cache responses for semantically similar queries (not just exact matches):

def get_with_semantic_cache(query: str, threshold: float = 0.95) -> str | None:
    query_embedding = embed(query)
    
    # Search cache for similar queries
    cached = vector_cache.query(query_embedding, k=1)
    if cached and cached[0].similarity > threshold:
        return cached[0].response
    
    # Cache miss — call LLM
    response = call_llm(query)
    vector_cache.insert(query_embedding, response)
    return response

Cache TTL by query type:

• Factual ("What is X?") → 7 days
• Procedural ("How do I Y?") → 30 days
• Creative ("Generate Z") → 1 day or never

Phase 5: Batching

Route non-urgent work to batch APIs (typically 50% discount):

# Submit batch job (asynchronous, up to 24hr turnaround)
batch = anthropic.messages.batches.create(requests=[
    {"custom_id": f"req-{i}", "params": {"model": "...", "messages": [...]}}
    for i in range(1000)
])

# Poll for completion
while batch.processing_status != "ended":
    time.sleep(60)
    batch = anthropic.messages.batches.retrieve(batch.id)

Use batching for:

• Content generation, classification, or analysis jobs
• Background processing (nightly reports, batch exports)
• Any work with latency tolerance >5 minutes

Do NOT batch:

• User-facing chat (needs <1s response)
• Real-time decisions (fraud, moderation)

Phase 6: Cost Governance

Budget Controls

const budgets = {
  per_feature: {
    search: 5000,    // $5K/month
    chat: 8000,      // $8K/month
    reports: 2000,   // $2K/month
  },
  per_user_tier: {
    free:       0.10,   // $0.10/user/month
    pro:        3.00,   // $3/user/month
    enterprise: 25.00,  // $25/user/month
  },
};

// Alert when feature reaches 80% of budget
if (featureCostMtd > budgets.per_feature[feature] * 0.80) {
  alertSlack(`#ai-costs: ${feature} at 80% of monthly budget`);
}

Cost vs Quality Decision Framework

Common Anti-Patterns

Verification Checklist

Visibility:

• All LLM calls instrumented with feature, model, token counts, cost
• Monthly spend queryable by feature and model
• Top 3 cost drivers identified

Routing:

• Routing rules defined by task type and quality requirement
• Fallback chain implemented and fallback rate <10% per task

Optimization:

• Input token reduction applied to top cost drivers (>20% savings)
• System prompt caching enabled
• Output length constraints set per task type
• Batch processing for non-urgent work

Governance:

• Feature budgets defined and alerting at 80%
• Per-user-tier cost limits enforced
• Monthly cost review scheduled
• Quality metrics monitored alongside cost metrics