Amey-Thakur LLM cost latency
LLM cost is tokens times model price; latency is time-to-first-token plus tokens times generation speed. Every optimization attacks one of those four factors: send fewer tokens, generate fewer tokens, use a cheaper model, or start visibly sooner.
Method
- Measure per feature before optimizing. Token counts (in/out separately: output usually costs several times input), cost and p50/p95 latency per feature and per request class (see llm-observability): the bill concentrates: one verbose feature is typically 60% of spend, and that is where the week of work goes (see cloud-cost-optimization's attribution ethic).
- Exploit prompt caching deliberately. Stable prefixes (system prompt, tool definitions, few-shot examples) cached at the provider cut cost and time-to-first-token dramatically on the majors: structure prompts stable-first, variable-last (see context-window-management's partition: caching rewards the same layout), and verify cache hit rates in telemetry rather than assuming.
- Tier models by task difficulty. Route each feature to the cheapest model that passes its eval bar (see llm-eval-design: the eval suite is what makes tiering safe): classification, extraction, and routing on small/ fast models; generation and reasoning on the big ones; with an escalation path (small model's low-confidence or failed-validation cases retry on the larger: see ml-baselines' ladder logic). Re-run the tier decision at every provider price/model refresh: the frontier moves quarterly.
- Put every prompt on a diet. Trim boilerplate and redundant instructions (each competes for attention anyway), cut few-shot examples to the minimum the evals require, summarize instead of pasting raw data (see context-window-management step 4), and cap output: ask for terse formats (IDs, not restated context), set max tokens, and instruct against preamble ("answer only"): output tokens are the expensive, slow ones.
- Batch the batchable. Non-interactive work (labeling, enrichment, evals, digests) goes through batch APIs at half price with relaxed latency (see background-jobs' queue discipline); interactive traffic never waits behind it: separate the pools (see message-queues' latency-class separation).
- Engineer perceived latency. Stream tokens to the UI (time-to-first-token becomes the felt latency), show progressive structure (see empty-and-error-states), parallelize independent calls, and pre-warm/speculate where flows are predictable (draft the likely next step while the user reads). Semantic caching (same question, cached answer) helps FAQ-shaped traffic; guard it with freshness and personalization keys (see caching-strategy, http-caching instincts).
Boundaries
- Cost cuts that fail the eval gate are quality regressions wearing a smaller bill; every tiering and diet change runs the suite first (see llm-eval-design).
- Latency floors are physical: model size and output length dominate; when the product needs instant, the design answer may be a smaller task (classify then fetch) rather than a faster generation.
- Self-hosting for cost (see inference-serving-optimization, vllm-serving) trades bill for operational burden; run the managed-vs-selfhosted math with real utilization, not peak-hour dreams.