AI Cost Finops

Overview

LLM cost is now a measurable engineering surface, not a line on the credit-card statement. Three levers move it most in 2026: prompt caching (Anthropic 90% / OpenAI 50% discount on cached tokens), Batch APIs (50% off across major providers), and model routing (Haiku vs Sonnet vs Opus at 5× cost spreads). This skill encodes the price book, the discount mechanics, and the optimization patterns with current numbers and primary-source citations.

Use this skill when projecting LLM cost for a new feature, hunting waste in an existing pipeline, picking a model, setting up per-feature attribution, or building a FinOps dashboard. Skip it for one-off prototypes where total spend stays under $100/month — over-optimizing tiny budgets is the engineering equivalent of premature optimization.

Mental model

Every LLM call has a cost equation: input_tokens × $/M_in + output_tokens × $/M_out plus retrieval/rerank/storage if RAG. Three discounts apply: prompt caching on the input side, Batch APIs on whole calls, model routing on which model gets the call. Every optimization is a move along one of those axes. The cheapest call is the one not made; the second cheapest is the one cached; the third is the one batched; the fourth is the one routed to the smaller model.

Output tokens are 3–8× input tokens in cost across providers — the dominant lever after caching is output-length discipline (no markdown when consumer is machine, no preambles, no "let me think through this" verbosity on tasks that don't need reasoning).

Instructions

1. Provider pricing (May 2026, per 1M tokens, input/output)

Sources: Anthropic pricing, OpenAI pricing, Gemini pricing. Verify against live pages — prices update. Output tokens cost ~3–5× input on most models; o1 sits at 4×; some frontier models reach 8× (verify GPT-5.2 Pro reportedly $21/$168 — claim seen in secondary sources, verify on primary).

2. Prompt caching

Anthropic prompt caching (docs):

• Cache writes cost 1.25× input (5-min TTL) or 2.0× input (1-hour TTL).
• Cache reads cost 0.10× input — 90% discount on cached tokens.
• Up to 85% latency reduction on cached prefixes (per Anthropic's docs).
• Mechanism: "cache_control": {"type": "ephemeral", "ttl": "1h"} on message blocks. Default TTL was silently changed from 1h to 5min in March 2026 — set explicitly.
• Stacks with Batch API (batch 50% off + cache 90% off ≈ 95% off the input portion).

OpenAI automatic prompt caching (announcement, guide):

• Automatic 50% discount on cached prefixes — no code change.
• Kicks in at ≥1024 tokens.
• Cache evicted after 5–10 min idle, 1h max.

What to cache: system prompts, long tool/skill definitions, retrieval pre-context, few-shot examples — anything long, stable, and re-sent. The break-even on Anthropic 5-min cache writes is ~2 cache hits (you pay 1.25× write, save 0.90× × N reads); for 1-hour cache the break-even is ~3 hits.

3. Batch APIs (50% off everything that doesn't need real-time)

• Anthropic Message Batches — 50% discount, up to 10,000 requests per batch, 24-hour SLA, stacks with prompt caching. docs.
• OpenAI Batch API — 50% discount on all models including embeddings; batched-on-cached gets additional 50% (so 25% of original price). docs.
• Gemini Batch — 50% off, inline or JSONL input, context caching applies. docs.

Use cases: evals, bulk processing, async pipelines, embeddings backfill, dataset enrichment. If real-time isn't required, batch it — there's no reason not to take the discount.

4. Model routing

Practitioner pattern (cheaper and more predictable than commercial routers): default to the small model (Haiku, Gemini Flash, GPT-4o-mini) for typical traffic; escalate to mid-tier (Sonnet, Gemini Pro, GPT-4o) when classification confidence is low or output validation fails; reserve frontier (Opus, o1) for hard-reasoning paths only. GPT-4o-mini is 16× cheaper than GPT-4o; the typical Haiku-vs-Opus spread is 5× on input and 5× on output.

Source: RouterArena benchmark, arXiv:2510.00202 — commercial routers often gain accuracy at higher cost; manual rules with explicit fallback are competitive on cost-effectiveness.

5. Token economics

Source: codeant.ai — input vs output token cost.

Practical implications:

• Strip markdown when the consumer is a machine — every `` and # is wasted tokens.
• Use structured outputs (JSON Schema / function calling) — kills preambles and apologies; constrains the output surface.
• Ban "let me think through this..." preambles via system prompt for tasks that don't benefit from reasoning.
• Use Chain-of-Draft for arithmetic / logic over full CoT.

6. FinOps observability and attribution

Sources: Langfuse cost docs, Phoenix repo, Vantage FinOps for AI.

Common production pattern: Helicone as gateway for always-on cost/latency logging + Langfuse or Phoenix for trace-level analysis + Vantage for FinOps roll-up across AI providers and cloud infra in one pane.

Per-feature attribution: tag every call with feature, team, customer_tier, experiment_arm headers (Helicone) or trace attributes (Langfuse / Phoenix). Without attribution, you can't optimize what you can't measure.

7. Common waste patterns

8. RAG cost stack

Embeddings (per 1M tokens):

• text-embedding-3-small: $0.02
• voyage-3 lite: $0.02; voyage-3: $0.06; voyage-3-large: $0.12
• Cohere Embed v3: $0.10; v4: $0.12 (text)

Reranking (Cohere pricing):

• Cohere Rerank 3.5: $2 per 1K searches (1 query + up to 100 docs)
• Cohere Rerank v3: was $2/M tokens

Vector DB cost at 10M vectors (approximate, May 2026):

• Pinecone Serverless: ~$70/mo ($8.25/M reads, $2/M writes, $0.33/GB)
• Weaviate Flex: ~$45/mo + $0.095/M dims
• Qdrant Cloud: ~$65/mo (1GB free)
• Turbopuffer: min $64/mo, ~$9/M reads+writes, S3-backed storage $0.02/GB

At 100M vectors: Pinecone $700+, self-hosted Qdrant/Weaviate often <$100. Source: particula.tech vector DB comparison. Numbers verified at time of writing; vector DB pricing changes — verify on the provider's pricing page for production budgeting.

9. Optimization patterns checklist

For an existing system, work top-down — biggest wins first:

• Caching — system prompts + long tool/skill defs + few-shot examples marked with cache_control or wrapped in the OpenAI auto-cache range (≥1024 tokens)
• Batching — any non-real-time path moved to Batch API
• Model routing — small model default, escalate on signal; manual rules with eval gates per tier
• Output format — structured outputs; no markdown when consumer is machine; ban CoT preambles via system prompt
• RAG over-fetch — measure context-token waste; tighten top-k; rerank; check for duplicate chunks
• Per-feature attribution — every call tagged with feature, team, customer-tier
• Streaming + abort — wired for any user-facing path
• Embedding choice — voyage-3-lite or text-embedding-3-small for typical retrieval; only upgrade when RAG eval shows lift
• Reranker is on top-50 first-stage, not top-500 — reranker cost scales with candidate count

10. Production case studies (with caveats)

• Klarna AI assistant — 2.3M chats in 30 days (≈700 FTE), 67% automation, reported $40M annual savings; 80% lower resolution time. LangChain customer story. Caveat: Klarna publicly walked back parts of the "AI-first" claim in 2025 and rehired human support; the cost story is real, the quality story was overstated.
• Notion — 100M+ users; reportedly 90% of AI dev time spent on evals/observability; tiered eval stack (cheap unit tests run frequently, expensive regressions gated). Verify source attribution before citing externally.
• Intercom Fin — agent resolution rates moved from ~25% to ~60% via GPT-4 + custom RAG + reranking. Verify exact figures via Intercom's own materials before publishing.

Case studies are useful for direction; production budgeting requires your own attribution data, not vendor blog numbers.

Composition / References

Within-domain pairings:

• Pairs with the sibling AI skill on RAG patterns (RAG cost stack — embeddings, rerank, vector DB).
• Pairs with the sibling AI skill on agent design (multi-agent and tool-call cost trade-offs).
• Pairs with the sibling AI skill on evals (eval workloads belong on Batch APIs).

Primary sources:

• Anthropic pricing
• Anthropic prompt caching
• Anthropic Message Batches
• OpenAI pricing
• OpenAI prompt caching announcement
• OpenAI Batch API guide
• Gemini pricing
• Gemini Batch API
• Cohere pricing
• Langfuse token and cost tracking
• Helicone
• Phoenix (Arize)
• Vantage — FinOps for AI
• RouterArena benchmark, arXiv:2510.00202
• codeant.ai — input vs output token cost