LLM Cost Optimization
Optimize cost per completed task, not cost per request. A cheaper request that needs more turns,
retries, or a human fix is not cheaper. Work the free levers first — they cut cost without touching
quality — and treat every lever that trades capability for cost as an experiment that must pass an eval.
When to Use
- An LLM feature's API spend is too high, growing faster than traffic, or unknown.
- Before a launch or a traffic increase, to forecast and cap spend.
- Someone proposes "just use a smaller model" or a router, and you need to know if it pays.
- Agent loops whose cost grows with every turn.
Not for a coding agent's own usage limits (Claude Code, Cursor) — use coding-agent-token-efficiency.
Workflow
Cost pass:
- [ ] 0. Quality bar and baseline agreed
- [ ] 1. Token profile built
- [ ] 2. Free wins applied and verified
- [ ] 3. Tradeoffs tested on the eval (only with approval)
- [ ] 4. Report written
0. Agree the quality bar and baseline
Ask the user, do not assume:
- What counts as a completed task, and how is it checked (tests, a validator, a rubric, human review)?
- Which routes or features are in scope, and which are latency-sensitive?
- Is there an eval set? If not, offer to build a small one: 20–30 real requests, frozen, each with a
pass/fail check. Without one, only the free levers can be applied.
Every run that calls the model spends money. Get approval for a measurement budget before running
baselines or sweeps.
1. Build the token profile
Log usage from every response (it is free to add and is the measurement channel for everything else),
tagged with a task id and the outcome. Then:
python3 scripts/cost_per_task.py usage.jsonl --input-price <USD/MTok> --output-price <USD/MTok> \
--cache-read-mult <x> --cache-write-mult <x> --batch-discount <fraction>
The script (standard library only, path relative to this skill) accepts Anthropic or OpenAI usage
fields and reports cost per request, per task, and per successful task; cache hit rate; spend by token
type; and two ceilings: all traffic batched, and every prompt token served from cache. Take prices from
the provider's pricing page today — never from memory.
Read the profile: which token type dominates, how deep the loops run, how much of the prompt repeats,
and which routes have no user waiting.
2. Apply the free wins, in this order
2.1 Prompt caching — first, and it stays on. Every turn re-sends the whole prefix; caching reprices
the repeated part. Caching is a prefix match, so any changed byte invalidates everything after it.
- Order the prompt by stability: tools → system prompt → static context → conversation → the volatile
tail (timestamps, request ids, the question). Never put a timestamp or a random id in the system prompt.
- Keep tool order and JSON key order deterministic; do not toggle tools, model, or reasoning settings
mid-conversation — each one restarts the cache.
- Verify from usage, not code review:
cache_read_input_tokens (Anthropic) or cached_tokens (OpenAI)
should dominate on a warm loop. Zero hits on repeated identical prefixes means a silent invalidator.
Anthropic specifics (confirm on the prompt-caching page before computing): explicit cache_control
breakpoints (at most 4 per request) or top-level automatic caching; a 5-minute TTL by default or 1 hour
on request; cache writes cost 1.25× the input price for 5 minutes and 2× for 1 hour, reads about 0.1×
(lower on some current models); prefixes below a model-dependent minimum (512 to 4,096 tokens) silently
do not cache. With the 5-minute TTL two requests already break even; the 1-hour TTL needs three, so use
it only when gaps between requests regularly exceed five minutes. Other providers cache automatically
or explicitly with different minimums and discounts — read their caching page.
2.2 Input-token hygiene.
- Move large reference documents behind a tool or retrieval step if most requests use only a section;
keep them inline (and cached) if most requests need most of it.
- Delete prose that restates tool schemas. Load rarely used tool definitions on demand (tool search /
deferred loading) once schemas pass roughly 10K tokens.
- Downscale images to what the task needs: vision cost scales with pixel area, not information.
- Give list tools
limit, fields, and date filters; prefer narrow accessors to data dumps.
- Count tokens on unbounded user input before sending it, and route oversize payloads elsewhere.
- Prompt compression (LLMLingua and successors) helps on long, redundant retrieved context. Test it on
the eval: compression that drops a constraint costs more than it saves.
2.3 Agent-loop hygiene (only when loops are long and tool results pile up).
- Push self-contained, bulky steps into a sub-call that returns one summary line.
- Use server-side compaction or context clearing only to make room in the window. Each pass rewrites the
cached conversation, so trigger it rarely and in large batches.
2.4 Output control.
- Specify the exact output shape with an example; ask for diffs, not whole files.
max_tokens is a backstop, not a tuning knob: the model never sees it, and a truncated answer is a
failed attempt you pay for twice. Set it generously and treat stop_reason: max_tokens as a failure.
- Add a stop sentinel for "cannot proceed" cases so the model stops instead of explaining.
2.5 Batch everything no one is waiting for. Evaluations, backfills, nightly jobs. Anthropic's
Message Batches API is 50% off every token, including cache reads and writes; results arrive within 24
hours. Batch requests are single-shot, so tool loops do not batch without restructuring.
3. Test the tradeoffs (one at a time, on the eval)
From here every lever can lower quality. Present each with its measured effect and apply it only when
the user accepts the trade.
- Effort / reasoning level first. Sweep it on the current model before changing models. Research
and Q&A curves are often flat (a lower level holds accuracy for much less); long-horizon coding loses
a few points per step down. Change it between conversations, not mid-conversation.
- Re-run failures at higher effort. When a cheap checker exists, run everything at a low setting and
retry only the failures higher.
- Task budgets / token ceilings where the provider supports a model-visible budget; set from the
90th-percentile task, then tighten while watching pass rate.
- Model choice, last. First check whether a newer model in the same tier at lower effort beats the
old one. Then try one tier down, re-sweeping effort. Compare on the hardest tenth of tasks — the tail
decides the bill.
- Routing and cascades (FrugalGPT, RouteLLM): send easy requests to a cheap model and escalate on
low confidence or a failed check. The papers report large savings — FrugalGPT matched its best single
model's accuracy at up to 98% lower cost on its benchmarks; RouteLLM cut cost more than 2× without
quality loss in some settings — but caches are per model, so a cascade forfeits cache reuse across
tiers, and a bad router silently degrades quality. Build one only after the simpler levers, and only
with an eval and a fallback.
4. Report
# LLM cost review — <app / route>
Baseline: <cost per completed task>, <pass rate>, <requests/day>, measured <date> from <source>
Prices used: <model, $/MTok in/out, cache multipliers, source URL, date>
| Lever | Type | Expected saving | Measured | Quality | Decision |
| --- | --- | ---: | ---: | --- | --- |
| Prompt caching (static prefix) | free | | | unchanged | applied |
| Batch nightly summarization | free | | | unchanged | applied |
| Effort high → medium on /search | tradeoff | | | -1 task of 30 | proposed |
Skipped levers and why: <list>
Next: <approval or eval needed>
"No changes recommended" is a valid outcome when the profile shows no lever with headroom.
Gotchas
- Validate caching with a warm cache. The first request writes at a premium; measure from the
second request on, or caching looks like a regression.
- Caches are scoped (per model, and per workspace or organization depending on provider). Traffic
split across keys or workspaces caches separately.
- Concurrency defeats caching: parallel requests sent before the first finishes all miss.
- Prices change. Never ship a cost model with remembered prices; cite the pricing page and the date.
- Do not cap output to save money without checking pass rate; truncated answers inflate cost per
completed task.
Related skills
coding-agent-token-efficiency, context-compression, agentic-eval, eval-driven-dev,
amazon-bedrock (Bedrock pricing, caching, and batch specifics).
Sources
- Anthropic, Optimizing for cost and intelligence,
Prompt caching,
Batch processing,
Pricing.
- Chen, Zaharia & Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving
Performance, arXiv:2305.05176.
- Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, arXiv:2406.18665.
- Jiang et al., LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models,
arXiv:2310.05736.
1---2name: llm-cost-optimization3description: >-4 Use when an application that calls an LLM API costs too much, needs a cost forecast, or is about to5 ship at scale — "the bill is too high", "make this cheaper", "should we route to a smaller model".6 Profiles spend per completed task from usage logs, then applies levers in order: prompt caching,7 input-token hygiene, agent-loop hygiene, output control, and batch APIs before effort, budget, and8 model-routing tradeoffs validated on an eval. Provider-agnostic, with Anthropic specifics.9version: "1.0.0"10license: MIT11source: "https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence; https://arxiv.org/abs/2305.05176; https://arxiv.org/abs/2406.18665; https://arxiv.org/abs/2310.05736"12attribution: "Written for this collection from Anthropic's cost-optimization and prompt-caching guides and the FrugalGPT, RouteLLM, and LLMLingua papers"13metadata:14 source: https://skillmd.com/skills/mouadja02/llm-cost-optimization15---1617# LLM Cost Optimization1819Optimize **cost per completed task**, not cost per request. A cheaper request that needs more turns,20retries, or a human fix is not cheaper. Work the free levers first — they cut cost without touching21quality — and treat every lever that trades capability for cost as an experiment that must pass an eval.2223## When to Use2425- An LLM feature's API spend is too high, growing faster than traffic, or unknown.26- Before a launch or a traffic increase, to forecast and cap spend.27- Someone proposes "just use a smaller model" or a router, and you need to know if it pays.28- Agent loops whose cost grows with every turn.2930Not for a coding agent's own usage limits (Claude Code, Cursor) — use `coding-agent-token-efficiency`.3132## Workflow3334```35Cost pass:36- [ ] 0. Quality bar and baseline agreed37- [ ] 1. Token profile built38- [ ] 2. Free wins applied and verified39- [ ] 3. Tradeoffs tested on the eval (only with approval)40- [ ] 4. Report written41```4243### 0. Agree the quality bar and baseline4445Ask the user, do not assume:4647- What counts as a completed task, and how is it checked (tests, a validator, a rubric, human review)?48- Which routes or features are in scope, and which are latency-sensitive?49- Is there an eval set? If not, offer to build a small one: 20–30 real requests, frozen, each with a50 pass/fail check. Without one, only the free levers can be applied.5152Every run that calls the model spends money. Get approval for a measurement budget before running53baselines or sweeps.5455### 1. Build the token profile5657Log `usage` from every response (it is free to add and is the measurement channel for everything else),58tagged with a task id and the outcome. Then:5960```bash61python3 scripts/cost_per_task.py usage.jsonl --input-price <USD/MTok> --output-price <USD/MTok> \62 --cache-read-mult <x> --cache-write-mult <x> --batch-discount <fraction>63```6465The script (standard library only, path relative to this skill) accepts Anthropic or OpenAI usage66fields and reports cost per request, per task, and per successful task; cache hit rate; spend by token67type; and two ceilings: all traffic batched, and every prompt token served from cache. Take prices from68the provider's pricing page today — never from memory.6970Read the profile: which token type dominates, how deep the loops run, how much of the prompt repeats,71and which routes have no user waiting.7273### 2. Apply the free wins, in this order7475**2.1 Prompt caching — first, and it stays on.** Every turn re-sends the whole prefix; caching reprices76the repeated part. Caching is a prefix match, so any changed byte invalidates everything after it.7778- Order the prompt by stability: tools → system prompt → static context → conversation → the volatile79 tail (timestamps, request ids, the question). Never put a timestamp or a random id in the system prompt.80- Keep tool order and JSON key order deterministic; do not toggle tools, model, or reasoning settings81 mid-conversation — each one restarts the cache.82- Verify from usage, not code review: `cache_read_input_tokens` (Anthropic) or `cached_tokens` (OpenAI)83 should dominate on a warm loop. Zero hits on repeated identical prefixes means a silent invalidator.8485Anthropic specifics (confirm on the prompt-caching page before computing): explicit `cache_control`86breakpoints (at most 4 per request) or top-level automatic caching; a 5-minute TTL by default or 1 hour87on request; cache writes cost 1.25× the input price for 5 minutes and 2× for 1 hour, reads about 0.1×88(lower on some current models); prefixes below a model-dependent minimum (512 to 4,096 tokens) silently89do not cache. With the 5-minute TTL two requests already break even; the 1-hour TTL needs three, so use90it only when gaps between requests regularly exceed five minutes. Other providers cache automatically91or explicitly with different minimums and discounts — read their caching page.9293**2.2 Input-token hygiene.**9495- Move large reference documents behind a tool or retrieval step if most requests use only a section;96 keep them inline (and cached) if most requests need most of it.97- Delete prose that restates tool schemas. Load rarely used tool definitions on demand (tool search /98 deferred loading) once schemas pass roughly 10K tokens.99- Downscale images to what the task needs: vision cost scales with pixel area, not information.100- Give list tools `limit`, `fields`, and date filters; prefer narrow accessors to data dumps.101- Count tokens on unbounded user input before sending it, and route oversize payloads elsewhere.102- Prompt compression (LLMLingua and successors) helps on long, redundant retrieved context. Test it on103 the eval: compression that drops a constraint costs more than it saves.104105**2.3 Agent-loop hygiene** (only when loops are long and tool results pile up).106107- Push self-contained, bulky steps into a sub-call that returns one summary line.108- Use server-side compaction or context clearing only to make room in the window. Each pass rewrites the109 cached conversation, so trigger it rarely and in large batches.110111**2.4 Output control.**112113- Specify the exact output shape with an example; ask for diffs, not whole files.114- `max_tokens` is a backstop, not a tuning knob: the model never sees it, and a truncated answer is a115 failed attempt you pay for twice. Set it generously and treat `stop_reason: max_tokens` as a failure.116- Add a stop sentinel for "cannot proceed" cases so the model stops instead of explaining.117118**2.5 Batch everything no one is waiting for.** Evaluations, backfills, nightly jobs. Anthropic's119Message Batches API is 50% off every token, including cache reads and writes; results arrive within 24120hours. Batch requests are single-shot, so tool loops do not batch without restructuring.121122### 3. Test the tradeoffs (one at a time, on the eval)123124From here every lever can lower quality. Present each with its measured effect and apply it only when125the user accepts the trade.1261271. **Effort / reasoning level first.** Sweep it on the current model before changing models. Research128 and Q&A curves are often flat (a lower level holds accuracy for much less); long-horizon coding loses129 a few points per step down. Change it between conversations, not mid-conversation.1302. **Re-run failures at higher effort.** When a cheap checker exists, run everything at a low setting and131 retry only the failures higher.1323. **Task budgets / token ceilings** where the provider supports a model-visible budget; set from the133 90th-percentile task, then tighten while watching pass rate.1344. **Model choice, last.** First check whether a newer model in the same tier at lower effort beats the135 old one. Then try one tier down, re-sweeping effort. Compare on the hardest tenth of tasks — the tail136 decides the bill.1375. **Routing and cascades** (FrugalGPT, RouteLLM): send easy requests to a cheap model and escalate on138 low confidence or a failed check. The papers report large savings — FrugalGPT matched its best single139 model's accuracy at up to 98% lower cost on its benchmarks; RouteLLM cut cost more than 2× without140 quality loss in some settings — but caches are per model, so a cascade forfeits cache reuse across141 tiers, and a bad router silently degrades quality. Build one only after the simpler levers, and only142 with an eval and a fallback.143144### 4. Report145146```markdown147# LLM cost review — <app / route>148149Baseline: <cost per completed task>, <pass rate>, <requests/day>, measured <date> from <source>150Prices used: <model, $/MTok in/out, cache multipliers, source URL, date>151152| Lever | Type | Expected saving | Measured | Quality | Decision |153| --- | --- | ---: | ---: | --- | --- |154| Prompt caching (static prefix) | free | | | unchanged | applied |155| Batch nightly summarization | free | | | unchanged | applied |156| Effort high → medium on /search | tradeoff | | | -1 task of 30 | proposed |157158Skipped levers and why: <list>159Next: <approval or eval needed>160```161162"No changes recommended" is a valid outcome when the profile shows no lever with headroom.163164## Gotchas165166- **Validate caching with a warm cache.** The first request writes at a premium; measure from the167 second request on, or caching looks like a regression.168- **Caches are scoped** (per model, and per workspace or organization depending on provider). Traffic169 split across keys or workspaces caches separately.170- **Concurrency defeats caching:** parallel requests sent before the first finishes all miss.171- **Prices change.** Never ship a cost model with remembered prices; cite the pricing page and the date.172- **Do not cap output to save money** without checking pass rate; truncated answers inflate cost per173 completed task.174175## Related skills176177`coding-agent-token-efficiency`, `context-compression`, `agentic-eval`, `eval-driven-dev`,178`amazon-bedrock` (Bedrock pricing, caching, and batch specifics).179180## Sources181182- Anthropic, [Optimizing for cost and intelligence](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence),183 [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching),184 [Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing),185 [Pricing](https://platform.claude.com/docs/en/about-claude/pricing).186- Chen, Zaharia & Zou, *FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving187 Performance*, [arXiv:2305.05176](https://arxiv.org/abs/2305.05176).188- Ong et al., *RouteLLM: Learning to Route LLMs with Preference Data*, [arXiv:2406.18665](https://arxiv.org/abs/2406.18665).189- Jiang et al., *LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models*,190 [arXiv:2310.05736](https://arxiv.org/abs/2310.05736).