Reduce LLM Token Costs

Reduce LLM Token Costs (~90%)

Core insight: "LLM systems work best when the model sees the right information, not the most information." The prompt you write is usually a small fraction of the tokens — the cost is the system around it: repeated system prompts, verbose tool definitions, conversation history, RAG chunks, and raw telemetry re-sent on every request. Multi-step agents make this worse: a 10-step task can process 50k–100k tokens because each step re-sends context.

Rule of thumb: Never send raw data to the model if it can be compressed first.

When to use this

• One request's "plumbing" (system + tools + history + retrieved docs + logs) dwarfs the actual question.
• An agent loops N reasoning steps and each step re-sends the whole context.
• RAG stuffs 5+ chunks × ~800 tokens straight into the prompt.
• You paste raw logs/telemetry/JSON dumps into prompts.
• Token bill scales with conversation length or step count, not with task difficulty.

The worked example (from the source article)

A telemetry-analysis agent, per request:

The eight techniques

1. Compress tool definitions

Verbose natural-language tool docs cost hundreds of tokens each. Replace prose with a terse signature list — the model keeps performing.

# Before (~verbose)
Tool: analyze_cpu_usage
This tool analyzes CPU usage in Windows telemetry logs and should be used when the user wants to understand CPU spikes...

# After
tools:
- cpu_analyze(json)
- mem_analyze(json)
- disk_analyze(json)
- generate_fix(issue)

2. Summarize logs & telemetry before sending

Add a preprocessing layer that extracts key signals from raw data. Don't paste the dump.

# After — signal only
top_cpu_processes:
- chrome.exe 32%
- node.exe 14%
top_memory_processes:
- chrome.exe 1.2GB

3. Compress RAG context

Insert a summarization step between retrieval and the final prompt. 5 chunks × 800 = 4,000 tokens → a compressed summary. Cuts RAG payload 80–90%.

retrieve → summarize → send compressed context

4. Replace conversation history with external state

Stop re-sending the whole transcript. Store workflow state (Redis/DB) and send only the relevant fields.

{ "issue": "high_cpu", "suspected_process": "chrome.exe", "previous_actions": ["restart_service"] }

5. Hierarchical context compression

Multi-stage funnel for large document sets:

documents → chunk summaries → merged summary → final reasoning
# 5×800 = 4,000  →  chunk summaries 600  →  merged 200
# final prompt: system 1,500 + context 200 = 1,700  (vs 5,500)

6. Smaller models for preprocessing

Two-model pipeline: a small/cheap model filters & summarizes raw data; the large model reasons over only the compressed signal.

raw data → small model (summarize/filter) → compressed context → large model (reasoning)

7. Prompt caching

For large, stable prefixes (system prompt + tool defs) sent every request.

• OpenAI: automatic caching.
• Claude: explicit cache_control; up to 4 cache breakpoints; TTL 5 min default (1-hour option); cached tokens billed at ~10% of normal input price; minimum cacheable prefix 1,024 tokens.

8. Conversation chaining (OpenAI Responses API)

Use previous_response_id to reference an earlier response instead of resending full history (e.g. request 1 analyzes → request 2 generates a fix referencing request 1). Note: earlier tokens can still count, so combine with state (#4).

Reference architecture

Raw Data / Logs
  ↓ Preprocessing layer (summarize, filter)     # small model, #2 #6
  ↓ State store (Redis / DB)                     # #4
  ↓ RAG retrieval
  ↓ Context compression                          # #3 #5
  ↓ LLM reasoning (OpenAI or Claude)             # #7 #8
  ↓ Structured output

Apply in priority order (highest ROI first)

1. Compress RAG documents (#3)
2. Store workflow state externally (#4)
3. Summarize logs/telemetry (#2)
4. Smaller models for preprocessing (#6)
5. Prompt caching (#7)

Then layer in compressed tool defs (#1), hierarchical compression (#5), and conversation chaining (#8).

How to verify the win

Count real tokens before/after with the provider's tokenizer (tiktoken for OpenAI o200k_base; Anthropic's token-counting endpoint for Claude). Track tokens per request and per completed task (agents), and confirm output quality holds — the goal is right information, not most information.

Adapted from Yuval Ben-itzhak, "How I Reduced LLM Token Costs by 90% Building AI Agents With OpenAI and Claude" (Medium, Mar 2026): https://medium.com/@ravityuval/how-i-reduced-llm-token-costs-by-90-using-prompt-rag-and-ai-agent-optimization-f64bd1b56d9f