# Reduce LLM Token Costs

> Use when an LLM agent, RAG pipeline, or multi-step tool workflow is burning too many tokens/dollars per request — cut token cost ~90% by compressing tool defs, summarizing telemetry/logs, compressing RAG context, replacing conversation history with external state, hierarchical compression, small-model preprocessing, prompt caching, and conversation chaining. Reach for it when a single request exceeds a few thousand tokens of "plumbing" (system prompt + tools + history + retrieved chunks + raw data) rather than useful signal.

- Skill: `nsharandroidnstudio/reduce-llm-token-costs` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nsharandroidnstudio/reduce-llm-token-costs`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nsharandroidnstudio/reduce-llm-token-costs/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: nsharandroidnstudio (https://skillmd.com/u/nsharandroidnstudio)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/nsharandroidnstudio/reduce-llm-token-costs

---


# Reduce LLM Token Costs (~90%)

**Core insight:** *"LLM systems work best when the model sees the right information, not the most information."* The prompt you write is usually a small fraction of the tokens — the cost is the **system around it**: repeated system prompts, verbose tool definitions, conversation history, RAG chunks, and raw telemetry re-sent on every request. Multi-step agents make this worse: a 10-step task can process **50k–100k tokens** because each step re-sends context.

**Rule of thumb:** *Never send raw data to the model if it can be compressed first.*

## When to use this

- One request's "plumbing" (system + tools + history + retrieved docs + logs) dwarfs the actual question.
- An agent loops N reasoning steps and each step re-sends the whole context.
- RAG stuffs 5+ chunks × ~800 tokens straight into the prompt.
- You paste raw logs/telemetry/JSON dumps into prompts.
- Token bill scales with conversation length or step count, not with task difficulty.

## The worked example (from the source article)

A telemetry-analysis agent, per request:

| Component | Before | After | Technique |
|---|---:|---:|---|
| System prompt | 3,500 | (cached) | Prompt caching |
| Tool definitions | 2,000 | 200 | Compress tool defs |
| Telemetry logs | 4,000 | 300 | Summarize logs |
| Conversation history | 1,000 | 150 | State instead of history |
| Compressed RAG context | — | 200 | Compress RAG |
| **Total** | **10,500** | **~650** | **≈94% reduction** |

## The eight techniques

### 1. Compress tool definitions
Verbose natural-language tool docs cost hundreds of tokens each. Replace prose with a terse signature list — the model keeps performing.

```
# Before (~verbose)
Tool: analyze_cpu_usage
This tool analyzes CPU usage in Windows telemetry logs and should be used when the user wants to understand CPU spikes...

# After
tools:
- cpu_analyze(json)
- mem_analyze(json)
- disk_analyze(json)
- generate_fix(issue)
```

### 2. Summarize logs & telemetry before sending
Add a preprocessing layer that extracts key signals from raw data. Don't paste the dump.

```
# After — signal only
top_cpu_processes:
- chrome.exe 32%
- node.exe 14%
top_memory_processes:
- chrome.exe 1.2GB
```

### 3. Compress RAG context
Insert a **summarization step between retrieval and the final prompt**. 5 chunks × 800 = 4,000 tokens → a compressed summary. Cuts RAG payload **80–90%**.

```
retrieve → summarize → send compressed context
```

### 4. Replace conversation history with external state
Stop re-sending the whole transcript. Store workflow state (Redis/DB) and send only the relevant fields.

```json
{ "issue": "high_cpu", "suspected_process": "chrome.exe", "previous_actions": ["restart_service"] }
```

### 5. Hierarchical context compression
Multi-stage funnel for large document sets:

```
documents → chunk summaries → merged summary → final reasoning
# 5×800 = 4,000  →  chunk summaries 600  →  merged 200
# final prompt: system 1,500 + context 200 = 1,700  (vs 5,500)
```

### 6. Smaller models for preprocessing
Two-model pipeline: a **small/cheap model** filters & summarizes raw data; the **large model** reasons over only the compressed signal.

```
raw data → small model (summarize/filter) → compressed context → large model (reasoning)
```

### 7. Prompt caching
For large, stable prefixes (system prompt + tool defs) sent every request.
- **OpenAI:** automatic caching.
- **Claude:** explicit `cache_control`; up to **4** cache breakpoints; TTL **5 min** default (**1-hour** option); cached tokens billed at **~10%** of normal input price; minimum cacheable prefix **1,024 tokens**.

### 8. Conversation chaining (OpenAI Responses API)
Use `previous_response_id` to reference an earlier response instead of resending full history (e.g. request 1 analyzes → request 2 generates a fix referencing request 1). Note: earlier tokens can still count, so combine with state (#4).

## Reference architecture

```
Raw Data / Logs
  ↓ Preprocessing layer (summarize, filter)     # small model, #2 #6
  ↓ State store (Redis / DB)                     # #4
  ↓ RAG retrieval
  ↓ Context compression                          # #3 #5
  ↓ LLM reasoning (OpenAI or Claude)             # #7 #8
  ↓ Structured output
```

## Apply in priority order (highest ROI first)

1. **Compress RAG documents** (#3)
2. **Store workflow state externally** (#4)
3. **Summarize logs/telemetry** (#2)
4. **Smaller models for preprocessing** (#6)
5. **Prompt caching** (#7)

Then layer in compressed tool defs (#1), hierarchical compression (#5), and conversation chaining (#8).

## How to verify the win

Count real tokens before/after with the provider's tokenizer (`tiktoken` for OpenAI `o200k_base`; Anthropic's token-counting endpoint for Claude). Track tokens **per request** and **per completed task** (agents), and confirm output quality holds — the goal is *right information, not most information*.

---

*Adapted from Yuval Ben-itzhak, "How I Reduced LLM Token Costs by 90% Building AI Agents With OpenAI and Claude" (Medium, Mar 2026): https://medium.com/@ravityuval/how-i-reduced-llm-token-costs-by-90-using-prompt-rag-and-ai-agent-optimization-f64bd1b56d9f*