viktorbezdek Context Optimization Techniques
Context optimization extends effective capacity through strategic compression, masking, caching, and partitioning. The goal is making better use of available capacity — not magically increasing windows. Effective optimization can double or triple effective context without larger models.
When to Use / Not Use
Use when:
- Context limits constrain task complexity
- Optimizing for cost reduction (fewer tokens = lower costs)
- Reducing latency for long conversations
- Implementing long-running agent systems
- Building production systems at scale
Do NOT use when:
- Compressing or summarizing context -> use
context-compression - Diagnosing context failures or degradation -> use
context-degradation - Learning context basics -> use
context-fundamentals - File-based context patterns or scratch pads -> use
filesystem-context
Decision Tree
What optimization problem are you solving?
├── Tool outputs dominate token usage (>80%)
│ └── Observation Masking -> Replace verbose outputs with references
├── Context approaching limits (>70% utilization)
│ ├── Message history dominates -> Compaction + Summarization
│ ├── Retrieved docs dominate -> Summarization or Partitioning
│ └── Multiple components -> Combine strategies
├── Repeated requests with common prefixes
│ └── KV-Cache Optimization -> Stable prefix ordering
├── Single context too large for one agent
│ └── Context Partitioning -> Sub-agent isolation
├── Need to measure if optimization is working
│ └── See §Performance Targets
└── Not about extending capacity? -> See related skills
Four Primary Strategies
| Strategy | Mechanism | Best For | Expected Savings |
|---|---|---|---|
| Compaction | Summarize context near limits, reinitialize with summary | Message history dominating | 50-70% token reduction |
| Observation Masking | Replace verbose tool outputs with compact references | Tool output dominance (80%+ of tokens) | 60-80% reduction in masked obs |
| KV-Cache Optimization | Reuse cached KV computations across shared prefixes | Repeated requests with stable prefixes | 70%+ cache hit rate |
| Context Partitioning | Split work across sub-agents with isolated contexts | Single context too large | Isolation + clean focus |
Compaction
Summarize context sections when approaching limits, then reinitialize with the summary. Priority order for compression:
- Tool outputs — Replace with summaries of key findings, metrics, conclusions
- Old turns — Summarize early conversation, preserve decisions and context shifts
- Retrieved docs — Summarize if recent versions exist
- Never compress — System prompt
Target: 50-70% token reduction with less than 5% quality degradation.
Observation Masking
Replace verbose tool outputs with compact references once the output has served its purpose.
| Category | Action | Examples |
|---|---|---|
| Never mask | Critical to current task, most recent turn, active reasoning | Current file being edited, error being debugged |
| Consider masking | 3+ turns old, verbose with extractable key points, purpose served | Large file reads, search results already acted on |
| Always mask | Repeated outputs, boilerplate headers/footers, already summarized | Help text, version banners, duplicate reads |
if len(observation) > max_length:
ref_id = store_observation(observation)
return f"[Obs:{ref_id} elided. Key: {extract_key(observation)}]"
KV-Cache Optimization
KV-cache stores Key and Value tensors from inference, growing linearly with sequence length. Prefix caching reuses KV blocks across requests with identical prefixes via hash-based block matching.
Cache-friendly ordering principle: Stable elements first, unique elements last.
context = [system_prompt, tool_definitions] # Cacheable (stable)
context += [reused_templates] # Reusable (semi-stable)
context += [unique_content] # Unique (never cached)
Cache stability design:
- Avoid dynamic content (timestamps, random IDs) in prefixes
- Use consistent formatting across sessions
- Keep structure stable even when content changes
Context Partitioning
Most aggressive optimization: partition work across sub-agents with isolated contexts. Each sub-agent operates in a clean context focused on its subtask.
When to partition:
- Single context would exceed limits
- Independent subtasks with minimal dependencies
- Need separation of concerns (search context vs. synthesis context)
Result aggregation: Validate all partitions completed → merge compatible results → summarize if still too large.
Context Budget Management
| Budget Category | Typical Allocation | Notes |
|---|---|---|
| System prompt | 5-10% | Never compress |
| Tool definitions | 10-15% | Semi-stable, cacheable |
| Retrieved documents | 15-30% | Dynamic, filterable |
| Message history | 20-40% | Growing, compactable |
| Tool outputs | 30-50% | Dominant, maskable |
| Reserved buffer | 10% | Headroom for responses |
Trigger signals for optimization:
- Token utilization above 80%
- Degradation indicators (quality dropping)
- Performance drops (latency increasing)
- Cost thresholds exceeded
Anti-Patterns
| Anti-Pattern | Problem | Solution |
|---|---|---|
| Optimizing without measuring | Cannot distinguish effective optimization from harmful compression | Measure before and after: token counts, quality metrics, cache hit rates |
| Masking too aggressively | Masking recent or critical observations causes agent to lose context mid-task | Follow masking categories: never mask current-turn or active-reasoning outputs |
| Ignoring cache stability | Dynamic content in prefixes (timestamps, random IDs) breaks cache hits | Place stable content first; push dynamic content to end of context |
| Partitioning too early | Isolation overhead exceeds savings for moderately-sized contexts | Partition only when single context would exceed limits or subtasks are genuinely independent |
| Compressing system prompt | System prompt establishes identity; compression loses behavioral constraints | Never compress system prompt; always include in cacheable prefix |
| No compaction triggers | Context fills silently until severe degradation | Implement triggers at 70-80% utilization; monitor usage against budget |
| Applying single strategy | Different context components need different optimization techniques | Combine strategies based on context composition (see Decision Tree) |
| Assuming optimization is free | Each strategy has quality cost; aggressive targets degrade output | Target 50-70% compaction with <5% quality loss; measure, don't assume |
Guidelines
- Measure before optimizing — know your current state
- Apply compaction before masking when possible
- Design for cache stability with consistent prompts
- Partition before context becomes problematic
- Monitor optimization effectiveness over time
- Balance token savings against quality preservation
- Test optimization at production scale
- Implement graceful degradation for edge cases
Integration
- context-fundamentals - Prerequisite: basic context concepts
- context-degradation - Understanding when optimization is needed
- context-compression - Compression as one optimization technique
- multi-agent-patterns - Partitioning as isolation
- evaluation - Measuring optimization effectiveness
- memory-systems - Offloading context to memory
References
Internal reference:
- Optimization Techniques Reference - Detailed technical reference
External resources:
- Research on context window limitations and KV-cache optimization
- Production engineering guides from leading AI labs