eyadsibai Context Optimization Techniques
Extend effective context capacity through compression, masking, caching, and partitioning. Effective optimization can 2-3x effective context capacity without larger models.
Optimization Strategies
| Strategy | Token Reduction | Use Case |
|---|---|---|
| Compaction | 50-70% | Message history dominates |
| Observation Masking | 60-80% | Tool outputs dominate |
| KV-Cache Optimization | 70%+ cache hits | Stable workloads |
| Context Partitioning | Variable | Complex multi-task |
Compaction
Summarize context when approaching limits:
if context_tokens / context_limit > 0.8:
context = compact_context(context)
Priority for compression:
- Tool outputs → replace with summaries
- Old turns → summarize early conversation
- Retrieved docs → summarize if recent versions exist
- Never compress system prompt
Summary generation by type:
- Tool outputs: Preserve findings, metrics, conclusions
- Conversational: Preserve decisions, commitments, context shifts
- Documents: Preserve key facts, remove supporting evidence
Observation Masking
Tool outputs can be 80%+ of tokens. Replace verbose outputs with references:
if len(observation) > max_length:
ref_id = store_observation(observation)
return f"[Obs:{ref_id} elided. Key: {extract_key(observation)}]"
Masking rules:
- Never mask: Current task critical, most recent turn, active reasoning
- Consider: 3+ turns old, key points extractable, purpose served
- Always mask: Repeated outputs, boilerplate, already summarized
KV-Cache Optimization
Cache Key/Value tensors for requests with identical prefixes:
# Cache-friendly ordering: stable content first
context = [
system_prompt, # Cacheable
tool_definitions, # Cacheable
reused_templates, # Reusable
unique_content # Unique
]
Design for cache stability:
- Avoid dynamic content (timestamps)
- Use consistent formatting
- Keep structure stable across sessions
Context Partitioning
Split work across sub-agents with isolated contexts:
# Each sub-agent has clean, focused context
results = await gather(
research_agent.search("topic A"),
research_agent.search("topic B"),
research_agent.search("topic C")
)
# Coordinator synthesizes without carrying full context
synthesized = await coordinator.synthesize(results)
Budget Management
context_budget = {
"system_prompt": 2000,
"tool_definitions": 3000,
"retrieved_docs": 10000,
"message_history": 15000,
"reserved_buffer": 2000
}
# Monitor and trigger optimization at 70-80%
When to Optimize
| Signal | Action |
|---|---|
| Utilization >70% | Start monitoring |
| Utilization >80% | Apply compaction |
| Quality degradation | Investigate cause |
| Tool outputs dominate | Observation masking |
| Docs dominate | Summarization/partitioning |
Performance Targets
- Compaction: 50-70% reduction, <5% quality loss
- Masking: 60-80% reduction in masked observations
- Cache: 70%+ hit rate for stable workloads
Best Practices
- Measure before optimizing
- Apply compaction before masking
- Design for cache stability
- Partition before context becomes problematic
- Monitor effectiveness over time
- Balance token savings vs quality
- Test at production scale
- Implement graceful degradation