Cost-Aware LLM Pipeline
Specialist for architecting cost-disciplined LLM applications and API integrations.
This skill balances output quality against token expenditure using complexity-based model routing, aggressive prompt caching, token budget ceilings, and graceful degradation cascades.
Boundaries & Handoffs
| Need |
Route |
| Application bundle size, page latency, web vitals, database query profiling |
full-performance-audit |
| Refining individual prompt wording, instructions, or negative constraints |
prompt-optimizer |
| Benchmarking accuracy tradeoffs across cheaper models or prompt versions |
eval-harness |
| Investigating failure loops or context pollution in an agent loop |
agent-architecture-audit |
| Engineering cost-optimized LLM architectures, routing, and token budgets |
cost-aware-llm-pipeline |
Core Patterns
Consult references/patterns.md for concrete implementation patterns:
Complexity-Based Model Tiering:
- Tier 1 (Fast/Cheap): Simple classification, routing, entity extraction, straightforward summarization, unit-test parsing.
- Tier 2 (Balanced): Standard code generation, multi-file refactoring, initial technical writing, triage.
- Tier 3 (Frontier Reasoning): Architectural synthesis, deep multi-hop debugging, security threat modeling, formal verification.
- Rule: Always route to the lowest sufficient tier; escalate only on detected ambiguity or failed evaluation checks.
Prompt Caching Structure:
- Organize prompts with invariant static prefixes (system instructions, tool definitions, reference schemas) at the top to maximize prefix-cache hits.
- Place volatile per-request dynamic data, user queries, and timestamps strictly at the end of the prompt buffer.
Token Budget Fences:
- Establish hard limits on max completion tokens per turn.
- Enforce sliding context compaction thresholds (e.g. compact conversation at 60% of window capacity rather than overflowing at 95%).
Retry Backoff & Fallback Cascades:
- Implement exponential backoff with full jitter for rate limits (
429) and server errors (5xx).
- Cascade down to smaller models or cached responses when upstream limits are encountered.
Cost Telemetry:
- Log input tokens, output tokens, cache-read tokens, and calculated cost per transaction.
- Set automated circuit breakers that halt autonomous loops when a cost threshold is exceeded.
1---2name: cost-aware-llm-pipeline3description: Design and optimize cost-efficient LLM application architectures. Covers model tiering, task complexity routing, prompt caching strategies, token budget fences, and retry backoff. Use when reducing API spend, establishing token budgets, or architecting multi-model fallback cascades. Not for client runtime latency/bundle audits (full-performance-audit) or prompt phrasing refinement (prompt-optimizer).4license: MIT5---67# Cost-Aware LLM Pipeline89Specialist for architecting cost-disciplined LLM applications and API integrations.1011This skill balances output quality against token expenditure using complexity-based model routing, aggressive prompt caching, token budget ceilings, and graceful degradation cascades.1213## Boundaries & Handoffs1415| Need | Route |16|---|---|17| Application bundle size, page latency, web vitals, database query profiling | `full-performance-audit` |18| Refining individual prompt wording, instructions, or negative constraints | `prompt-optimizer` |19| Benchmarking accuracy tradeoffs across cheaper models or prompt versions | `eval-harness` |20| Investigating failure loops or context pollution in an agent loop | `agent-architecture-audit` |21| **Engineering cost-optimized LLM architectures, routing, and token budgets** | **`cost-aware-llm-pipeline`** |2223## Core Patterns2425Consult [references/patterns.md](references/patterns.md) for concrete implementation patterns:26271. **Complexity-Based Model Tiering:**28 - **Tier 1 (Fast/Cheap):** Simple classification, routing, entity extraction, straightforward summarization, unit-test parsing.29 - **Tier 2 (Balanced):** Standard code generation, multi-file refactoring, initial technical writing, triage.30 - **Tier 3 (Frontier Reasoning):** Architectural synthesis, deep multi-hop debugging, security threat modeling, formal verification.31 - *Rule:* Always route to the lowest sufficient tier; escalate only on detected ambiguity or failed evaluation checks.32332. **Prompt Caching Structure:**34 - Organize prompts with invariant static prefixes (system instructions, tool definitions, reference schemas) at the top to maximize prefix-cache hits.35 - Place volatile per-request dynamic data, user queries, and timestamps strictly at the end of the prompt buffer.36373. **Token Budget Fences:**38 - Establish hard limits on max completion tokens per turn.39 - Enforce sliding context compaction thresholds (e.g. compact conversation at 60% of window capacity rather than overflowing at 95%).40414. **Retry Backoff & Fallback Cascades:**42 - Implement exponential backoff with full jitter for rate limits (`429`) and server errors (`5xx`).43 - Cascade down to smaller models or cached responses when upstream limits are encountered.44455. **Cost Telemetry:**46 - Log input tokens, output tokens, cache-read tokens, and calculated cost per transaction.47 - Set automated circuit breakers that halt autonomous loops when a cost threshold is exceeded.
kuker24