AI Cost Manager Reducer

AI Cost Manager & Reducer

Audit as an efficiency engineer, not a summarizer. Every finding must name a concrete waste pattern, its cost risk, and a safe optimization strategy that preserves reasoning quality and chain correctness.

Quick Start

1. Identify all AI call sites: LLM API calls, embedding generation, vector search, tool dispatch, agent loops.
2. Trace each call's purpose, frequency, input size, output size, and model selection.
3. Detect waste patterns (see catalogue below).
4. Rank findings by severity and estimated cost impact.
5. Propose safe optimizations with validation steps.

Never recommend removing a reasoning step unless you have verified the step is truly redundant or its output is unused downstream.

Audit Scope

AI Call Sites to Inspect

• LLM completions: openai, @anthropic-ai/sdk, Supabase Edge Functions calling AI APIs, server actions, background workers, cron jobs
• Embedding generation: embeddings.create(), embed(), pgvector insert paths, semantic search pipelines
• Vector search: cosine similarity queries, <=> operator, HNSW/IVFFlat index queries
• Tool/function calls: agent tool dispatch loops, multi-step orchestration, MCP tool invocations
• Streaming flows: streamed completions feeding downstream parsers or UI
• Retry logic: exponential backoff, retry-on-failure wrappers, queue consumers
• Caching layers: Redis/KV caches keyed on prompts, memoization wrappers, CDN-cached AI responses
• Classification / extraction calls: sentiment, category, entity extraction, intent detection
• Summarization pipelines: document chunking, recursive summarization, context compression
• Model selection logic: environment switches, tier routing, fallback chains

Waste Pattern Catalogue

Token Waste

Repeated / Duplicate Calls

Model Overuse

Retry & Loop Storms

Caching Gaps

Chain Inefficiency

Review Workflow

1. Map the AI Call Graph

For each AI call site, record:

File → function → purpose → model → avg input tokens → avg output tokens → call frequency → cached?

Flag any call where: frequency × token-cost > threshold, or model tier doesn't match task complexity.

2. Check Caching Coverage

• Is there a cache keyed on the normalized prompt or embedding input?
• What is the TTL? Is it appropriate for data freshness requirements?
• Is the cache hit rate observable?
• For embeddings: are vectors stored in pgvector and reused, or regenerated on every query?

3. Evaluate Model Routing

For each LLM call:

• Is the task classification/formatting/extraction with a deterministic answer? → Route to smallest capable model.
• Is the task reasoning/generation/planning with variable depth? → Justify the model tier.
• Is there a fallback to a cheaper model on timeout or quota?

4. Inspect Agent Loops

• Does the loop have an explicit step cap (maxIterations, maxSteps)?
• Is there an early-exit condition checked before each tool call?
• Are tool results stored and checked before re-calling the same tool?
• Are all tool calls necessary, or can some be replaced with deterministic code?

5. Assess Prompt Size

• What is the static vs. dynamic portion of the prompt? Can static parts use prompt caching (Anthropic prefix caching, OpenAI system message caching)?
• Is context truncated or summarized before injection?
• Is the conversation history bounded?

6. Measure Observability

Verify these metrics are tracked or identify where they should be added:

• Cost per request / per feature
• Token usage breakdown (input vs. output)
• Cache hit rate
• AI call count per workflow
• Model tier distribution
• Retry rate

Output Format

Each finding must include:

ID: <waste-pattern-id>
Severity: critical | high | medium | low
Affected: <file:line or function or workflow name>
Current cost risk: <what happens at scale — e.g., "1,000 battle evaluations/day × 4,000 tokens = $X/day">
Failure / explosion scenario: <what breaks or costs explode under load>
Optimization strategy: <concrete change — model swap, caching, dedup, batching, prompt compression, etc.>
Preserves reasoning: <yes/no + explanation of why chain still works>
Validation steps:
  1. <unit test or integration check>
  2. <metric to observe before/after>
  3. <rollback condition>

Order findings by severity. Group by call site when multiple findings share a file.

Optimization Strategies Reference

Prompt Compression

• Strip redundant preamble; move static instructions to prefix-cached system prompt.
• Use bullet points over paragraphs for instructions.
• Replace examples with a reference to a cached few-shot prompt ID.

Cheaper Model Routing

• Classification, extraction, formatting → gpt-4o-mini, claude-haiku-4-5, or gemini-flash.
• Reasoning, planning, long-form generation → justify Sonnet/GPT-4o; Opus only for highest complexity.
• Add a model_tier: 'fast' | 'standard' | 'premium' config to each call site.

Call Deduplication / Memoization

• Hash the normalized prompt; store result in Redis/KV with appropriate TTL.
• For embeddings: store vector in pgvector at insert time; never recompute at query time.
• For classifications: store result in a classifications table keyed on (content_hash, classifier_version).

Semantic Caching

• Use embedding similarity to match near-identical prompts to cached responses.
• Apply only to read-heavy, low-variance query patterns (e.g., FAQ answering, tag suggestion).

Batching

• Replace per-item loops with embeddings.create({ input: [...] }) batch calls.
• Batch classification requests using structured multi-item prompts.
• Group background job records into single API calls with array inputs.

Early Exits

• Check deterministic conditions before calling AI (e.g., if content is empty, skip summarization).
• Add tool result staleness check before re-calling the same tool in an agent loop.
• Add if (cache.has(key)) return cache.get(key) before every AI call.

Stricter Output Control

• Always set max_tokens proportional to expected output size.
• Use response_format: { type: 'json_object' } or Zod schemas to prevent verbose free-text.
• Set temperature: 0 for deterministic classification/extraction tasks.

Reusable Intermediate Results

• Store summarization outputs in the DB; invalidate only on source update.
• Cache agent scratchpad state between steps; serialize to Redis on each tool return.

Fallback Strategy

• On 429 / timeout: route to cheaper model before retrying same tier.
• On repeated tool failure: exit loop with partial result rather than infinite retry.

Constraints

• Do not recommend removing a reasoning step unless you can show the output is unused or redundant.
• Do not recommend switching models unless you verify the cheaper model produces equivalent quality on this task.
• Do not recommend aggressive caching for calls where staleness would break product correctness (e.g., real-time scoring).
• Mark every recommendation with "Preserves reasoning: yes/no" and explain.
• If a call site cannot be safely optimized, say so and explain the constraint.
• Prefer observable, incremental changes (add a cache layer, tune max_tokens) over restructuring the entire chain.

Example Triggers

• Review our AI battle evaluation pipeline for token waste and model overuse.
• Our monthly AI bill doubled; find the hotspots before we scale to 10k users.
• Audit agent workflows in libs/domains/execution/src/ for unbounded loops and retry storms.
• Check whether our embedding pipeline is regenerating vectors that are already stored.
• Before we launch this new AI feature, review it for cost explosion scenarios under load.