Plan LLM Cost Guardrails

LLM Cost-Guardrail Audit + Remediation Plan

Degree of freedom: HIGH — inventory call sites, score unbounded paths, plan. Stay plan-only. No limits or routing changes until approved.

This skill vs neighbors

How to reason (every plan item)

1. Propose — token cap, kill switch, breaker, or fallback
2. Risk — worst-case spend × who can reach the path
3. Keep-working — call sites that already bound tokens and account usage
4. Phase — caps → breakers → fallback → visibility (do not execute)

Worked example

Propose: per-user daily token cap + hard daily-spend kill switch on lib/ai.ts. Risk: public /api/chat can replay a 50K context until the bill dies; RPM-only does not count. Keep-working: summarizer path already sets max_tokens. Phase: Phase 1 — caps & kill switch.

Role: Senior platform engineer (LLM spend + abuse resistance).

Task: Inventory every LLM call site, test against the 3-layer guardrail model, score unbounded paths, phase remediations, emit plan-llm-cost-guardrails.md. Audit & plan only — no limits or routing changes until approved.

Find every path to a runaway bill. Cap it. Change nothing until approved.

Token cost scales with input + output tokens, not request count — a single 50K-token context replayed three times can exhaust a budget while staying under any RPM cap. Vibe-coded AI features ship with no spend cap, no per-user quota, no max_tokens, no circuit breaker — compounded by prompt-injection cost amplification and forged-webhook quota fraud (the empty-signing-secret bypass class).

This is the prevention counterpart to Langfuse observability: Langfuse tells you what spend happened; this audits what's capped.

When this fires

Trigger phrases: "cap my AI costs", "my LLM bill could explode", "rate limit my AI", "token budget", "someone could drain my quota", "runaway agent loop", "per-user AI limits", "pre-launch cost hardening".

Do not fire for: output quality/evals (audit-langfuse-llm), generic API performance, or trace visibility (plan-error-handling). This owns bounded spend.

The audit — 3-layer guardrail model [HIGH freedom]

Layer 1 · Limits (token-aware, not request-count)

• Token-bucket / quota per (user, model) — any per-identity limit?
• Token-based, not just RPM — prompt-TPM and output-TPM ceilings.
• max_tokens / context caps — bound worst-case cost; truncate RAG context.
• Short + long windows — per-minute burst and per-day/month budget.
• Tiered limits — free vs paid wired to Stripe entitlement.

Layer 2 · Circuit breakers

• Cost-velocity breaker — spend/min threshold.
• Loop / repeat detection — retry-storms, growing-context loops.
• Daily-spend kill switch — hard cap backstop.
• Error-rate breaker — mostly-failing caller identity.

Layer 3 · Fallback chain

• Primary → cheaper model → cache → graceful 503.
• Semantic cache before paid calls.
• Model routing by complexity — flag everything-to-Opus patterns.

Cross-cutting

• Streaming usage accounting — stream_options.include_usage or spend is invisible.
• Retry discipline — token-aware backoff.
• Abuse vectors — unauthenticated AI endpoints; hand boundary fixes to plan-input-validation.
• Langfuse cost alerts — 50/75/90% thresholds; per-user attribution.

Procedure [HIGH freedom]

1. Inventory LLM call sites and public AI endpoints.
2. Test each against Layers 1–3.
3. Score = worst-case spend × reachability.
4. Phase — Layer 1 caps and daily kill switch first.
5. Emit plan-llm-cost-guardrails.md. End the turn.

Guardrails [LOW freedom — run exactly]

• Plan only. No limits, gateway config, or routing changes.
• Spend cap is non-negotiable for launch — flag absence as at least High.
• Token-aware or it doesn't count — don't credit RPM-only limits.
• Bounded blast radius, not zero runaways.
• Cross-hand abuse to plan-input-validation.

Self-critique before the burndown [LOW freedom — do not skip]

1. evidenced-not-assumed — each unbounded path names a call site, not "we should cap AI"
2. plan-only — no limits, gateway, or routing edits this pass
3. phase justified — Layer 1 caps + daily kill switch before fallback polish
4. right-owner — quality/evals → audit-langfuse-llm; unauth AI input → plan-input-validation
5. no-false-safety — RPM-only is not a token cap; missing spend cap is at least High

Report template — plan-llm-cost-guardrails.md

# LLM Cost-Guardrail Audit — <repo>

_Audit-only. No limits or routing change until each phase is approved._

## Scope
- LLM call sites: n  | Public AI endpoints: n  | Langfuse present: ☐

## Verdict
| Layer | Present? | Worst gap |
|-------|----------|-----------|
| 1 Limits (token-aware) | partial | unbounded max_tokens |
| 2 Circuit breakers     | ❌ | no daily kill switch |
| 3 Fallback chain       | ❌ | limit = hard error |

## Findings
| # | Call site | Unbounded path | Worst-case | Missing layer | Sev | Direction |
|---|-----------|----------------|------------|---------------|-----|-----------|

## Phased burndown
- **Phase 1 — Caps & kill switch** → `backend-patterns`
- **Phase 2 — Circuit breakers** → `backend-patterns`
- **Phase 3 — Fallback chain** → `backend-patterns`
- **Phase 4 — Visibility** → `audit-langfuse-llm` / `backend-observability`
- **Cross-hand** → `plan-input-validation`

## Execution handoff
Simulate a runaway in test env after Phase 1; confirm cap holds before bill moves.

Chains with

• Observability & spend loop — pairs with plan-error-handling (visibility) and audit-langfuse-llm (quality); this owns bounded spend.
• audit-llm-security — LLM10 unbounded consumption is the attack-shaped twin.
• audit-infra-cost — hosting/egress, not tokens.
• Execution: backend-patterns, audit-langfuse-llm, backend-observability.
• Verify: sandbox load/abuse test — caps, breakers, fallback trip before spend escapes.

Plan with a strong model; execute with composer-2.5-execution.mdc.