rulebase-co Model cost management
Support AI costs appear as a monthly invoice with no tie to outcomes. Finance asks "cost per ticket"; engineering sees token spikes with no attribution. You cannot optimise what you do not decompose.
Build cost from your logs and your pricing — never cite invented $/ticket industry benchmarks. Vendor list prices × your measured tokens × your resolution definition.
Define cost per resolution honestly
Pick a resolution definition and stick to it:
- Bot-contained conversation — no human agent message after bot last spoke
- Any bot participation — cheaper denominator, mixes handoffs
- Issue resolved — needs repeat-contact window; harder, more meaningful
Cost per resolution = total model spend in window / count of resolutions
Report median and p90 — a few long multi-tool threads dominate means.
Segment by channel, language, contact driver, and model version. One headline average hides drivers where automation is uneconomic.
Where tokens actually go
Decompose a sample of conversations (or all if logging allows):
| Stage | Typical drivers | What to log |
|---|---|---|
| System prompt | Static instructions, few-shots | Input tokens fixed per call |
| Conversation history | Unbounded thread growth | Turns included, summarisation |
| Retrieval context | k chunks × chunk size | Chars retrieved, deduped or not |
| Tool loops | Each round re-sends context | Iteration count |
| Grader / classifier | Second model pass | Calls per conversation |
| Retries | Rate limits, validation failures | Retry count, backoff |
Prompt bloat — system prompt and few-shots growing every sprint without eval — is the most common silent doubling of input tokens.
Retrieval size — retrieving twelve long articles "for safety" when the model only uses one paragraph.
Retry loops — tool error → model retries → exponential context resend.
Produce a waterfall for a median conversation and for p90.
Cost vs quality frontier
Optimisation is not "cheapest model." Map points on your own frontier:
- Fix quality bar (eval set or production sample grade).
- Vary one lever: model tier, retrieval k, history window, summarisation.
- Measure quality delta and cost delta on the same cases.
| Lever | Cost effect | Quality risk |
|---|---|---|
| Smaller / cheaper model | ↓ spend | Regulated drivers, non-English |
| Shorter history | ↓ input | Loses thread context |
| Aggressive summarisation | ↓ input | Omits constraints customer stated |
| Lower retrieval k | ↓ input | Misses right doc |
| Drop grader on low-risk path | ↓ calls | Bad sends slip through |
Do not cut on segments where error cost exceeds token savings — compute that trade-off explicitly for your highest-risk drivers.
Practical reductions (after measurement)
Only after attribution:
- Trim system prompt — remove dead instructions; version and diff like code.
- Cap history — last N turns or summarise with eval regression on multi-turn cases.
- Right-size retrieval — rerank to fewer chunks; measure recall impact separately.
- Stop double model calls — grader on sample, not every message.
- Cache stable context — system prompt and KB snippets where platform supports it.
- Fix retry storms — tool timeouts and validation before the model loops.
Each change ships through the same eval gate as prompt changes — cost cuts that break deferral are net negative.
Monitoring
Track weekly:
- Total spend, conversations, resolutions, cost per resolution
- Input vs output token ratio (output-heavy → verbose prompts or long answers)
- Tool iterations p50 / p95
- Spend by model version (detect accidental prod on flagship tier)
Alert on spend per resolution spike with driver breakdown, not only total invoice.
Traps
- Attributing shared infra only to bot — include retrieval, embeddings, grader.
- Ignoring human cost — cheap model + more escalations can raise total cost to serve.
- Optimising English only — other languages may need different tier for same quality.
- One-week window after launch — novelty and marketing traffic skew denominators.
Present results to the user
- Resolution definition and denominators — what the ratio means.
- Headline cost per resolution — median, p90, by segment table.
- Token waterfall — where spend accumulates on median vs p90 conversations.
- Top levers — ranked by savings potential with measured or estimated impact.
- Frontier points — quality vs cost trades tested or proposed; no fake precision.
- Recommendations — specific changes, eval required, owner.
- Gaps in logging — what could not be attributed and what to instrument next.