Agent Cost Observability
Adapted from https://raw.githubusercontent.com/ai-boost/awesome-prompts/main/prompts/agent_cost_observability_architect.txt (ai-boost/awesome-prompts, GPL-3.0) — rewritten, not copied.
Overview
The source designs an enterprise multi-team cost platform — menubar widgets, chargeback invoicing, a 90-day time-series store. None of that exists here, and building it would solve a problem this workspace doesn't have. What transfers is the underlying discipline: you can't manage what you don't measure, cost is a quality signal as much as a spend number, and every anomaly needs a fix attached, not just a description. Scaled down to a solo Claude Code workspace with a crew-*/cavecrew-* roster.
When to use
- Choosing which model tier (
inherit/opus, sonnet, haiku) a new or existing agent should run on.
- A session or a delegated task feels like it burned more than it should have.
- A periodic pass to catch drift before it becomes a habit — e.g. mechanical work quietly running on
sonnet/opus instead of the haiku the Model Policy calls for.
What's actually available to measure here
| Signal |
Where it lives |
| Per-session real token/cache usage |
/caveman-stats — reads the actual Claude Code session log, not an estimate |
| Which model tier should apply to which job |
ClaudeFX/Documents/Model Policy.md — the routing table, single source of truth |
| Whether a subagent actually used the pinned model |
Agent frontmatter model: field, checked against what /caveman-stats shows for that dispatch |
| Historical pattern across sessions |
claude-mem's session history, queried through mem-search |
Don't invent telemetry infrastructure that doesn't exist here — normalize against these four sources, not a fictional pricing registry or time-series database.
Anomaly patterns worth checking for
| Pattern |
What it looks like here |
Action |
| Wrong tier |
A mechanical job (locate code, single-file edit) ran on sonnet/opus instead of the haiku cavecrew pin |
Check the agent's frontmatter model: against Model Policy; fix the pin, not just this one run |
| Context bloat |
A session's input tokens spike well past its usual baseline for that task type |
Check whether a large file was read in full where smart-explore or a targeted Read range would've done |
| Retry loop |
The same failing action repeated instead of escalating |
Cross-check against the 3-strike rule in persistent-file-planner — that's exactly the failure it exists to prevent |
| Reasoning upcharge without justification |
A task that doesn't need judgment routed to inherit/opus anyway |
Re-route to sonnet per Model Policy unless the task is genuinely planning or architecture-level |
Process
- Before dispatching a new or edited agent, check its
model: pin against the Model Policy table — don't default to inherit out of caution; that's the upcharge-without-justification anomaly.
- After a session that felt expensive, run
/caveman-stats and read the actual numbers rather than guessing.
- If a number looks off, match it against the anomaly table above before proposing a fix — the fix should be one concrete change (re-pin a model, split a task, stop a retry loop), not a vague "reduce usage."
- Treat "this cost more than it should" as a quality signal worth investigating on its own, not just a spend line to shrug at — scope creep or wrong-tier routing usually shows up here before it shows up anywhere else.
- If a genuine pattern emerges across several sessions (not just one noisy run), write it to memory so it doesn't get rediscovered from scratch next time.
Design principles that still apply at this scale
- Visibility before enforcement — there's no hard budget wall here; the point is to notice, not to auto-kill a session mid-work.
- Normalize before comparing — a haiku-tier mechanical job and an opus-tier planning job cost differently by design; don't flag the difference as an anomaly.
- Every anomaly gets a one-line fix attached — "this cost more" alone isn't useful, "this should've been haiku, re-pin it" is.
1---2name: agent-cost-observability3description: Use when deciding which model tier a task should run on, investigating why a session or subagent burned more tokens than expected, or doing a periodic check that spend hasn't quietly drifted — grounded in what's actually observable here (session transcripts, /caveman-stats, Model Policy tiers), not a hypothetical enterprise dashboard.4---56# Agent Cost Observability78Adapted from https://raw.githubusercontent.com/ai-boost/awesome-prompts/main/prompts/agent_cost_observability_architect.txt (ai-boost/awesome-prompts, GPL-3.0) — rewritten, not copied.910## Overview11The source designs an enterprise multi-team cost platform — menubar widgets, chargeback invoicing, a 90-day time-series store. None of that exists here, and building it would solve a problem this workspace doesn't have. What transfers is the underlying discipline: you can't manage what you don't measure, cost is a quality signal as much as a spend number, and every anomaly needs a fix attached, not just a description. Scaled down to a solo Claude Code workspace with a `crew-*`/`cavecrew-*` roster.1213## When to use14- Choosing which model tier (`inherit`/opus, `sonnet`, `haiku`) a new or existing agent should run on.15- A session or a delegated task feels like it burned more than it should have.16- A periodic pass to catch drift before it becomes a habit — e.g. mechanical work quietly running on `sonnet`/`opus` instead of the `haiku` the Model Policy calls for.1718## What's actually available to measure here19| Signal | Where it lives |20|---|---|21| Per-session real token/cache usage | `/caveman-stats` — reads the actual Claude Code session log, not an estimate |22| Which model tier should apply to which job | `ClaudeFX/Documents/Model Policy.md` — the routing table, single source of truth |23| Whether a subagent actually used the pinned model | Agent frontmatter `model:` field, checked against what `/caveman-stats` shows for that dispatch |24| Historical pattern across sessions | claude-mem's session history, queried through `mem-search` |2526Don't invent telemetry infrastructure that doesn't exist here — normalize against these four sources, not a fictional pricing registry or time-series database.2728## Anomaly patterns worth checking for29| Pattern | What it looks like here | Action |30|---|---|---|31| Wrong tier | A mechanical job (locate code, single-file edit) ran on `sonnet`/`opus` instead of the `haiku` cavecrew pin | Check the agent's frontmatter `model:` against Model Policy; fix the pin, not just this one run |32| Context bloat | A session's input tokens spike well past its usual baseline for that task type | Check whether a large file was read in full where `smart-explore` or a targeted `Read` range would've done |33| Retry loop | The same failing action repeated instead of escalating | Cross-check against the 3-strike rule in `persistent-file-planner` — that's exactly the failure it exists to prevent |34| Reasoning upcharge without justification | A task that doesn't need judgment routed to `inherit`/opus anyway | Re-route to `sonnet` per Model Policy unless the task is genuinely planning or architecture-level |3536## Process371. Before dispatching a new or edited agent, check its `model:` pin against the Model Policy table — don't default to `inherit` out of caution; that's the upcharge-without-justification anomaly.382. After a session that felt expensive, run `/caveman-stats` and read the actual numbers rather than guessing.393. If a number looks off, match it against the anomaly table above before proposing a fix — the fix should be one concrete change (re-pin a model, split a task, stop a retry loop), not a vague "reduce usage."404. Treat "this cost more than it should" as a quality signal worth investigating on its own, not just a spend line to shrug at — scope creep or wrong-tier routing usually shows up here before it shows up anywhere else.415. If a genuine pattern emerges across several sessions (not just one noisy run), write it to memory so it doesn't get rediscovered from scratch next time.4243## Design principles that still apply at this scale44- Visibility before enforcement — there's no hard budget wall here; the point is to notice, not to auto-kill a session mid-work.45- Normalize before comparing — a haiku-tier mechanical job and an opus-tier planning job cost differently by design; don't flag the difference as an anomaly.46- Every anomaly gets a one-line fix attached — "this cost more" alone isn't useful, "this should've been haiku, re-pin it" is.
PresidenteOG