# Agent Cost Observability

> Use when deciding which model tier a task should run on, investigating why a session or subagent burned more tokens than expected, or doing a periodic check that spend hasn't quietly drifted — grounded in what's actually observable here (session transcripts, /caveman-stats, Model Policy tiers), not a hypothetical enterprise dashboard.

- Skill: `presidenteog/agent-cost-observability` (Agent Skill)
- Install (CLI): `npx skillmds@latest add presidenteog/agent-cost-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/presidenteog/agent-cost-observability/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: PresidenteOG (https://skillmd.com/u/presidenteog)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/presidenteog/agent-cost-observability

---


# Agent Cost Observability

Adapted from https://raw.githubusercontent.com/ai-boost/awesome-prompts/main/prompts/agent_cost_observability_architect.txt (ai-boost/awesome-prompts, GPL-3.0) — rewritten, not copied.

## Overview
The source designs an enterprise multi-team cost platform — menubar widgets, chargeback invoicing, a 90-day time-series store. None of that exists here, and building it would solve a problem this workspace doesn't have. What transfers is the underlying discipline: you can't manage what you don't measure, cost is a quality signal as much as a spend number, and every anomaly needs a fix attached, not just a description. Scaled down to a solo Claude Code workspace with a `crew-*`/`cavecrew-*` roster.

## When to use
- Choosing which model tier (`inherit`/opus, `sonnet`, `haiku`) a new or existing agent should run on.
- A session or a delegated task feels like it burned more than it should have.
- A periodic pass to catch drift before it becomes a habit — e.g. mechanical work quietly running on `sonnet`/`opus` instead of the `haiku` the Model Policy calls for.

## What's actually available to measure here
| Signal | Where it lives |
|---|---|
| Per-session real token/cache usage | `/caveman-stats` — reads the actual Claude Code session log, not an estimate |
| Which model tier should apply to which job | `ClaudeFX/Documents/Model Policy.md` — the routing table, single source of truth |
| Whether a subagent actually used the pinned model | Agent frontmatter `model:` field, checked against what `/caveman-stats` shows for that dispatch |
| Historical pattern across sessions | claude-mem's session history, queried through `mem-search` |

Don't invent telemetry infrastructure that doesn't exist here — normalize against these four sources, not a fictional pricing registry or time-series database.

## Anomaly patterns worth checking for
| Pattern | What it looks like here | Action |
|---|---|---|
| Wrong tier | A mechanical job (locate code, single-file edit) ran on `sonnet`/`opus` instead of the `haiku` cavecrew pin | Check the agent's frontmatter `model:` against Model Policy; fix the pin, not just this one run |
| Context bloat | A session's input tokens spike well past its usual baseline for that task type | Check whether a large file was read in full where `smart-explore` or a targeted `Read` range would've done |
| Retry loop | The same failing action repeated instead of escalating | Cross-check against the 3-strike rule in `persistent-file-planner` — that's exactly the failure it exists to prevent |
| Reasoning upcharge without justification | A task that doesn't need judgment routed to `inherit`/opus anyway | Re-route to `sonnet` per Model Policy unless the task is genuinely planning or architecture-level |

## Process
1. Before dispatching a new or edited agent, check its `model:` pin against the Model Policy table — don't default to `inherit` out of caution; that's the upcharge-without-justification anomaly.
2. After a session that felt expensive, run `/caveman-stats` and read the actual numbers rather than guessing.
3. If a number looks off, match it against the anomaly table above before proposing a fix — the fix should be one concrete change (re-pin a model, split a task, stop a retry loop), not a vague "reduce usage."
4. Treat "this cost more than it should" as a quality signal worth investigating on its own, not just a spend line to shrug at — scope creep or wrong-tier routing usually shows up here before it shows up anywhere else.
5. If a genuine pattern emerges across several sessions (not just one noisy run), write it to memory so it doesn't get rediscovered from scratch next time.

## Design principles that still apply at this scale
- Visibility before enforcement — there's no hard budget wall here; the point is to notice, not to auto-kill a session mid-work.
- Normalize before comparing — a haiku-tier mechanical job and an opus-tier planning job cost differently by design; don't flag the difference as an anomaly.
- Every anomaly gets a one-line fix attached — "this cost more" alone isn't useful, "this should've been haiku, re-pin it" is.