Cost Tracking

Cost tracking

You are building the money meter for an AI or cloud app: every model call gets a token count, a price, a ledger row, and a budget check — so a cap fires before the invoice, not when finance forwards it in a panic. Model-API spend roughly doubled from $3.5B to $8.4B between late 2024 and mid 2025 (firecrawl.dev best-llm-observability-tools, accessed 2026-06-02); the bill is now big enough to need a guardrail, not a spreadsheet at month-end.

The chain you build, in order — each step feeds the next:

1. Capture — read tokens from the response, not from a pre-send guess.
2. Price — multiply tokens by a versioned, dated rate table.
3. Ledger — append one idempotent row per request, tagged for attribution.
4. Budget — roll the ledger up against a soft and a hard threshold.
5. Guard — alert, degrade, or refuse before the threshold becomes an invoice.

The one rule that organizes everything: bill against the response usage object. Anything you compute before the call is an estimate — good only for the pre-flight cap check, never for the ledger.

What this skill produces

A checkable cost setup: a pricing table (each model row carries effective_date + source), an append-only ledger schema (idempotency key + attribution keys), and a budget with both a soft and a hard threshold. scripts/verify.sh lints those artifacts (last section). Prose alone is not a deliverable — emit the config.

Capture: read usage, do not estimate

Pre-send token counts (tiktoken, Anthropic client.messages.count_tokens()) are estimates. They exist for the pre-flight cap check — "will this request likely breach the budget?" — and nothing else. The truth lands in the response: input, output, cached, and reasoning tokens, plus audio/image tokens where the modality applies. Bill the ledger off that object.

Two provider facts that bite if you assume otherwise:

• Anthropic count_tokens() returns input tokens only, is free, has its own rate limit, and is still an estimate. Anthropic is not tiktoken-compatible — do not reuse an OpenAI tokenizer to price Claude (platform.claude.com token-counting; github.com/anthropics/anthropic-tokenizer-typescript, accessed 2026-06-02).
• Output and reasoning tokens are the expensive half — output runs 4-5x the input rate. A meter that only counts input is wrong by most of the bill.

# Bad: pricing off a pre-send character/word guess. Wrong, and ignores output.
est_tokens = len(prompt) // 4
cost = est_tokens * rate_in            # output + reasoning never counted

# Good: capture every field the response actually reports, then price that.
resp = client.messages.create(model=model, messages=msgs, max_tokens=1024)
u = resp.usage
record = {
    "input_tokens":  u.input_tokens,
    "output_tokens": u.output_tokens,
    "cache_read_tokens":     getattr(u, "cache_read_input_tokens", 0),
    "cache_write_tokens":    getattr(u, "cache_creation_input_tokens", 0),
    # OpenAI exposes cached as usage.prompt_tokens_details.cached_tokens
}
cost = price(model, record)            # see "pricing is data" below

Wrap the SDK call once so capture cannot be skipped. A meter you have to remember to call is a meter that's already missing rows (langfuse.com token-and-cost-tracking, accessed 2026-06-02).

Pricing is data, never literals

Rates drift fast and silently mis-bill when stale. Keep prices in a versioned table — one row per model, each with effective_date and source — loaded as data. Never write a rate as a literal in business logic. Look up by model and fail loud on an unknown model; never default to $0, or a new model silently bills as free and the leak is invisible.

# pricing.yaml — perishable. Verify against source before trusting. Dated 2026-06-02.
models:
  - model: claude-haiku-4.5
    effective_date: 2026-06-02
    source: cloudzero.com/blog/claude-api-pricing
    input_per_mtok: 1.00
    output_per_mtok: 5.00
    cache_read_per_mtok: 0.10
  - model: claude-sonnet-4.6
    effective_date: 2026-06-02
    source: cloudzero.com/blog/claude-api-pricing
    input_per_mtok: 3.00
    output_per_mtok: 15.00
    cache_read_per_mtok: 0.30
  - model: claude-opus-4.7
    effective_date: 2026-06-02
    source: cloudzero.com/blog/claude-api-pricing
    input_per_mtok: 5.00
    output_per_mtok: 25.00
    cache_read_per_mtok: 0.50
  - model: gpt-5.5
    effective_date: 2026-06-02
    source: openai.com/api/pricing
    input_per_mtok: 5.00
    output_per_mtok: 40.00
    cache_read_per_mtok: 0.50   # OpenAI cached input = 90% off standard input

These numbers are a snapshot, not a constant — model names and rates move month to month. Dated 2026-06-02 from the sources above. The dated per-provider snapshots, the usage-field map per provider, and refresh instructions live in references/pricing-tables.md; read it before you trust a rate.

Ledger: append-only, idempotent, attributed

One row per request, append-only. Two things must be on every row or the ledger lies:

• A request/idempotency key. Retries and SDK auto-retries fire the same logical call twice; without a key the row is written twice and you double-count spend.
• At least one attribution key (user, tenant, feature, model). A per-org total can tell you the bill is high; it cannot tell you which feature or customer is the leak. Attribution is the difference between "spend is up" and "the summarize-document feature on the enterprise tenant tripled."

-- append-only; (request_id) is the idempotency key — upsert, never plain insert
CREATE TABLE llm_cost_ledger (
  request_id     TEXT PRIMARY KEY,           -- idempotency: retries collapse to one row
  ts             TIMESTAMPTZ NOT NULL,
  model          TEXT NOT NULL,
  input_tokens   INTEGER NOT NULL,
  output_tokens  INTEGER NOT NULL,
  cached_tokens  INTEGER NOT NULL DEFAULT 0,
  cost_usd       NUMERIC(12,6) NOT NULL,      -- priced from the table above
  user_id        TEXT,                        -- attribution keys
  tenant_id      TEXT,
  feature        TEXT
);

This is an operational ledger, not the accounting record — categorizing the spend into the books is bookkeeping, and the same rows feed the cost numerator in unit-economics and one input line in finance-ops. "Cost per active user" on the behavior side is analytics; charging customers for metered usage is stripe.

Budgets, alerts, caps

A budget needs a soft state (alert + degrade) and a hard state (refuse). Roll the ledger up per window (day/month) and per attribution key, then branch:

Two distinct checks, do not conflate them:

• Pre-flight gate uses the estimate (pre-send token count × rate) to refuse a request that would obviously blow the hard cap — this is the only legitimate use of an estimate.
• Post-hoc reconciliation rolls up the ledger (real usage) against the budget. When someone asks "why is the bill 3x the estimate," you compare provider-billed usage to your ledgered usage — the gap is almost always uncounted output/reasoning/cache-write tokens or missing rows from un-wrapped call sites.

The three levers that move the number

Don't assert savings — show the break-even. Pricing per fact-checked sources accessed 2026-06-02 (platform.claude.com prompt-caching; finout.io anthropic-api-pricing).

• Prompt caching. Cache reads cost 0.1x base input. The 5-min write costs 1.25x, the 1-hour write 2x. So a 5-min entry pays for itself after roughly one cache hit: 1.25 + 0.1·h (cached) beats 1·(1+h) (uncached) once h ≥ 1. Caching a stable system prompt across a session is almost always net cheaper.
• Batch API. 50% off on both OpenAI and Anthropic, async within 24h, and it stacks with caching — combined up to ~95% off. Use it for anything not user-facing-realtime: evals, backfills, nightly summaries.
• Model downshift. The largest lever. Route easy requests to Haiku/cheap models and reserve Opus/GPT-5.5 for hard ones — at 5x the rate, downshifting the routable half of traffic dwarfs a few percent of caching.

"We'll add caching later" without measuring the hit rate is a guess, not a lever. Instrument cache_read_tokens in the ledger first, then you know.

Cloud spend is the slow backstop

App-level metering is your real-time guard. Cloud billing alerts are a delayed backstop — useful, but never the thing standing between you and a runaway loop.

• AWS Cost Anomaly Detection is ML-based, runs ~3x/day with up to 24h data delay → SNS → Lambda. AWS Budgets adds threshold alerts on the same pipe.
• GCP/Azure are threshold-only. GCP's budget → Pub/Sub → Cloud Function is the one that can actually pause or throttle a workload programmatically.

Route every cloud alert into the same alert pipe as your app-level budget alerts so there's one place to look. The 24h delay is exactly why the in-app cap exists: by the time AWS notices the anomaly, the loop already spent the money. Recipes and the alert-routing pattern are in references/cloud-caps.md; the cap plumbing in depth is aws-essentials / gcp-essentials.

Build vs buy

(firecrawl.dev best-llm-observability-tools; guptadeepak.com top-5-llm-observability-platforms-2026, accessed 2026-06-02.) Buy the proxy/platform when you want spend visibility fast; build the ledger when caps and per-feature attribution must live inside your own logic.

Anti-patterns

Verify the artifact

scripts/verify.sh [path] lints a candidate cost config/ledger (yaml/json/ts) and fails if: a pricing entry lacks effective_date or source; a model referenced in logic is missing from the table; the ledger schema lacks an idempotency/request key or any attribution key; the budget declares no soft+hard pair; or cost looks derived from a len()/char estimate instead of a usage field. It is read-only and exits 0 on a clean config and on no config found — no false failure.