Dev Guide Cache Hit Rate

Cache Hit Rate

Nested lingtai-dev-guide reference. Read this after the top-level router sends you here when you need to know how well prompt caching has been working recently for one or more LingTai agents, grounded in the token ledger rather than guessed.

This pairs with reference/runtime-self-check/SKILL.md §6: when a cache/affinity fix "should be live," the token ledger is the observable that proves it. This reference is the measurement; runtime-self-check is the did-the-object-rebuild diagnosis.

Core principle

A read-only metric. It only reads append-only logs/token_ledger.jsonl files; it never writes, rotates, or mutates runtime state. Report rates without pasting private absolute paths into human-facing deliverables — the ledger holds no secrets, but its parent paths can be private, so generalize to ~/.lingtai-tui/... or <project>/.lingtai/<agent>/.

Data source: the token ledger

Single source of truth: logs/token_ledger.jsonl, one JSON object per LLM call, written after every call by lingtai/kernel/token_ledger.py (append_token_entry). Required fields:

Optional tags on some entries: source (main, soul, tc_wake, daemon), and for daemon-attributed rows em_id / run_id / api_call_id / codex_*.

The kernel normalizes every provider's usage into these same fields before writing, so the metric is provider-agnostic (verified across gpt-5.5, mimo-v2.5-pro, deepseek-v4-pro, and the Anthropic adapters). Key invariant, confirmed in the adapters (lingtai/llm/anthropic/adapter.py, lingtai/llm/claude_code/adapter.py) and empirically over a full ledger: 0 <= cached <= input, so the hit rate is always in [0, 1].

Formula

For a window [now - W, now], over all entries whose ts lies in it:

hit_rate(W) = sum(cached) / sum(input)

• Denominator sum(input) already includes cached tokens, making this a token-weighted rate (a 100k-token call counts more than a 1k-token call) — which is what "how much of the prompt volume came from cache" means.
• Windows are a rolling lookback from now in UTC; defaults 1h, 5h, 1d, 3d. An entry is in the window iff now - W <= ts <= now. Ledger timestamps are UTC and "now" is computed in UTC — no local timezone involved.
• Zero denominator (no calls, or only zero-input rows) reports n/a, never a divide-by-zero.
• Missing/garbage rows — not valid JSON, no parseable ts, or non-numeric input/cached — are skipped and counted under skipped, so silent data loss is visible.

Caveat on cached. Native streaming/non-streaming adapters set cached = cache_read only; cache writes are billed into input but not counted as cached. CLI-backed daemon runs (lingtai/tools/daemon/run_dir.py) instead document cached as cache_read + cache_creation, because the CLI backend only exposes an aggregate. So CLI-backed entries lean slightly optimistic (first-write tokens read as "cached"). The cached <= input bound still holds — just don't over-interpret sub-percent differences on daemon CLI traffic.

The double-count hazard (important)

Daemon LLM calls are written to two ledgers: the daemon's own daemons/<run>/logs/token_ledger.jsonl and the parent agent's logs/token_ledger.jsonl (tagged source="daemon" + em_id/run_id). Naively globbing **/logs/token_ledger.jsonl under a project double-counts every daemon token.

The rule this reference and the script follow:

• Agent workdir — read only <workdir>/logs/token_ledger.jsonl; it already contains the daemon-tagged rows.
• Project root — read only each direct child <root>/<agent>/logs/token_ledger.jsonl; never recurse into daemons/.

Script: scripts/cache_hit_rate.py

Deterministic, read-only, standard library only. Accepts an agent workdir, a project root, or a single ledger file.

SCRIPT=~/.lingtai-tui/utilities/lingtai-dev-guide/reference/cache-hit-rate/scripts/cache_hit_rate.py
PY="$HOME/.lingtai-tui/runtime/venv/bin/python"   # or any python3.11+

# Current agent workdir (run from e.g. <project>/.lingtai/codex)
"$PY" "$SCRIPT" .

# A specific agent workdir
"$PY" "$SCRIPT" <project>/.lingtai/codex

# A whole project root: each agent's ledger, aggregated (daemons not double-counted)
"$PY" "$SCRIPT" <project>/.lingtai

# A single ledger file, custom windows, JSON output
"$PY" "$SCRIPT" logs/token_ledger.jsonl --windows 1h 6h 1d --json

# Only main-chat turns; pin the clock for a reproducible result
"$PY" "$SCRIPT" . --source main --now 2026-06-22T01:00:00Z

Flags: --windows (<int><s|m|h|d|w>, e.g. 90m 1d 1w), --source (filter to one source tag), --now ISO (override the clock for deterministic runs), --json, --help.

Example text output:

 window    calls           input          cached  hit_rate
----------------------------------------------------------
     1h       43       3,933,316       1,627,648     41.4%
     5h       43       3,933,316       1,627,648     41.4%
     1d       43       3,933,316       1,627,648     41.4%
     3d       43       3,933,316       1,627,648     41.4%

Equal rows across windows just mean all recent activity fell inside the smallest window (e.g. one active session in the last hour).

Exit codes: 0 success (including empty windows); 1 no ledger found under the path; 2 bad argument (missing path, bad --now, bad --windows).

One-liner without the script

For a single window against one explicit ledger path:

PY="$HOME/.lingtai-tui/runtime/venv/bin/python"
"$PY" - logs/token_ledger.jsonl 5 <<'PY'
import json, sys
from datetime import datetime, timedelta, timezone
path, hours = sys.argv[1], float(sys.argv[2])
cut = datetime.now(timezone.utc) - timedelta(hours=hours)
inp = cac = 0
for line in open(path):
    line = line.strip()
    if not line: continue
    try: d = json.loads(line)
    except ValueError: continue
    ts = d.get("ts","")
    try: t = datetime.fromisoformat(ts.replace("Z","+00:00"))
    except ValueError: continue
    if t >= cut:
        inp += d.get("input",0); cac += d.get("cached",0)
print(f"{cac}/{inp} = {100*cac/inp:.1f}%" if inp else "n/a (no input in window)")
PY

It takes one explicit path, so it cannot trip the double-count hazard — but it does not report skipped rows. Prefer the script for anything you will report.

Troubleshooting

• no token_ledger.jsonl found — the path is neither an agent workdir (logs/token_ledger.jsonl) nor a project root with child agents. Point at the agent dir (e.g. .lingtai/codex), the .lingtai/ root, or the ledger file.
• All windows n/a — no input tokens in those windows: idle agent, or --now predates the activity. Widen the window, drop --source, or check ts ranges with head -1 / tail -1.
• Rate looks too high on daemon traffic — the CLI cache_creation caveat above.
• skipped is non-zero — corrupt/rotated lines or pre-schema rows. A handful is normal (e.g. a partially written final line); a large fraction suggests schema drift.
• Schema drift — if a future kernel renames input/cached/ts, this reference and the script must be updated in the same spirit as the "anatomy travels with code" rule. Re-read lingtai/kernel/token_ledger.py and the active provider adapter to re-confirm field semantics before trusting numbers.
• Project-root totals equal one agent's — expected when only one agent has recent activity; idle colleagues contribute zero.

Validating after a change

Sanity-check the script against a known answer with a fixture and a pinned clock:

T=$(mktemp -d); mkdir -p "$T/a/logs"
printf '%s\n' \
 '{"source":"main","ts":"2026-06-22T00:50:00Z","input":1000,"output":1,"thinking":0,"cached":800}' \
 '{"source":"main","ts":"2026-06-21T22:00:00Z","input":1000,"output":1,"thinking":0,"cached":500}' \
 > "$T/a/logs/token_ledger.jsonl"
"$PY" "$SCRIPT" "$T/a" --now 2026-06-22T01:00:00Z   # 1h -> 80.0%, 5h -> 65.0%
rm -rf "$T"

Related references

• reference/runtime-self-check/SKILL.md — §6 live-object lifecycle: the ledger as proof a cache/affinity fix took effect after refresh.
• reference/debug-troubleshoot/SKILL.md — broader runtime diagnostics when a low/zero hit rate points at a misbehaving session rather than a metric.
• reference/architecture/SKILL.md — where runtime state (including .lingtai/<agent>/logs/) lives.