# Tokens

> Token-budget observability for godmode loops. Per-round cost tracking, token-log.tsv append, session-end summary, per-skill breakdown, delta vs prior sessions.

- Skill: `arbazkhan971/tokens` (Agent Skill)
- Install (CLI): `npx skillmds@latest add arbazkhan971/tokens`
- Raw SKILL.md: https://api.skillmd.com/api/skills/arbazkhan971/tokens/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Finance & Business
- Author: arbazkhan971 (https://skillmd.com/u/arbazkhan971)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/arbazkhan971/tokens

---


## Activate When
- `/godmode:tokens`, "token budget", "how expensive was this loop", "context cost"
- Any multi-round skill (optimize, refactor, debug, bench, terse) that runs a loop with DECIDE/LOG boundaries
- At session end, when the user asks "is my token usage trending up or down?"
- NOT for live streaming or real-time metering — this is append-per-round observability only. If you want a live meter, use a shell prompt, not this skill.
- Opt-out: if `GODMODE_TOKENS=0` is set in the environment, this skill is a no-op. Privacy-sensitive workflows (medical, secrets, PII sweeps) can set this to disable all round-size logging for the session. Opt-out is sticky per session; never overridden by auto-activation.

## Inputs
Ask once, cache for the session:
- `session_id` — stable id for the run. Default: `date +%s` at session start, reused across every round.
- `skill` — the skill driving the current round (read from session-state.json, not re-asked).
- `round` — monotonic counter, read from session-state.json.
- `terse_on` — boolean. `1` if `$GODMODE_TERSE` is non-empty, else `0`.
- `rtk_detected` — `1` if `command -v rtk` returns zero, else `0`. Logged as environment context, not used for computation.

## Workflow
At each round boundary — specifically AFTER the DECIDE/LOG step of the Universal Protocol, so a full round has been emitted:

1. Skip entirely if `GODMODE_TOKENS=0`. Emit nothing, write nothing.
2. Measure `input_chars` — sum of bytes of all files READ during this round plus the emit text of the prior round. Source: session-state.json's `round_context_files` array (maintained by the orchestrator) and `.godmode/last-round-emit.txt` (atomically rewritten after every round).
3. Measure `output_chars` — byte count of the current round's emit text, captured into `.godmode/last-round-emit.txt` as the round closes.
4. Convert to approximate tokens via the heuristic in the "Token Heuristic" section below.
5. Read the prior `cumulative_tokens` for this `session_id` from the last matching row of `.godmode/token-log.tsv`. Default 0 for round 1.
6. Append ONE row to `.godmode/token-log.tsv` with the schema in "TSV Schema".
7. Roll over the log if row count exceeds 10000 (see "Log Rotation").

At session end (any stop_reason), emit the summary described in "Output Format".

## Token Heuristic
Exact tokenization varies per model and is not worth a dependency on model-specific libraries for an observability skill. This skill uses one explicit, reproducible rule:

```
approx_tokens = ceil(char_count / 4)
```

Rationale: for English prose and code, 1 token averages ~3.8-4.2 characters across mainstream BPE tokenizers (GPT-4 and Llama families). Dividing by 4 gives a stable, model-agnostic estimate that is wrong by a consistent factor. Since this skill measures *trends* (is my loop getting more or less expensive?), a stable-biased estimator is sufficient. It is NOT accurate enough for billing, and the skill never claims to be.

Computed with `awk` — no python, no tiktoken, no external tokenizer binary:
```
tokens=$(awk -v c="$char_count" 'BEGIN { print int((c + 3) / 4) }')
```

`char_count` is `wc -c` of the measured file or text. Bytes, not graphemes. This is documented here so downstream readers of token-log.tsv can reproduce the math byte-for-byte.

## TSV Schema
`.godmode/token-log.tsv` is append-only. Header written on first create:
```
timestamp	session_id	round	skill	input_tokens	output_tokens	total_tokens	terse_on	rtk_detected	cumulative_tokens
```

Column meanings:
- `timestamp` — ISO 8601, UTC, second precision.
- `session_id` — matches session-state.json.
- `round` — integer, 1-indexed, monotonic within a session.
- `skill` — name of the driving skill for this round (e.g. `optimize`, `refactor`).
- `input_tokens` — approximate tokens read as context during the round.
- `output_tokens` — approximate tokens emitted by the round.
- `total_tokens` — `input_tokens + output_tokens`.
- `terse_on` — 0 or 1. Useful for comparing terse vs. non-terse sessions.
- `rtk_detected` — 0 or 1. Useful for comparing environments that have rtk installed (input-side compression) vs. those that do not.
- `cumulative_tokens` — running total for this `session_id` including the current round.

Never rewrite history. Never insert. One row per `(session_id, round)` pair. If a row already exists for that pair, SKIP — do not double-log.

## Output Format
At session end, emit a terse summary line (respects GODMODE_TERSE):

Terse:
```
Tokens: session={id} rounds={N} total={T} top={skill}({Ttop}) Δ={±pct}% vs last
```

Normal:
```
Tokens: session {id} completed {N} rounds, total {T} tokens.
Top skill: {skill} at {Ttop} tokens. Delta vs. last session for same skill: {±pct}%.
```

With `--verbose`, also print per-skill breakdown and top-3 expensive rounds:
```
Per-skill breakdown:
  optimize    4 rounds   12800 tok
  debug       2 rounds    3400 tok

Top 3 expensive rounds:
  R7 optimize  4200 tok   (1900 in / 2300 out)
  R3 optimize  3800 tok   (1700 in / 2100 out)
  R5 debug     2100 tok   ( 900 in / 1200 out)
```

Delta vs. last session is computed by scanning token-log.tsv backwards for the previous `session_id` whose top-skill matches, comparing `total` tokens, and printing signed percent.

## Success Criterion
One shell command must answer "is my session's total token consumption going up or down over time?":

```bash
awk -F'\t' 'NR>1 { sum[$2]+=$7; t[$2]=$1 } END {
  for (s in sum) print t[s], s, sum[s]
}' .godmode/token-log.tsv | sort
```

This prints every session chronologically with its total. Piping to `awk 'NR>1 { printf "%s %d delta=%+d\n", $2, $3, $3 - prev } { prev=$3 }'` yields round-over-round deltas. If the last few deltas trend negative, the loop is getting cheaper (terse mode working, rtk helping, prompts tightening). If they trend positive, investigate.

## Hard Rules
1. READ-ONLY to the codebase. The only files this skill may write are `.godmode/token-log.tsv`, `.godmode/token-log.tsv.*.gz`, `.godmode/last-round-emit.txt`, and the session-end summary it prints to stdout. Touching any source file = abort + fail loud.
2. Append-only. Never rewrite or edit existing rows. Rotation is the only way to shrink the active log, and it preserves history as a gzip.
3. Never block the loop. If measurement fails (disk full, permission denied, racing write), emit a warning to stderr and skip the append for that round. The loop must proceed.
4. Never depend on model-specific tokenizers. The char/4 heuristic is the contract. If a future version adds a more accurate estimator, it MUST be additive — old rows stay readable under the documented rule.
5. No real-time streaming. This is per-round append, not a live meter. One row per round, at the DECIDE/LOG boundary, no sooner.
6. Respect `GODMODE_TOKENS=0` unconditionally. Opt-out wins over every other flag, including `--verbose`.
7. Persists across sessions. The file lives in `.godmode/`, which is committed with the rest of godmode state. Never gitignore token-log.tsv.
8. No external deps beyond `awk`, `wc`, `gzip`, `date`, and standard Unix utilities. No python, no jq, no node, no tiktoken.
9. `.godmode/last-round-emit.txt` is a disposable scratch cache: it holds only the previous round's emit text and is atomically rewritten every round. Projects that commit `.godmode/` state SHOULD still gitignore this one file (the project maintainer adds `.godmode/last-round-emit.txt` to `.gitignore`) — it frequently quotes repo code and can carry secrets surfaced in prior output. This is the one deliberate exception to Rule 7: `token-log.tsv` (append-only history) stays committed; `last-round-emit.txt` (rewritten scratch) does not need to be.

## Keep / Discard Discipline
This skill does not keep or discard anything — it only observes. Rounds driven by other skills may themselves be kept or discarded per the Universal Protocol; the token row is logged either way, with the driving skill's final keep/discard status inferred from `results.tsv` (not re-measured here). Observing a discarded round is still valuable signal: expensive discards point to skills that burn context without moving the metric.

## Stop Conditions
- `session_ended` — the driving skill hit target_reached, budget_exhausted, diminishing_returns, or stuck. Emit summary, flush any pending row, exit.
- `opted_out` — `GODMODE_TOKENS=0`. Exit silently after the first check.
- `rotation_failed` — gzip returned non-zero or disk full. Emit stderr warning, continue without rotating, do NOT lose the active log.

On any stop: the last row written is authoritative; the summary reflects exactly what is in token-log.tsv, never more.