# Token Doctor

> Audits and reduces token usage in context files (CLAUDE.md, AGENTS.md, memory files, docs). Measures token cost per file, finds waste (filler words, verbose phrases, blank bloat, oversized blocks), and safely compresses prose while preserving all code, inline code, URLs, file paths and tables. Use when your context files feel bloated, prompts are expensive, or you want to trim CLAUDE.md without losing meaning. Trigger: "reduce tokens", "token usage", "trim my CLAUDE.md", "compress context", "shrink memory file", "context too big", "save tokens", "audit token cost", /token-doctor. For skill description collisions, see skill-doctor.

- Skill: `enmr10/token-doctor` (Agent Skill, multi-file: 12 files)
- Install (CLI): `npx skillmds@latest add enmr10/token-doctor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/enmr10/token-doctor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: enmr10 (https://skillmd.com/u/enmr10)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/enmr10/token-doctor

---


# Token-Doctor

Measure and cut the token cost of the files that load into every prompt.

## When to run

- A `CLAUDE.md` / `AGENTS.md` / memory file has grown large
- Prompts feel expensive or context fills up fast
- Before committing context files (keep them lean)
- As a CI gate so no context file silently bloats

## Input

A single file, or a directory (scans `*.md`, `*.txt`, `CLAUDE.md`, `AGENTS.md`,
`llms.txt`, etc.; skips code, `node_modules`, `.git`).

## How to run

The engine lives in `scripts/` (Python 3.8+ stdlib, zero dependencies).

```bash
# Audit (read-only) — see token cost + savings
python3 -m scripts /path/to/project

# Audit a single file
python3 -m scripts CLAUDE.md

# Apply the reductions (keeps a .bak backup per file)
python3 -m scripts CLAUDE.md --apply

# CI gate: fail if any context file exceeds a token budget
python3 -m scripts /path/to/project --fail-over 1500

# Machine-readable
python3 -m scripts /path/to/project --format json --out audit.json
```

## What it does

1. **Measures** estimated tokens per file (offline heuristic; % saved is reliable).
2. **Splits** each file into prose vs protected regions (code, inline code, URLs,
   paths, tables, frontmatter).
3. **Reduces** ONLY prose: drops filler, shortens verbose phrases, collapses blank
   bloat, removes duplicate adjacent lines.
4. **Reports** before/after tokens, % saved, biggest-savings files, and waste findings.
5. **Applies** safely on `--apply` (writes `<file>.bak` first).

## Guarantees

- **Never edits** fenced code, inline `code`, URLs, markdown links, file paths,
  env vars, tables, or YAML frontmatter — these survive byte-for-byte.
- **Read-only by default.** `--apply` always writes a `.bak` backup.
- Meaning is preserved; only token-wasteful wording is trimmed.

## Waste codes

| Code | Severity | Meaning |
|------|----------|---------|
| `LARGE_FILE` | 🔴 | File exceeds the token budget (taxes every prompt) |
| `FILLER_HEAVY` | 🟡 | Many removable filler words |
| `VERBOSE_PHRASES` | 🟡 | Wordy phrases that have short equivalents |
| `DUPLICATE_LINES` | 🟡 | Repeated adjacent lines |
| `HUGE_CODE_BLOCK` | 🔵 | A code/log block dominates the file |
| `BLANK_BLOAT` | 🔵 | Runs of blank lines |

## Workflow for the agent

1. **Run** the audit on the user's project or file → read the report.
2. **Summarize** total % savings and the biggest-savings files.
3. **Offer** `--apply` (explain the `.bak` backups) before changing anything.
4. **For LARGE_FILE / HUGE_CODE_BLOCK**, suggest moving detail into `references/`
   loaded on demand rather than the always-loaded file.
5. **Re-run** to confirm the new token total.

See `references/methodology.md` for the estimator and `references/preservation.md`
for exactly what is never touched.