Token Optimizer

Token Optimizer

Reduce tokens in prompts, docs, and prose while preserving meaning.

When to Use

• Prompt >1500 tokens or contains redundant phrasing
• Docs/markdown need compression for LLM context
• Writing prose for humans (docs, commits, PRs, error messages)
• Converting JSON/YAML/XML data for LLM input (TOON)

Quick Routing

Compression Levels

1. Prompt Compression

Process

1. Identify core intent
2. Extract essential context only
3. Remove filler phrases
4. Apply abbreviations & symbols
5. Output compressed version w/ token savings %

Remove Phrases

Transform Patterns

Example

Before (847 tokens):

Hello! I hope you're doing well. I was wondering if you could help me.
I'm trying to build a React app and I need a custom hook that fetches
user data from /api/users. It returns JSON. I'd like loading, error
states, and caching. I think useEffect and useState but not sure...

After (156 tokens):

Goal: React hook for user data fetching
- Endpoint: /api/users -> JSON user obj
- Handle: loading, error states
- Cache response
Stack: React (useEffect, useState)

2. Doc Formatting

Remove Filler Words

Remove when possible: "basically", "essentially", "actually", "really"

Compress Lists

Before:

The following features are included:
- User authentication with JWT tokens
- Role-based access control for authorization
- Password hashing using bcrypt algorithm

After:

Features:
- JWT auth
- RBAC authz
- bcrypt passwords

Structured Data Over Prose

Before:

The API accepts three parameters. The first is "userId" which is a
required string. The second is "limit", optional number, defaults to 10.

After:

Params:
- userId: str (req) - user ID
- limit: num (opt, def=10)

Heading Compression

Before: # Comprehensive Guide to User Authentication and Authorization After: # Auth Guide

Table Compression

Shorten column headers, use abbreviations in cells:

Code Block Compression

Remove comments that restate the code. Collapse trivial logic:

Before:

// Check if email is valid
const isEmailValid = emailRegex.test(email);
// Check if password is at least 8 characters
const isPasswordValid = password.length >= 8;
// Return true if both are valid
return isEmailValid && isPasswordValid;

After:

return emailRegex.test(email) && password.length >= 8;

URL & Path Compression

Replace full home paths: /Users/username/Documents/Projects/MyApp/config/settings.json -> ~/Projects/MyApp/config/settings.json

Error Message Compression

Keep error codes and stack traces exact. Compress surrounding prose: Err: Invalid/expired token. Re-authenticate at /auth/login

3. Prose Clarity (Strunk's Rules)

Apply when writing prose for humans (docs, commits, PRs, errors).

Grammar Essentials

1. Form possessive singular by adding 's
2. Use comma after each term in series except last
3. Don't join independent clauses by comma (use semicolon or period)
4. Participial phrase at beginning refers to grammatical subject

Composition Essentials

1. One paragraph per topic
2. Begin paragraph with topic sentence
3. Express coordinate ideas in similar form
4. Keep related words together
5. Place emphatic words at end of sentence

Limited Context Strategy

When context is tight: write draft, dispatch subagent with draft + these rules, have subagent copyedit and return revision.

Core Principles

Omit needless words:

Use active voice:

Positive form over negative:

Specific over vague:

Technical Writing

# Commit: Bad
Made some changes to fix the bug that was causing issues

# Commit: Good
Fix null pointer in user authentication

# Error: Bad
An error occurred while processing your request

# Error: Good
Database connection failed: timeout after 30s

# Doc: Bad
This function is used for the purpose of validating user input

# Doc: Good
Validates user input

Editing Checklist

• Unnecessary words (especially "that", "very", "really", "just")
• Passive voice -> convert to active
• Negative statements -> make positive
• Vague language -> make specific
• Long sentences -> break up or simplify

4. TOON Format (Data Serialization)

TOON replaces JSON/YAML/XML data serialization when sending to LLMs. ~40% fewer tokens.

Convert to TOON: JSON, YAML, XML (data objects/arrays only) Keep as-is: Markdown, plain text, code files, CSV

Precedence note: TOON converts the data format structure (braces, quotes, colons). The "Never Compress" rules protect values inside data (API keys, URLs, version numbers). Both apply: convert structure to TOON, but preserve exact values.

JSON:

{"users":[{"id":1,"name":"John","role":"admin"},{"id":2,"name":"Jane","role":"user"}]}

TOON:

users[2]{id,name,role}:
  1,John,admin
  2,Jane,user

See: .claude/skills/document-skills/toon/SKILL.md

Never Compress

See references/never_compress.md for full list:

• Auth tokens, API keys, credentials, secrets
• Error stack traces (keep full)
• Code blocks, inline code, regex
• URLs, UUIDs, version numbers
• SQL queries, shell commands
• JSON keys, config values, date formats
• Legal text, mathematical formulas

Abbreviations

fn=function  ret=return  str=string  num=number
bool=boolean arr=array   obj=object  param=parameter
config=configuration     env=environment
auth=authentication      authz=authorization
db=database  repo=repository  dir=directory
req=required opt=optional def=default
max=maximum  min=minimum  ex=example
impl=implementation      docs=documentation
app=application          info=information

Symbols

-> = returns/produces    & = and
|  = or                  w/ = with
w/o = without            ~ = approximately
=> = therefore           bc = because
Y  = yes                 N  = no
-  = none/null

Output Format

## Compressed

[Content]

---
Original: X tokens | Compressed: Y tokens | Saved: Z%

Scripts

# Compress text
python scripts/compress.py input.md > compressed.md
python scripts/compress.py input.md --stats
python scripts/compress.py --level 2 < input.md

# Count tokens
python scripts/count_tokens.py document.md

# Compress prompt
python scripts/compress_prompt.py "your prompt text"
python scripts/compress_prompt.py --file prompt.txt --level heavy

Integration

Called by: brainstorm (Phase 5 docs), any skill producing documentation Pairs with: document-skills/toon (data serialization)