Token Formatter

Token Formatter Skill

Convert verbose documentation, markdown, and text files into token-efficient formats while preserving essential information.

When to Use

Invoke this skill when:

• User wants to reduce token count in documentation
• User needs to compress markdown/text for LLM context
• User asks to optimize content for AI consumption
• Commands: /token-format, /compress, /toon

Token Reduction Rules

1. Remove Redundancy

Before:

## Introduction

In this document, we will explore and discuss the various different ways
that you can implement authentication in your application. We will cover
multiple approaches and methods that are available to developers.

After:

## Auth Implementation

Methods covered:
- JWT tokens
- Session-based
- OAuth2

2. Use Symbols & Abbreviations

3. Compress Lists

Before:

The following features are included:
- User authentication with JWT tokens
- Role-based access control for authorization
- Password hashing using bcrypt algorithm
- Session management with Redis store
- Two-factor authentication support

After:

Features:
- JWT auth
- RBAC authz
- bcrypt passwords
- Redis sessions
- 2FA support

4. Remove Filler Words

Remove these words when possible:

• "basically", "essentially", "actually", "really"
• "in order to" → "to"
• "due to the fact that" → "because"
• "at this point in time" → "now"
• "in the event that" → "if"
• "for the purpose of" → "for"
• "has the ability to" → "can"
• "it is important to note that" → (remove)
• "please note that" → (remove)
• "as mentioned previously" → (remove)

5. Structured Data Format

Before (prose):

The API endpoint accepts three parameters. The first parameter is called
"userId" which is a required string that identifies the user. The second
parameter is "limit" which is an optional number that defaults to 10.
The third parameter is "offset" which is also optional and defaults to 0.

After (structured):

Params:
- userId: str (req) - user ID
- limit: num (opt, def=10)
- offset: num (opt, def=0)

6. Code Block Compression

Before:

// This function validates the user input
// It checks if the email is valid and the password meets requirements
function validateUserInput(email: string, password: string): boolean {
  // Check if email is valid
  const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
  const isEmailValid = emailRegex.test(email);

  // Check if password is at least 8 characters
  const isPasswordValid = password.length >= 8;

  // Return true if both are valid
  return isEmailValid && isPasswordValid;
}

After:

fn validateUserInput(email: str, password: str): bool {
  return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email) && password.length >= 8;
}

7. Table Compression

Before:

| Parameter Name | Type | Required | Default Value | Description |
|---------------|------|----------|---------------|-------------|
| userId | string | yes | none | The unique identifier for the user |
| limit | number | no | 10 | Maximum number of results to return |
| offset | number | no | 0 | Number of results to skip |

After:

| Param | Type | Req | Def | Desc |
|-------|------|-----|-----|------|
| userId | str | Y | - | user ID |
| limit | num | N | 10 | max results |
| offset | num | N | 0 | skip count |

8. Heading Compression

Before:

# Comprehensive Guide to User Authentication and Authorization

## Chapter 1: Introduction to Authentication Concepts

### Section 1.1: Understanding the Basics of User Identity

After:

# Auth Guide

## 1. Intro

### 1.1 Identity Basics

9. URL & Path Compression

Before:

The configuration file is located at `/Users/username/Documents/Projects/MyApp/config/settings.json`

After:

Config: `~/Projects/MyApp/config/settings.json`

10. Error Message Compression

Before:

Error: The authentication token that was provided in the request header
is either invalid, expired, or has been revoked. Please obtain a new
token by logging in again through the authentication endpoint.

After:

Err: Invalid/expired token. Re-authenticate at /auth/login

Compression Process

Step 1: Analyze Content

1. Identify content type (docs, code, config, etc.)
2. Find repetitive patterns
3. Locate verbose sections
4. Note essential vs non-essential info

Step 2: Apply Rules

1. Remove filler words
2. Apply abbreviations
3. Convert prose to structured data
4. Compress code blocks
5. Shorten headings

Step 3: Validate

1. Ensure meaning preserved
2. Check readability
3. Verify no critical info lost
4. Test comprehension

Output Format

# [Compressed Title]

## Summary
[1-2 line summary]

## Content
[Compressed content using rules above]

---
Original: X tokens | Compressed: Y tokens | Saved: Z%

Compression Levels

Level 1: Light (20-30% reduction)

• Remove filler words
• Shorten obvious terms
• Keep full sentences

Level 2: Medium (40-50% reduction)

• Apply abbreviations
• Use structured format
• Remove redundancy

Level 3: Heavy (60-70% reduction)

• Maximum compression
• Symbols over words
• Minimal prose

Examples

Example 1: API Documentation

Original (847 tokens):

# User Management API Documentation

## Introduction

Welcome to the User Management API documentation. This comprehensive guide
will walk you through all the available endpoints and explain how to use
them effectively in your application.

## Authentication

Before making any API requests, you need to authenticate your application.
Authentication is handled through JWT (JSON Web Tokens). To obtain a token,
send a POST request to the `/auth/login` endpoint with your credentials.

### Request Format

The request should be sent as JSON with the following structure:
{"email": "user@example.com", "password": "your_password"}

### Response Format

A successful authentication will return:
{"token": "eyJhbGciOiJIUzI1NiIs...", "expiresIn": 3600, "tokenType": "Bearer"}

Compressed (312 tokens):

# User API

## Auth

JWT-based. Get token via POST `/auth/login`

Req: {"email": "user@example.com", "password": "..."}
Res: {"token": "eyJ...", "expiresIn": 3600, "tokenType": "Bearer"}

---
Saved: 63%

Example 2: README

Original:

# My Awesome Project

## About This Project

This project was created to solve the problem of managing user data in
a scalable and efficient manner. It provides a comprehensive set of tools
and utilities that developers can use to build robust applications.

## Installation Instructions

To install this project, you will need to have Node.js version 18 or
higher installed on your system. Run: npm install && npm run dev

Compressed:

# My Project

Scalable user data management tools.

## Install

Requires: Node.js 18+
Run: npm install && npm run dev

Do NOT Compress

• Code syntax (keep valid)
• API keys/secrets (keep exact)
• Version numbers
• URLs/endpoints
• Error codes
• Mathematical formulas
• Legal text
• Security-critical info

Automation

Two Python scripts are available in the scripts/ directory to help automate the compression and validation process.

Prerequisites

For accurate token counting (optional but recommended), install tiktoken:

pip install tiktoken

(If not installed, scripts will fallback to character count estimation).

1. Compression Script (compress.py)

Automatically applies common compression rules (filler removal, abbreviations, symbols).

Usage:

# Compress a file
python .claude/skills/token-formatter/scripts/compress.py input.md > compressed.md

# Compress input from pipe
cat input.md | python .claude/skills/token-formatter/scripts/compress.py --level 2

# Show stats
python .claude/skills/token-formatter/scripts/compress.py input.md --stats

2. Token Counter (count_tokens.py)

Counts tokens in files or text input.

Usage:

# Count tokens in files
python .claude/skills/token-formatter/scripts/count_tokens.py document.md README.md

# Count tokens from pipe
echo "Hello World" | python .claude/skills/token-formatter/scripts/count_tokens.py

TOON Format (Data Serialization → TOON for LLM Input)

TOON replaces data serialization formats when sending to LLMs. ~40% fewer tokens.

Formats to convert → TOON:

• JSON / JSON compact
• YAML
• XML

Do NOT convert to TOON:

• Markdown - keep as markdown
• Plain text - keep as text
• Code files - keep original syntax
• CSV - already compact for flat tables

TOON sweet spot:

Uniform arrays of objects (same fields per item) from JSON/YAML/XML.

JSON:

{"users":[{"id":1,"name":"John","role":"admin"},{"id":2,"name":"Jane","role":"user"}]}

TOON:

users[2]{id,name,role}:
  1,John,admin
  2,Jane,user

When to ask:

"This contains JSON/YAML/XML data for LLM input. Convert to TOON for ~40% token savings?"

See: .claude/skills/document-skills/toon/SKILL.md

Quick Reference Card

Abbreviations

fn=function  ret=return  str=string  num=number
bool=boolean arr=array   obj=object  param=parameter
config=configuration     env=environment
auth=authentication      authz=authorization
db=database  repo=repository  dir=directory
req=required opt=optional def=default
max=maximum  min=minimum  ex=example
impl=implementation      docs=documentation

Symbols

&   = and           |   = or
w/  = with          w/o = without
=>  = therefore     ~   = approximately
Y   = yes           N   = no
-   = none/null