TokenCount
A real text and token counter for the terminal. Count words, lines, characters, and sentences. Estimate LLM token usage with cost projections. Analyze word frequency and compare files side by side.
Commands
| Command |
Description |
tokencount count <file|text> |
Count words, lines, characters, sentences, paragraphs, avg word length, and reading time. Works with files or inline text |
tokencount tokens <file> |
Estimate LLM token count using 3 methods (chars÷4, words×1.33, bytes÷3.5), shows cost estimates for GPT-4 class models and context window usage bars |
tokencount freq <file> |
Full word frequency analysis — ranked table with counts, percentages, bar chart, and vocabulary richness score |
tokencount top <file> [n] |
Show top N most common words (default: 20) |
tokencount diff <file1> <file2> |
Compare two files — side-by-side word/line/char/token counts, unique words in each, common vocabulary |
Requirements
- Standard Unix tools:
wc, awk, sort, tr, grep, comm
Examples
# Count everything in a file
tokencount count README.md
# Count inline text
tokencount count "Hello world, this is a test."
# Estimate tokens and costs
tokencount tokens article.txt
# Word frequency analysis
tokencount freq novel.txt
# Top 10 words
tokencount top essay.txt 10
# Compare two drafts
tokencount diff draft-v1.txt draft-v2.txt
1---2name: tokencount3description: Count words, characters, and estimate GPT tokens with readability. Use when tracking length, checking budgets, comparing complexity.4---5
6# TokenCount
7
8A real text and token counter for the terminal. Count words, lines, characters, and sentences. Estimate LLM token usage with cost projections. Analyze word frequency and compare files side by side.
9
10## Commands
11
12| Command | Description |
13|---------|-------------|
14| `tokencount count <file\|text>` | Count words, lines, characters, sentences, paragraphs, avg word length, and reading time. Works with files or inline text |
15| `tokencount tokens <file>` | Estimate LLM token count using 3 methods (chars÷4, words×1.33, bytes÷3.5), shows cost estimates for GPT-4 class models and context window usage bars |
16| `tokencount freq <file>` | Full word frequency analysis — ranked table with counts, percentages, bar chart, and vocabulary richness score |
17| `tokencount top <file> [n]` | Show top N most common words (default: 20) |
18| `tokencount diff <file1> <file2>` | Compare two files — side-by-side word/line/char/token counts, unique words in each, common vocabulary |
19
20## Requirements
21
22- Standard Unix tools: `wc`, `awk`, `sort`, `tr`, `grep`, `comm`
23
24## Examples
25
26```bash
27# Count everything in a file
28tokencount count README.md
29
30# Count inline text
31tokencount count "Hello world, this is a test."
32
33# Estimate tokens and costs
34tokencount tokens article.txt
35
36# Word frequency analysis
37tokencount freq novel.txt
38
39# Top 10 words
40tokencount top essay.txt 10
41
42# Compare two drafts
43tokencount diff draft-v1.txt draft-v2.txt
44```
bytesagain