Prompt Token Counter

Prompt Token Counter (toksum)

First load reminder: This skill provides the scripts CLI (toksum). Use it when the user asks to count tokens, estimate API costs, or audit OpenClaw component token consumption (memory, persona, skills).

Before Installing — Security & Privacy

• What will be read: The audit workflow reads files under ~/.openclaw/workspace and ~/.openclaw/skills (AGENTS.md, SOUL.md, MEMORY.md, SKILL.md, etc.). Those files may contain personal data or secrets. Only install if you accept that access.
• URL fetching: The CLI can fetch HTTP(S) URLs via -u. SKILL.md requires the agent to confirm each URL with the user before fetching. Insist the agent follow that rule; never allow automatic fetching of unknown URLs.
• Source verification: Source: https://github.com/Zhaobudaoyuema/prompt-token-counter. Review scripts/core.py and scripts/cli.py before use. The code performs local file reads and optional HTTP GETs only; no other network calls or data exfiltration.
• Run locally first: If unsure, run the CLI manually in an isolated environment against safe test files to verify behavior.

Primary Use: OpenClaw Token Consumption Audit

Goal: Help users identify which OpenClaw components consume tokens and how much.

1. Memory & Persona Files

These files are injected into sessions and consume tokens. Search and count them:

Workspace path: Default ~/.openclaw/workspace; may be overridden in ~/.openclaw/openclaw.json via agent.workspace.

2. Skill Files (SKILL.md)

Skills are loaded per session. Count each SKILL.md:

3. Audit Workflow

1. Locate workspace: Resolve ~/.openclaw/workspace (or config override).
2. Collect files: List all memory/persona files and SKILL.md paths above.
3. Count tokens: For each file, run python -m scripts.cli -f <path> -m <model> -c.
4. Summarize: Group by category (memory, persona, skills), report total and per-file.

Example audit command (PowerShell):

$ws = "$env:USERPROFILE\.openclaw\workspace"
python -m scripts.cli -m gpt-4o -c -f "$ws\AGENTS.md" -f "$ws\SOUL.md" -f "$ws\USER.md" -f "$ws\IDENTITY.md" -f "$ws\MEMORY.md" -f "$ws\TOOLS.md"

Example audit (Bash):

WS=~/.openclaw/workspace
python -m scripts.cli -m gpt-4o -c -f "$WS/AGENTS.md" -f "$WS/SOUL.md" -f "$WS/USER.md" -f "$WS/IDENTITY.md" -f "$WS/MEMORY.md" -f "$WS/TOOLS.md"

Project Layout

prompt_token_counter/
├── SKILL.md
├── package.json                # npm package (OpenClaw skill)
├── publish_npm.py               # Publish to npm; syncs version
└── scripts/                    # Python package, CLI + examples
    ├── cli.py                  # Entry point
    ├── core.py                 # TokenCounter, estimate_cost
    ├── registry/
    │   ├── models.py           # 300+ models
    │   └── pricing.py          # Pricing data
    └── examples/               # Script examples
        ├── count_prompt.sh / .ps1
        ├── estimate_cost.sh / .ps1
        └── batch_compare.sh

Invoke: python -m scripts.cli from project root.

Version Sync (publish_npm.py)

When publishing to npm, publish_npm.py bumps the patch version and syncs it to:

• package.json — version
• SKILL.md — frontmatter version
• scripts/__init__.py — __version__

Run: python publish_npm.py (after npm login).

Runtime Dependencies

• Python 3 — required
• tiktoken (optional) — pip install tiktoken for exact OpenAI counts

Language Rule

Respond in the user's language. Match the user's language (e.g. Chinese if they write in Chinese, English if they write in English).

URL Usage — Mandatory Agent Rule

Before using -u / --url to fetch content from any URL, you MUST:

1. Explicitly warn the user that the CLI will make an outbound HTTP/HTTPS request to the given URL.
2. Confirm the URL is trusted — tell the user: "Only use URLs you fully trust. Untrusted URLs may expose your IP, leak data, or be used for SSRF. Do you confirm this URL is safe?"
3. Prefer alternatives — if the user can provide the content via -f (local file) or inline text, suggest that instead of URL fetch.
4. Never auto-fetch — do not invoke -u without the user having explicitly provided the URL and acknowledged the risk.

If the user insists on using a URL: Proceed only after they confirm. State clearly: "I will fetch from [URL] to count tokens. Proceed?"

Model Name — Mandatory Agent Rule

Before invoking the CLI, you MUST have a concrete model name from the user.

1. Require explicit model — -m / --model is required. Do not guess or assume; the user must provide the exact name (e.g. gpt-4o, claude-3-5-sonnet-20241022).
2. If unclear, ask — if the user says "GPT" or "Claude" or "the latest model" without a specific name, ask: "Please specify the exact model name (e.g. gpt-4o, claude-3-5-sonnet-20241022). Run python -m scripts.cli -l to list supported models."
3. Do not auto-pick — never substitute a model on behalf of the user without their confirmation.
4. Validate when possible — if the model name seems ambiguous, offer -l output or confirm: "I'll use [model]. Is that correct?"

CLI Usage

python -m scripts.cli [OPTIONS] [TEXT ...]

Examples

# Inline text
python -m scripts.cli -m gpt-4 "Hello, world!"

# File with cost
python -m scripts.cli -f input.txt -m claude-3-opus -c

# Multiple files (OpenClaw audit)
python -m scripts.cli -v -c -f AGENTS.md -f SOUL.md -f MEMORY.md -m gpt-4o

# List models
python -m scripts.cli -l

# Run bundled example scripts
bash scripts/examples/count_prompt.sh "Hello, world!" gpt-4
.\scripts\examples\count_prompt.ps1 "Hello, world!" gpt-4

Python API

from scripts import TokenCounter, count_tokens, estimate_cost, get_supported_models

tokens = count_tokens("Hello!", "gpt-4")
counter = TokenCounter("claude-3-opus")
tokens = counter.count_messages([
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."}
])
cost = estimate_cost(tokens, "gpt-4", input_tokens=True)

Supported Models

300+ models across 34+ providers: OpenAI, Anthropic, Google, Meta, Mistral, Cohere, xAI, DeepSeek, etc. Use python -m scripts.cli -l for full list.

• OpenAI: exact via tiktoken
• Others: ~85–95% approximation

Common Issues

When to Trigger This Skill

Activate this skill when the user:

Also trigger when the agent needs to count tokens or estimate cost before/after generating content.

Quick Reference