560 2026 02 01 Token Counting Logic 44b0e408

Research – Token Counting Logic

Date: 2026-02-01 Phase: Research

Structure

• src/tunacode/utils/messaging/ (adapter + token_counter)
• src/tunacode/types/canonical.py (canonical message + usage types)
• src/tunacode/core/state.py (session token count computation + persistence)
• src/tunacode/core/types/state_structures.py (ConversationState.total_tokens)
• src/tunacode/core/agents/main.py (token count refresh after message changes)
• src/tunacode/core/agents/resume/ (prune/summary/filter using heuristic tokens)
• src/tunacode/core/agents/agent_components/orchestrator/ (provider usage tokens)
• src/tunacode/ui/app.py (resource bar + response panel token display)
• src/tunacode/ui/commands/__init__.py (clear command preserves total_tokens)

Key Files

• src/tunacode/utils/messaging/token_counter.py:12-129 defines heuristic constants (CHARS_PER_TOKEN=4), MessageTokenCacheEntry, MessageTokenCache, estimate_tokens, estimate_tokens_from_length, get_message_text_length.
  • MessageTokenCache.rebuild_total() (line 34) iterates messages, uses id(message) as the cache key, calls _get_or_update, sums token counts, then _prune.
  • _get_or_update() (line 52) calls get_message_text_length() then estimate_tokens_from_length().
  • _prune() (line 65) removes cache entries not in active ids.
  • estimate_tokens() (line 73) uses len(text) then estimate_tokens_from_length().
  • get_message_text_length() (line 99) canonicalizes and computes part length.
• src/tunacode/utils/messaging/adapter.py:57-205 defines _get_parts() and to_canonical() used by token_counter to normalize messages into CanonicalMessage.
• src/tunacode/types/canonical.py:104-133 defines CanonicalMessage and get_text_content(); UsageMetrics at lines 184-214; NormalizedUsage at lines 260-289.
• src/tunacode/core/state.py:65-75 SessionState.update_token_count() computes conversation.total_tokens via MessageTokenCache.rebuild_total. adjust_token_count() is defined at lines 72-75. Session persistence stores total_tokens at lines 273 and loads at lines 321-322.
• src/tunacode/core/types/state_structures.py:24-33 ConversationState.total_tokens and max_tokens fields.
• src/tunacode/core/agents/main.py:222,303,350,583,590,594,598 call session.update_token_count() after message list changes and cleanup.
• src/tunacode/core/agents/resume/prune.py:100-224 uses estimate_tokens to compute token counts for tool return parts and pruning thresholds.
• src/tunacode/core/agents/resume/summary.py:97-201 uses estimate_tokens to compute message totals for compaction and summary token_count.
• src/tunacode/core/agents/resume/filter.py:52-76 prepare_history() combines summary filtering and pruning and returns tokens_reclaimed.
• src/tunacode/core/agents/agent_components/orchestrator/usage_tracker.py:15-53 update_usage() reads provider usage via normalize_request_usage() and updates UsageMetrics.
• src/tunacode/core/agents/agent_components/orchestrator/orchestrator.py:217-258 calls update_usage() on model_response.
• src/tunacode/ui/app.py:217-236 uses session.usage.last_call_usage.completion_tokens for response metadata; src/tunacode/ui/app.py:365-376 uses conversation.total_tokens for the resource bar.
• src/tunacode/ui/commands/__init__.py:77-105 clear command preserves conversation.total_tokens and resets usage metrics.

Patterns Found

• Conversation token total (heuristic):
  • SessionState.update_token_count() → MessageTokenCache.rebuild_total() → _get_or_update() → get_message_text_length() → _canonicalize_message() → to_canonical() → _canonical_text_length(); then estimate_tokens_from_length() divides character length by CHARS_PER_TOKEN.
  • Cache keys are id(message) and _prune() removes entries not in the current message list.
• Heuristic token estimates in resume/compaction:
  • estimate_tokens() is called in resume/prune.py (tool-output pruning) and resume/summary.py (summary threshold + summary token_count).
• Provider usage tokens:
  • orchestrator.update_usage() reads model_response.usage, normalizes it via normalize_request_usage(), and updates session.usage.last_call_usage and session.usage.session_total_usage.
• Display:
  • Resource bar reads conversation.total_tokens (heuristic count).
  • Response panels read session.usage.last_call_usage.completion_tokens (provider usage).

Dependencies

• core/state.py imports MessageTokenCache from tunacode.utils.messaging.
• utils/messaging/token_counter.py imports CanonicalMessage and to_canonical.
• core/agents/resume/prune.py and core/agents/resume/summary.py import estimate_tokens.
• core/agents/agent_components/orchestrator/usage_tracker.py imports UsageMetrics and normalize_request_usage from types/canonical.py.
• UI modules read conversation.total_tokens and session.usage.last_call_usage.

Symbol Index

• tunacode.utils.messaging.token_counter exports MessageTokenCache, estimate_tokens, get_message_text_length (utils/messaging/__init__.py:16-20).
• tunacode.core.agents.resume exports prune_old_tool_outputs, filter_compacted, prepare_history, SummaryMessage, should_compact, generate_summary (core/agents/resume/__init__.py:15-41).
• tunacode.core.agents.agent_components.orchestrator.usage_tracker exports update_usage (usage_tracker.py:15).
• tunacode.types.canonical exports CanonicalMessage, UsageMetrics, NormalizedUsage, normalize_request_usage (types/canonical.py:104-319).