2692 Task 13 Otel Session Statistics 2a91ef62

TASK-13: OpenTelemetry Session Statistics

Status: ✅ Completed Version: 3.1.0 Created: 2025-10-20 Completed: 2025-10-20 Type: Enhancement (Minor Version)

Executive Summary

Goal: Replace hacky file-size estimation script with official OpenTelemetry metrics for real-time session statistics.

Impact:

• Real token usage (not estimates)
• Live session metrics (active time, cost, cache performance)
• Official API (won't break on Claude Code updates)
• Foundation for ROI measurement

Dependencies: Claude Code OpenTelemetry support (available now)

Context

Current Implementation (Hacky)

File: skills/nav-start/scripts/session_stats.py

# Rough estimation from file size
tokens = file_bytes // 4

# Static calculation
total_tokens = navigator["tokens"] + claude_md["tokens"]
available = 200000 - 50000 - total_tokens

Problems:

• ❌ Estimates, not real usage
• ❌ Static calculation (ignores actual session)
• ❌ No cache awareness
• ❌ No cost tracking
• ❌ No active time measurement

Official OpenTelemetry Support (New)

Claude Code now exports:

// Real metrics
claude_code.token.usage {
  type: "input" | "output" | "cacheRead" | "cacheCreation"
  model: "claude-sonnet-4-5-20250929"
}

claude_code.cost.usage {
  model: "claude-sonnet-4-5-20250929"
}

claude_code.active_time.total {
  // Actual seconds of active work
}

Benefits:

• ✅ Real data from Claude Code internals
• ✅ Cache-aware (shows free cache reads)
• ✅ Cost tracking (actual USD spent)
• ✅ Active time (not idle time)
• ✅ Official API (stable)

Implementation Plan

Phase 1: Research OTel Integration

Goal: Understand how to query OTel metrics from Python

Options:

1. OpenTelemetry Python SDK - Query metrics programmatically
2. Parse console output - Read OTEL_METRICS_EXPORTER=console stderr
3. Query OTLP endpoint - HTTP/gRPC to collector

Decision: Start with console parsing (simplest), add SDK support later

Tasks:

• Read Claude Code OTel documentation
• Test console exporter output format
• Determine which metrics are available during session
• Design parsing strategy

Phase 2: Implement OTel Session Stats Script

Goal: Replace session_stats.py with OTel-powered version

New File: skills/nav-start/scripts/otel_session_stats.py

Features:

#!/usr/bin/env python3
"""
Navigator Session Statistics (OpenTelemetry-powered)

Queries real token usage from Claude Code OpenTelemetry metrics.
Requires CLAUDE_CODE_ENABLE_TELEMETRY=1
"""

def check_otel_enabled():
    """Verify OTel is configured"""
    if os.getenv("CLAUDE_CODE_ENABLE_TELEMETRY") != "1":
        print("❌ OpenTelemetry not enabled")
        print("Enable with: export CLAUDE_CODE_ENABLE_TELEMETRY=1")
        sys.exit(1)

def query_session_metrics():
    """
    Query current session metrics from OTel.

    Returns:
    {
        "input_tokens": int,
        "output_tokens": int,
        "cache_read_tokens": int,  # Free!
        "cache_creation_tokens": int,
        "cost_usd": float,
        "active_time_seconds": int,
        "model": str
    }
    """
    # Implementation: Parse OTel console output or query OTLP endpoint

def display_navigator_stats(metrics):
    """
    Display Navigator-optimized session statistics.

    Shows:
    - Total tokens used (with cache breakdown)
    - Cache hit rate (% tokens from cache)
    - Cost analysis
    - Active time efficiency
    - Available context remaining
    """

    print("📊 Navigator Session Statistics (Real-time via OTel)")
    print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")

    # Token usage breakdown
    total_input = metrics["input_tokens"]
    cache_read = metrics["cache_read_tokens"]
    fresh_input = total_input - cache_read

    print(f"\n📥 Input Tokens:  {total_input:,}")
    print(f"   ├─ Cache read:  {cache_read:,} (free ✅)")
    print(f"   └─ Fresh:       {fresh_input:,} (charged)")

    print(f"\n📤 Output Tokens: {metrics['output_tokens']:,}")

    # Cache performance
    if total_input > 0:
        cache_hit_rate = (cache_read / total_input) * 100
        print(f"\n⚡ Cache Hit Rate: {cache_hit_rate:.1f}%")

    # Cost analysis
    print(f"\n💰 Session Cost:  ${metrics['cost_usd']:.4f}")

    # Active time
    minutes = metrics['active_time_seconds'] // 60
    seconds = metrics['active_time_seconds'] % 60
    print(f"\n⏱️  Active Time:   {minutes}m {seconds}s")

    # Context availability
    context_used = total_input + metrics['output_tokens']
    total_context = 200000
    available = total_context - context_used
    percent_available = int((available / total_context) * 100)

    print(f"\n📦 Context Usage:")
    print(f"   ├─ Used:        {context_used:,} tokens")
    print(f"   └─ Available:   {available:,} tokens ({percent_available}%)")

    print(f"\n🤖 Model:         {metrics['model']}")
    print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")

Implementation Steps:

1. Create new otel_session_stats.py
2. Implement OTel detection
3. Implement metric querying (console parser first)
4. Format output for Navigator context
5. Add error handling (OTel not enabled, no metrics yet)

Phase 3: Update nav-start Skill

Goal: Integrate OTel stats into session start

File: skills/nav-start/skill.md

Changes:

## Predefined Functions

### scripts/otel_session_stats.py

**Purpose**: Display real-time session statistics via OpenTelemetry

**When to call**: After loading navigator, before presenting session info

**Requirements**:
- CLAUDE_CODE_ENABLE_TELEMETRY=1 must be set
- Metrics must be available from current session

**Execution**:
```bash
python3 .agent/.nav-scripts/otel_session_stats.py

Output: Formatted statistics with token usage, cache performance, cost

Error Handling:

• If OTel not enabled: Show setup instructions
• If no metrics yet: Skip stats (early in session)

**Update skill workflow**:
1. Load DEVELOPMENT-README.md
2. Check for PM tool assignments
3. **[NEW] Display OTel session stats**
4. Present navigator index

### Phase 4: Documentation Updates

**Files to update**:

#### 1. `.agent/DEVELOPMENT-README.md`
```markdown
## 📊 Session Statistics (New in v3.1)

Navigator now uses OpenTelemetry for real-time session metrics.

**Setup** (one-time):
```bash
# Enable Claude Code telemetry
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console

What you get:

• Real token usage (not estimates)
• Cache hit rates (CLAUDE.md caching performance)
• Session costs (actual USD spent)
• Active time tracking

See: .agent/sops/integrations/opentelemetry-setup.md

#### 2. Create `.agent/sops/integrations/opentelemetry-setup.md`

**New SOP for OTel configuration**:
```markdown
# OpenTelemetry Setup for Navigator

## Quick Start

**Enable telemetry**:
```bash
# Add to ~/.zshrc or ~/.bashrc
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console

# Optional: Reduce export interval for development
export OTEL_METRIC_EXPORT_INTERVAL=10000  # 10 seconds

Restart shell:

source ~/.zshrc

What Navigator Tracks

Metrics used by Navigator:

• claude_code.token.usage - Input/output/cache breakdown
• claude_code.cost.usage - Session costs in USD
• claude_code.active_time.total - Active work time

Benefits:

• Prove Navigator ROI with real data
• Monitor cache performance (CLAUDE.md efficiency)
• Track session costs
• Measure productivity (LOC per token)

Advanced Configuration

For teams wanting centralized monitoring:

# Use OTLP exporter to send to Prometheus/Grafana
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

See: Official Claude Code docs for full OTel setup

#### 3. Update `CLAUDE.md` template

**Add OTel section**:
```markdown
## Session Statistics (Optional)

For real-time token tracking, enable OpenTelemetry:

```bash
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console

Navigator will show cache performance and session costs.

#### 4. Update `README.md`

**Add to features**:
```markdown
### 📊 Real-Time Session Statistics (v3.1+)

- **Official OTel integration** - Real metrics from Claude Code
- **Cache performance** - See CLAUDE.md caching in action
- **Cost tracking** - Monitor session spend
- **Active time** - Measure productivity

Phase 5: Testing

Test Scenarios:

Test 1: OTel Disabled (Graceful Handling)

# Unset OTel
unset CLAUDE_CODE_ENABLE_TELEMETRY

# Start session
"Start my Navigator session"

# Expected: Script shows setup instructions, doesn't crash

Test 2: OTel Enabled, Console Exporter

# Enable OTel
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console
export OTEL_METRIC_EXPORT_INTERVAL=10000

# Start new session
"Start my Navigator session"

# Expected: Real metrics displayed
# - Input tokens (with cache breakdown)
# - Output tokens
# - Cache hit rate
# - Cost
# - Active time

Test 3: Early Session (No Metrics Yet)

# Very first message after `claude` starts
"Start my Navigator session"

# Expected: Graceful handling if no metrics exported yet

Test 4: Cache Performance Validation

# Session 1: Load CLAUDE.md (creates cache)
"Start my Navigator session"
# Note cache_creation_tokens

# Session 2: Load CLAUDE.md again (uses cache)
"Start my Navigator session"
# Note cache_read_tokens (should be high)

# Expected: cache_read_tokens > 0 in session 2

Test 5: Cost Tracking Accuracy

# Enable OTel
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console

# Run multiple interactions
"Start session"
"Write a hello world function"
"Create tests"

# Check cost accumulation
python3 .agent/.nav-scripts/otel_session_stats.py

# Expected: Cost increases with each interaction

File Changes Summary

New Files

skills/nav-start/scripts/otel_session_stats.py  (replaces session_stats.py)
.agent/sops/integrations/opentelemetry-setup.md  (new SOP)

Modified Files

skills/nav-start/skill.md  (add OTel stats call)
.agent/DEVELOPMENT-README.md  (add session statistics section)
templates/CLAUDE.md  (add OTel optional section)
README.md  (add v3.1 feature)
scripts/post-install.sh  (auto-enable OTel on plugin install/update)

Deleted Files

skills/nav-start/scripts/session_stats.py  (replaced by otel_session_stats.py)

Auto-Enablement on Plugin Update

scripts/post-install.sh now includes OTel setup:

• Prompts user to enable OpenTelemetry after install/update
• Auto-detects shell config (.zshrc or .bashrc)
• Checks if already configured (doesn't duplicate)
• Adds environment variables automatically
• Provides manual instructions as fallback

User experience on upgrade to v3.1:

/plugin update navigator

# Post-install hook runs:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 Navigator v3.1 - OpenTelemetry Integration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Navigator can show real-time session statistics:
  • Real token usage (input/output/cache)
  • Cache hit rates (CLAUDE.md performance)
  • Session costs (actual USD spent)
  • Active time tracking

Enable OpenTelemetry? [Y/n]

# User types Y:
✅ Added OpenTelemetry configuration to .zshrc

⚠️  Restart your terminal or run:
   source ~/.zshrc

Result: Zero-config upgrade - telemetry works automatically after terminal restart

Technical Implementation Details

OTel Metrics Query Strategy

Phase 1: Console Parser (v3.1.0)

def parse_console_metrics():
    """
    Parse metrics from OTEL_METRICS_EXPORTER=console output.

    Console exporter writes to stderr in format:
    {
      "resource": {...},
      "scope_metrics": [
        {
          "metrics": [
            {
              "name": "claude_code.token.usage",
              "data": {
                "data_points": [...]
              }
            }
          ]
        }
      ]
    }
    """
    # Read from stderr/process output
    # Parse JSON metrics
    # Extract relevant counters

Challenges:

• Console output is JSON-formatted logs
• Need to parse most recent metrics
• Handle case where no exports yet

Phase 2: SDK Integration (Future v3.2+)

from opentelemetry import metrics

def query_via_sdk():
    """Query metrics programmatically via OTel SDK"""
    # Use meter provider to get current metrics
    # More reliable than parsing console output

Metric Aggregation

Claude Code exports cumulative counters:

// Metrics are session-total, not per-request
claude_code.token.usage: 15000  // Total since session start

Our script shows:

• Session totals (from counters)
• Derived metrics (cache hit rate, context remaining)
• No need to track deltas (session-level stats)

Error Handling

Scenarios to handle:

1. OTel not enabled

   if not check_otel_enabled():
       print("OpenTelemetry not enabled")
       print("See: .agent/sops/integrations/opentelemetry-setup.md")
       sys.exit(0)  # Exit gracefully, don't crash nav-start
   

2. No metrics exported yet

   metrics = query_session_metrics()
   if not metrics:
       print("No metrics available yet (early in session)")
       sys.exit(0)
   

3. Parse errors

   try:
       data = json.loads(console_output)
   except json.JSONDecodeError:
       print("Could not parse OTel output")
       sys.exit(0)
   

Success Metrics

Functionality

• OTel stats script works with CLAUDE_CODE_ENABLE_TELEMETRY=1
• Graceful handling when OTel disabled
• Cache performance metrics accurate
• Cost tracking matches Claude Console
• Active time displayed correctly

Integration

• nav-start skill calls OTel stats automatically
• Stats display before navigator index
• No errors when OTel unavailable
• Documentation clear and accurate

User Experience

• Real metrics more useful than estimates
• Cache hit rate validates Navigator efficiency
• Cost tracking enables ROI measurement
• Setup instructions work (one-time config)

Risks & Mitigation

Risk 1: OTel Output Format Changes

Problem: Console exporter format could change Impact: Parser breaks on Claude Code updates Mitigation:

• Use stable OTel spec (unlikely to change)
• Add version detection
• Fall back to "metrics unavailable" on parse errors
• Future: Move to SDK instead of parsing

Likelihood: Low Severity: Low (graceful degradation)

Risk 2: Metrics Not Available Early in Session

Problem: OTel exports every 60s (default), might not have data yet Impact: Empty stats on first /nav:start Mitigation:

• Handle empty metrics gracefully
• Show message: "Metrics not available yet"
• Recommend shorter export interval for dev

Likelihood: Medium Severity: Low (just wait for export)

Risk 3: Users Don't Enable OTel

Problem: Most users won't configure OTel Impact: Stats feature unused Mitigation:

• Make it optional (graceful when disabled)
• Clear setup instructions
• Show value prop (ROI measurement)
• Consider auto-detecting and prompting

Likelihood: High Severity: Low (optional feature)

Future Enhancements (Post-v3.1)

v3.2: ROI Dashboard

# skills/navigator-roi/
Generate Grafana dashboard from OTel metrics:
- Token savings (Navigator vs standard sessions)
- Cache performance over time
- Cost per feature delivered
- Productivity metrics (LOC per token)

v3.3: Team Analytics

# Multi-user tracking
export OTEL_RESOURCE_ATTRIBUTES="team=engineering,user=alice"

# Aggregate metrics across team
navigator-analytics --team engineering --period week

v3.4: SDK Integration

# Replace console parsing with SDK
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
counter = meter.get_counter("claude_code.token.usage")

Version Sync Checklist

Update version to 3.1.0 in:

• .claude-plugin/plugin.json → "version": "3.1.0"
• .claude-plugin/marketplace.json → "version": "3.1.0"
• .agent/.nav-config.json → "version": "3.1.0"
• README.md line ~5 → v3.1.0
• README.md line ~8 (badge) → 3.1.0
• README.md footer → 3.1.0
• .agent/DEVELOPMENT-README.md bottom → (v3.1.0)
• Git tag → v3.1.0
• GitHub release → v3.1.0

Run audit script:

./scripts/version-audit.sh 3.1.0

Timeline

Total Effort: 2 days

Day 1: Implementation

• Morning: Create otel_session_stats.py (console parser)
• Afternoon: Update nav-start skill integration
• Evening: Test with OTel enabled/disabled

Day 2: Documentation & Testing

• Morning: Create OpenTelemetry SOP
• Afternoon: Update all docs (README, CLAUDE.md, DEVELOPMENT-README)
• Evening: Comprehensive testing (all 5 test scenarios)

Target Release: 2025-10-22

Related Tasks

• TASK-06: Real Session Statistics (original hacky implementation)
• TASK-12: v3.0 Skills-Only Migration (foundation)
• Future TASK-14: ROI Dashboard & Team Analytics (builds on this)

Notes

Why This Matters

1. Validation: Proves Navigator's 92% token reduction with real data
2. ROI: Teams can measure actual savings (cost reduction)
3. Cache Performance: Shows CLAUDE.md caching working
4. Official API: Won't break on updates (vs parsing internals)
5. Foundation: Enables future analytics features

What Changed from TASK-06

TASK-06 (original):

• Parsed Claude Code internal files
• File-size estimation (bytes / 4)
• Static calculation
• Fragile (breaks on updates)

TASK-13 (new):

• Uses official OpenTelemetry API
• Real token counts from Claude
• Dynamic session metrics
• Stable (official spec)

Migration: Delete old session_stats.py, replace with otel_session_stats.py

Task created: 2025-10-20 Priority: Medium (enhancement, not critical) Effort: Small (2 days) Impact: High (validates Navigator value with real metrics)