Technical Design: worker-observability

Technical Design: worker-observability

Metadata

• Feature: worker-observability
• Status: APPROVED
• Created: 2026-02-02
• Author: ZERG Design Mode

1. Overview

1.1 Summary

Close remaining gaps on issues #27 (worker health) and #30 (repo map), then build the token usage metrics subsystem (#24). PR #95 delivered heartbeat, escalation, verification tiers, and repo map modules. This feature wires them into /zerg:status display, adds incremental repo map indexing, and creates token counting/tracking/aggregation with an optional Anthropic SDK integration.

1.2 Goals

• Per-worker HEALTH, REPO MAP, and TOKEN USAGE dashboard sections in /zerg:status
• Token counting with optional Anthropic SDK (off by default, heuristic fallback)
• Incremental repo map re-indexing (hash-based staleness)
• Close issues #27, #30, #24

1.3 Non-Goals

• Runtime token tracking inside worker subprocess (only pre-flight estimation)
• Auto-generating CHANGELOG content
• TUI dashboard implementation (#32 — separate issue)

2. Architecture

2.1 High-Level Design

┌──────────────┐     ┌──────────────┐     ┌──────────────────┐
│ TokenCounter │────▶│ TokenTracker │────▶│ TokenAggregator  │
│ (count/est)  │     │ (per-worker) │     │ (cumulative)     │
└──────────────┘     └──────────────┘     └──────────────────┘
                                                   │
       ┌───────────────┐    ┌──────────────┐       │
       │ HeartbeatMon  │    │ RepoMap      │       │
       │ (existing)    │    │ +Incremental │       │
       └───────┬───────┘    └──────┬───────┘       │
               │                   │               │
               ▼                   ▼               ▼
         ┌─────────────────────────────────────────────┐
         │           StatusFormatter                    │
         │  format_health_table()                       │
         │  format_repo_map_stats()                     │
         │  format_token_table() + format_savings()     │
         └──────────────────┬──────────────────────────┘
                            │
                            ▼
                  ┌──────────────────┐
                  │ status.core.md   │
                  │ (dashboard tmpl) │
                  └──────────────────┘

2.2 Component Breakdown

2.3 Data Flow

1. Context plugin builds worker context (command template + task context + repo map + security rules)
2. TokenCounter estimates token count (API with cache or heuristic)
3. TokenTracker writes per-task breakdown to .zerg/state/tokens-{worker_id}.json
4. TokenAggregator reads all worker files, computes totals and savings
5. StatusFormatter reads heartbeat, escalation, repo map index, and token data
6. /zerg:status displays formatted sections

3. Key Decisions

3.1 Token Counting Mode

Context: Anthropic SDK count_tokens() is an API call requiring ANTHROPIC_API_KEY.

Decision: Optional, off by default. Three modes:

1. api_counting=False (default): heuristic (chars / 4.0), marked "(estimated)"
2. api_counting=True + anthropic installed: API with SHA256 cache, marked "(exact)"
3. api_counting=True + anthropic missing: log warning once, heuristic fallback

Rationale: Never block execution. Token metrics are informational. Users opt in.

3.2 Optional Dependency

Context: Adding anthropic as a hard dependency would require API keys for basic ZERG usage.

Decision: pip install zerg[metrics] optional extra. Lazy try: import anthropic pattern.

Rationale: Core ZERG must work without API keys. Token counting is a bonus.

3.3 Status Formatter as Separate Module

Context: status.core.md is a command template (markdown), not Python code.

Decision: Create zerg/status_formatter.py with pure functions that format data into ASCII tables. The status command template references these formatters.

Rationale: Testable Python code. Reusable across CLI and slash command.

4. Implementation Plan

4.1 Phase Summary

4.2 File Ownership

4.3 Consumer Matrix

4.4 Dependency Graph

graph TD
    T001[TASK-001: Config + deps] --> T003[TASK-003: Token counter]
    T001 --> T008[TASK-008: Config docs + plugin wiring]
    T002[TASK-002: Incremental repo map] --> T007[TASK-007: REPO MAP + TOKENS sections]
    T003 --> T004[TASK-004: Token tracker]
    T004 --> T005[TASK-005: Token aggregator]
    T004 --> T008
    T005 --> T007
    T006[TASK-006: HEALTH section] --> T007
    T007 --> T009[TASK-009: Unit tests - token modules]
    T007 --> T010[TASK-010: Unit tests - formatter + integration]
    T008 --> T010
    T009 --> T011[TASK-011: CHANGELOG + docs]
    T010 --> T011

5. Risk Assessment

6. Testing Strategy

6.1 Unit Tests

• TokenCounter: mock anthropic, test 3 modes, cache hit/miss, error handling
• TokenTracker: atomic writes, per-task breakdown, file corruption handling
• TokenAggregator: multi-worker aggregation, savings math, empty data
• StatusFormatter: table formatting, column alignment, empty/missing data

6.2 Integration Tests

• Token metrics pipeline: count → track → aggregate → format
• Repo map incremental: initial index → modify file → re-index → verify selective parse

6.3 Verification Commands

• pytest tests/unit/ -v
• pytest tests/integration/ -v
• python -m zerg.validate_commands