Agent Observability

Agent Observability Skill

Prerequisites

• Bun runtime installed
• Claude Code with hooks configured
• PAI_DIR environment variable set

Installation

See SETUP.md for complete installation instructions.

Quick Setup:

# 1. Set environment variable
export PAI_DIR="$HOME/.claude"  # Add to ~/.zshrc or ~/.bashrc

# 2. Configure hooks (merge into ~/.claude/settings.json)
cat settings.json.example

# 3. Create directory structure
mkdir -p ~/.claude/history/raw-outputs

# 4. Install dependencies
cd apps/server && bun install
cd ../client && bun install

Usage

Start the Observability Dashboard

Terminal 1 - Server:

cd ~/Projects/PAI/skills/agent-observability/apps/server
bun run dev

Terminal 2 - Client:

cd ~/Projects/PAI/skills/agent-observability/apps/client
bun run dev

Open browser: http://localhost:5173

Using Claude Code

Once the dashboard is running, any Claude Code activity will appear in real-time:

1. Open Claude Code
2. Use any tool (Read, Write, Bash, etc.)
3. Launch subagents with Task tool
4. Watch events appear in the dashboard

Event Types Captured

• SessionStart - New Claude Code session begins
• UserPromptSubmit - User sends a message
• PreToolUse - Before a tool is executed
• PostToolUse - After a tool completes
• Stop - Main agent task completes
• SubagentStop - Subagent task completes
• SessionEnd - Session ends

Features

Real-Time Visualization

• Agent Swim Lanes: See multiple agents (kai, designer, engineer, etc.) running in parallel
• Event Timeline: Chronological view of all events
• Tool Usage Charts: Visualize which tools are being used most
• Session Tracking: Track individual sessions and their lifecycles

Filtering & Search

• Filter by agent name (kai, designer, engineer, pentester, etc.)
• Filter by event type (PreToolUse, PostToolUse, etc.)
• Filter by session ID
• Search event payloads

Data Storage

Events are stored in JSONL (JSON Lines) format:

~/.claude/history/raw-outputs/YYYY-MM/YYYY-MM-DD_all-events.jsonl

Each line is a complete JSON object:

{"source_app":"kai","session_id":"abc123","hook_event_type":"PreToolUse","payload":{...},"timestamp":1234567890,"timestamp_pst":"2025-01-28 14:30:00 PST"}

In-Memory Streaming

• Server keeps last 1000 events in memory
• Low memory footprint
• Fast real-time updates via WebSocket
• No database overhead

Architecture

┌─────────────────┐
│  Claude Code    │  Executes hooks on events
│   (with hooks)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ capture-all-    │  Appends events to JSONL
│ events.ts hook  │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────┐
│ ~/.claude/history/raw-outputs/      │  Daily JSONL files
│ 2025-01/2025-01-28_all-events.jsonl │
└────────┬────────────────────────────┘
         │
         ▼
┌─────────────────┐
│ file-ingest.ts  │  Watches files, streams to memory
│  (Bun server)   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Vue 3 Client   │  Real-time dashboard visualization
│  (Vite + Tail)  │
└─────────────────┘

Configuration

Environment Variables

PAI_DIR: Path to your PAI directory (defaults to ~/.claude/)

export PAI_DIR="/Users/yourname/.claude"

Hooks Configuration

Add to ~/.claude/settings.json (see settings.json.example for full template):

{
  "hooks": {
    "PreToolUse": [{
      "matcher": "*",
      "hooks": [{
        "type": "command",
        "command": "${PAI_DIR}/skills/agent-observability/hooks/capture-all-events.ts --event-type PreToolUse"
      }]
    }],
    // ... other hooks
  }
}

Troubleshooting

No events appearing

1. Check PAI_DIR is set: echo $PAI_DIR
2. Verify directory exists: ls ~/.claude/history/raw-outputs/
3. Check hook is executable: ls -l hooks/capture-all-events.ts
4. Look for today's events file: ls ~/.claude/history/raw-outputs/$(date +%Y-%m)/

Server won't start

1. Check Bun is installed: bun --version
2. Verify dependencies: cd apps/server && bun install
3. Check port 3001 isn't in use: lsof -i :3001

Client won't connect

1. Ensure server is running first
2. Check WebSocket connection in browser console
3. Verify no firewall blocking localhost:3001

Credits

Inspired by @indydevdan's pioneering work on multi-agent observability for Claude Code.

Our implementation differs by using filesystem-based event capture and in-memory streaming instead of SQLite database persistence. Both approaches have their merits! Check out indydevdan's work for a database-backed solution with full historical persistence.

Development

Running in Development

# Server (hot reload)
cd apps/server
bun --watch src/index.ts

# Client (Vite dev server)
cd apps/client
bun run dev

Building for Production

# Client build
cd apps/client
bun run build
bun run preview

Adding New Event Types

1. Update capture-all-events.ts hook if needed
2. Add hook configuration to settings.json
3. Client will automatically display new event types

Documentation

• README.md - Complete documentation
• SETUP.md - Installation guide
• history-structure/ - Data storage structure
• settings.json.example - Hook configuration template

License

Part of the PAI (Personal AI Infrastructure) project.

Contributing

Contributions welcome! Areas for improvement:

• Historical data persistence options
• Export functionality (CSV, JSON)
• Alert/notification system
• Advanced filtering and search
• Session replay capability
• Integration with other PAI skills