Plugin Introspector

Plugin Introspector — White-box Monitoring & Self-Improvement Meta-Plugin

A meta-plugin that monitors Claude Code workflow plugin execution at white-box level, collects execution data, and generates data-driven improvement proposals.

Role

A monitoring and analysis meta-plugin that provides end-to-end visibility into Claude Code workflow plugin execution. Collects tool traces, API interactions, token usage, and execution patterns via hooks. Provides analysis agents that identify bottlenecks, evaluate quality, detect anomalies, and generate concrete improvement proposals.

Terminology

• Plugin: An installable package (directory with .claude-plugin/plugin.json and skills/)
• Skill: An invocable unit within a plugin (directory with SKILL.md)
• Agent: A Task subagent definition (.md file in agents/)
• Proposal / Improvement: A concrete, ROI-scored change suggestion generated by improve

Core Principles

1. Data-driven improvement: Never guess — always base decisions on collected execution data
2. Minimal overhead: Hook scripts must complete in <50ms, use append-only JSONL
3. OTel compatibility: Trace data follows OpenTelemetry GenAI Semantic Conventions
4. Tiered collection: Tier 0 (pure hooks) → Tier 1 (+ OTel Collector with native telemetry)
5. Meta-rules compliance: All generated improvements must pass meta-rules validation
6. Zero external dependencies: bash + jq only — no Python, no Node.js (Tier 1 adds optional otelcol-contrib)

Pre-Execution: Session Resolution

Before any command, resolve the session directory:

1. Run: ls -t ~/.claude/plugin-introspector/sessions/ | head -1
   → Gives the most recent session ID
2. If --session {id} was provided: use that ID instead
3. Set: SESSION_DIR = ~/.claude/plugin-introspector/sessions/{resolved-id}
4. Verify: stats.json exists in SESSION_DIR
5. If not found: display "No session data. Run a session with hooks enabled first."

Full details: resources/orchestration-protocol.md

Pre-Execution: Target Plugin Resolution

When --target {plugin-name} is specified:

1. Search for plugin at: {working_dir}/plugins/{plugin-name}/
2. Fallback: check marketplace.json for source path
3. Catalog plugin components → store in SESSION_DIR/target_plugin.json
4. If not found: display "Plugin '{name}' not found" with available list

When --target is omitted:

• Infer plugin from traces: scan input_summary fields for plugin-specific paths
• If Skill tool invoked: identify from skill name
• If no plugin identified: treat as general session analysis

Data Storage

~/.claude/plugin-introspector/
├── sessions/{session-id}/
│   ├── meta.json              # Session metadata (git info, collection_tier)
│   ├── tool_traces.jsonl      # Pre/post tool traces with input_summary
│   ├── api_traces.jsonl       # API request/response metrics
│   ├── otel_traces.jsonl      # OTel spans (hook-generated or merged native)
│   ├── stats.json             # Aggregated session statistics
│   ├── evaluation.json        # Quality evaluation results (from evaluate)
│   ├── security_events.jsonl  # DLP violations, command risk events (from security hooks)
│   └── target_plugin.json     # Discovered target plugin info (from --target)
├── session_history.jsonl      # Cross-session summary records
├── evaluation_history.jsonl   # Cross-session evaluation scores
├── alerts.jsonl               # Anomaly detection alerts
├── improvement_log.jsonl      # Applied improvement history
├── aggregates.json            # Pre-computed cross-session aggregates (from Stop hook)
├── security_baseline.json     # Security baseline for anomaly comparison (optional)
├── telemetry.jsonl            # Opt-in anonymous telemetry (PI_TELEMETRY=1)
├── plugin-profiles/{plugin}/  # Per-plugin workflow profiles and baselines
├── otel-export/               # Tier 1: OTel Collector File Exporter output
└── otel-collector/            # Tier 1: Collector binary and config

Commands

Quick Reference

* available, experimental

Agent Invocation Pattern

Details: resources/orchestration-protocol.md

All analysis agents are invoked via the Task tool using the prompt template defined in orchestration-protocol.md:

1. Read the agent definition: agents/{agent-name}.md
2. Read the relevant JSONL data files from SESSION_DIR (apply truncation per Data Size Management)
3. Construct prompt from Task Prompt Template (see orchestration-protocol.md)
4. Invoke Task with subagent_type: "general-purpose", model per agent's Model Assignment
5. Parse agent's JSON output
6. Display results / write to SESSION_DIR

For multi-agent commands (report, improve, optimize):

1. Read all data files once and reuse across agents
2. For 'report': run independent agents in parallel (workflow-analyzer, token-optimizer, api-tracker, context-auditor), then quality-evaluator
3. For 'improve': run analysis agents + load cross-session data (evaluation_history, improvement_log, plugin profile), aggregate all, pass to improvement-generator with improvement-pipeline.md procedure
4. For 'optimize': read evaluation.json + evaluation_history + improvement_log + target component, pass to auto-optimizer

status — Current Session Overview

1. Resolve session directory
2. Read stats.json
3. Display: session ID, duration, tool calls, token estimate, error count, top tools

dashboard — htop-style Real-time Dashboard

1. Resolve session directory
2. Run Bash: scripts/dashboard.sh {session-id} [--full]
3. Renders token progress bar, per-tool breakdown, error rate

flow — Execution Flow Tree (OTel)

1. Resolve session directory
2. Read otel_traces.jsonl
3. Build parent-child span tree using parent_span_id
4. Display indented tree with timing and token info

profile — Generate Plugin Profile

1. Resolve target plugin (required: --target)
2. Read target plugin's main SKILL.md content
3. Read target plugin's component catalog (from target_plugin.json)
4. Read agent: agents/plugin-profiler.md
5. Invoke Task (haiku): pass SKILL.md content + component catalog
6. Output: workflow type, phases, detection patterns, key files
7. Write result to ~/.claude/plugin-introspector/plugin-profiles/{plugin}/profile.json
8. Create empty phase-baselines.json and learned-patterns.jsonl if not present

analyze — Deep Workflow Analysis

1. Resolve session + target plugin
2. Read tool_traces.jsonl + otel_traces.jsonl
3. Read agent: agents/workflow-analyzer.md
4. Invoke Task (sonnet): pass agent definition + data
5. Output: patterns, bottlenecks, efficiency ratio, recommendations

tokens / api / context — Single-Agent Analysis Commands

All follow the same pattern: resolve session → read data → invoke agent → display output.

evaluate — Quality Evaluation (LLM-as-Judge)

1. Resolve session + target plugin (optional: --target scopes evaluation)
2. Read tool_traces.jsonl + otel_traces.jsonl + stats.json
3. Read agent: agents/quality-evaluator.md
4. Invoke Task (sonnet): pass agent definition + data
5. Output: 4-dimension scores + quantified improvement signals
6. Write result to SESSION_DIR/evaluation.json
7. Append to ~/.claude/plugin-introspector/evaluation_history.jsonl:
   • From quality-evaluator output: weighted_score, scores
   • From stats.json: key_metrics (tool_calls, total_tokens_est, errors, error_rate, duration_ms)
   • From orchestrator phase detection (via plugin profile + trace analysis): phase_breakdown (optional, only for phased workflows)
   • From quality-evaluator improvement_signals: top_waste_sources (top 3 waste entries)
   • From improvement_log: improvements_active (IDs where status == "applied")
8. Update phase baselines (if target_plugin has profile with workflow.type == "phased"):
   • Read plugin-profiles/{plugin}/phase-baselines.json
   • Update running mean/stddev for each phase using current phase_breakdown
   • Increment sessions_count, set maturity.baselines_available = true if count ≥ 5
9. Closed Loop Check: Read ~/.claude/plugin-introspector/improvement_log.jsonl
   • Find entries where status == "applied" and post_score == null
   • If --target specified: only check entries matching target_plugin
   • For each pending entry: compare current weighted_score with pre_score
   • Update post_score and set status to validated or regressed
   • If regressed (score dropped >0.5): suggest rollback via improvement-apply-protocol.md

alerts — Anomaly Detection

1. Read ~/.claude/plugin-introspector/alerts.jsonl → display recent alerts
2. If deeper analysis requested:
   • Resolve session
   • Read tool_traces.jsonl + session_history.jsonl + alerts.jsonl
   • Read agent: agents/anomaly-detector.md
   • Invoke Task (haiku): pass agent definition + data (including alerts.jsonl for deduplication)
3. Output: alert list with severity, suggested actions

optimize — Auto-Optimize (APE Loop) (available, experimental)

1. Resolve target plugin (required: --target)
2. Resolve target component (--component or auto-select lowest-scoring)
3. Read evaluation.json (run evaluate first if missing)
4. Read evaluation_history.jsonl + improvement_log.jsonl (for contrastive analysis + historical learning)
5. Read agent: agents/auto-optimizer.md
6. Read target component file content
7. Invoke Task (opus): pass agent definition + evaluation + history + component content
8. Output: optimized version with diff + predicted score improvement
9. Display diff for user review — do NOT auto-apply

CAUTION: Review all diffs before applying. See resources/improvement-apply-protocol.md.

improve — Generate Improvement Proposals

1. Resolve session + target plugin (required: --target)
2. Run analysis pipeline (or reuse if already run in this session):
   • Parallel: workflow-analyzer, token-optimizer, context-auditor
   • Parallel: anomaly-detector, quality-evaluator
3. Load cross-session data:
   • evaluation_history.jsonl (last 20 records)
   • improvement_log.jsonl (all for target plugin)
   • Plugin profile: profile.json, phase-baselines.json, learned-patterns.jsonl
4. Read target plugin component files (apply file selection per orchestration-protocol.md)
5. Read agent: agents/improvement-generator.md
6. Read procedure: resources/improvement-pipeline.md
7. Invoke Task (opus): pass agent definition + pipeline procedure + all analysis results + cross-session data + selected plugin files
8. Output: ROI-scored proposals with quantified evidence, counterfactuals, diffs, meta-rules validation
9. Display proposals for user review — do NOT auto-apply

report — Full Analysis Report

1. Resolve session + target plugin
2. Read all JSONL data files once
3. Run independent agents in parallel (4 concurrent Task calls):
   • workflow-analyzer → analysis
   • token-optimizer → tokens
   • api-tracker → api
   • context-auditor → context
4. Run quality-evaluator sequentially (can use results from step 3 if needed)
5. Aggregate into unified report with sections:
   • Executive Summary, Workflow Analysis, Token Efficiency, API Performance, Context Audit, Quality Score, Recommendations

apply — Apply Improvement Proposals

1. Read the most recent improve or optimize output (proposals)
2. For each proposal, follow improvement-apply-protocol.md:
   • Display proposal summary + diff for user review
   • Wait for explicit user confirmation
   • Backup → Edit → Verify → Log to improvement_log.jsonl
3. After all approved proposals applied:
   • Display summary of changes
   • Suggest: "Run evaluate --target {plugin} after next session to validate improvements"

CAUTION: Never auto-apply. Each proposal requires explicit user approval.

quick-scan — Quick Plugin Diagnosis (1-minute)

1. Resolve target plugin (required: --target)
2. Analyze plugin structure: plugin.json, SKILL.md, agents, scripts, resources
3. Run security-scan.sh for security score
4. Read agent: agents/quick-scanner.md
5. Invoke Task (haiku): lightweight analysis
6. Output: formatted box diagram with structure summary, security score, recommendations
7. Suggest follow-up commands for detailed analysis

Use case: First-time plugin evaluation, PR review, quick health check

Details: agents/quick-scanner.md

rotate-data — Data Retention Management

1. Run Bash: scripts/rotate-data.sh
2. Deletes session directories older than PI_RETENTION_DAYS (default: 30)
3. Trims JSONL files to PI_RETENTION_LINES (default: 1000)
4. Displays summary of deleted sessions and freed space

Environment variables:

• PI_RETENTION_DAYS=30: Days to keep session directories
• PI_RETENTION_LINES=1000: Lines to keep in JSONL files
• PI_DRY_RUN=1: Preview what would be deleted without actually deleting
• PI_AUTO_ROTATE=1: Auto-run at session end (optional)

Example:

# Dry run to see what would be deleted
PI_DRY_RUN=1 /plugin-introspector rotate-data

# Keep only 7 days of data
PI_RETENTION_DAYS=7 /plugin-introspector rotate-data

security-scan — Plugin Static Security Analysis

1. Resolve target plugin (required: --target)
2. Run Bash: scripts/security-scan.sh {plugin-path}
3. Scans hook scripts for dangerous patterns (data exfiltration, reverse shells, credential theft)
4. Scans SKILL.md + resources for prompt injection patterns
5. Scans agent definitions for risky tool permission combinations
6. Output: JSON report with findings, severity levels, risk score
7. Logs findings to alerts.jsonl if any found

Environment variables:

• PI_ENABLE_SECURITY=1: Enables runtime command risk logging
• PI_ENABLE_DLP=1: Enables DLP scanning
• PI_SECURITY_BLOCK=1: Enables CRITICAL command blocking (use with caution)

Note: security-check.sh intentionally omits || true in plugin.json so that exit 2 can block CRITICAL commands when PI_SECURITY_BLOCK=1. All other hook scripts use || true per meta-rules.

Details: resources/security-patterns.md

security-audit — Session Security Audit

1. Resolve session
2. Read tool_traces.jsonl + otel_traces.jsonl + stats.json + security_events.jsonl + alerts.jsonl
3. Read agent: agents/security-auditor.md
4. Invoke Task (haiku): pass agent definition + all data (including stats.json)
5. Output: risk level, file access audit, command audit, suspicious sequences, DLP summary
6. Recommendations for security improvements

security-dashboard — Security Risk Visualization

1. Resolve session
2. Run Bash: scripts/security-dashboard.sh {session-id}
3. Renders security-focused dashboard with:
   • Overall risk score bar
   • DLP violations, sensitive reads/writes, risky commands
   • Recent security events
   • Security configuration status (DLP, security check, blocking)

compliance-report — Compliance Report Generation (available, experimental)

1. Read session_history.jsonl for specified period
2. Read alerts.jsonl + security_events.jsonl + improvement_log.jsonl (aggregated)
3. Optionally run security-scan for each active plugin
4. Read agent: agents/security-reporter.md
5. Invoke Task (sonnet): pass agent definition + period data
6. Output: SOC 2 / ISO 27001 compliance report with executive summary, security events, plugin audit, recommendations, and compliance mapping

otel-setup — OTel Collector Setup (Tier 1) (available, experimental)

1. Run Bash: scripts/setup-otel-collector.sh {subcommand}
   • install: Download OTel Collector Contrib binary (~100MB+)
   • start: Start collector with otel-config.yaml (OTLP receiver → file exporter)
   • stop: Stop collector
   • status: Show collector status, export data info, environment variables
   • env: Print shell environment variables to set
2. Display output to user

After setup, the collection tier automatically upgrades from Tier 0 to Tier 1. Hook scripts detect the running collector and skip redundant OTel span generation. Native OTel spans (with accurate token counts) are merged at session end.

See: Collection Tiers in orchestration-protocol.md

otel-security-map — OTel Security Event Mapper (available, experimental)

1. Run Bash: scripts/otel-security-mapper.sh [session-id|--watch|--stats]
2. Converts native Claude Code OTel events to PI security_events.jsonl format
3. Enables deeper security analysis using OTel's tool_parameters (bash_command, full_command)
4. Output modes:
   • <session-id>: Process specific session's OTel data
   • --watch: Real-time mapping mode
   • --stats: Show security statistics from OTel data

Use case: Enhanced security analysis with native OTel data (Tier 1+)

trace — Raw Trace Viewer

1. Resolve session
2. Read tool_traces.jsonl with filters: --tool {name}, --errors, --slow, --last {N}
3. Display formatted records

web — Export for External Tools

1. Resolve session
2. Export format: --otel, --json, --csv
3. Write export file, display path

Environment Variables

All environment variables are opt-in. Default behavior requires no configuration.

Telemetry Policy (PI_TELEMETRY):

• Data stored locally in telemetry.jsonl — never transmitted externally
• Collects: command counts only (no content, paths, or personal data)
• Use scripts/telemetry.sh status to view collected data

Resources (On-demand)

Agents

Skills (Knowledge Base)

Closed Loop: Improve → Apply → Validate

The full improvement cycle connects improve, evaluate, and the apply protocol:

┌─────────┐     ┌───────────┐     ┌─────────┐     ┌──────────┐
│ improve  │────▶│ user      │────▶│  apply   │────▶│ log to   │
│ (propose)│     │ review    │     │ (backup, │     │ improve- │
│          │     │ (approve?)│     │  edit,   │     │ ment_log │
│          │     │           │     │  verify) │     │ .jsonl   │
└─────────┘     └───────────┘     └─────────┘     └────┬─────┘
                                                       │
                                                       ▼
┌─────────┐     ┌───────────┐     ┌─────────────────────────┐
│ rollback │◀───│ regressed │◀───│ evaluate (next session)  │
│ (if bad) │    │ score?    │    │ → checks pending         │
│          │    │           │    │   post_scores in log     │
└─────────┘    └───────────┘    └─────────────────────────┘

Lifecycle

1. improve generates proposals with diffs and meta-rules validation
2. User reviews each proposal (never auto-apply)
3. Apply follows improvement-apply-protocol.md:
   • Backup → Edit → Verify → Log to improvement_log.jsonl with post_score: null
4. Next evaluate run (step 9 — Closed Loop Check) detects pending entries and fills post_score
5. If validated (post_score >= pre_score): mark status: "validated"
6. If regressed (post_score < pre_score - 0.5): suggest rollback, mark status: "regressed"

Rollback Trigger

When evaluate detects regression:

1. Display: "Improvement {proposal_id} caused score regression: {pre_score} → {post_score}"
2. Offer rollback: "Restore from backup? (yes/no)"
3. If yes: follow rollback procedure in improvement-apply-protocol.md
4. Log rollback to improvement_log.jsonl

Self-Referential Loop Prevention

1. Hook scripts check CLAUDE_TOOL_NAME + structured Skill name via jq — skip if Skill invocation targets plugin-introspector
2. Analysis agents exclude introspector-related tool calls from analysis
3. When running improve --target plugin-introspector: extra caution, meta-rules strictly enforced