Performance Monitor

Performance Monitor

When to Use

Trigger phrases:

• "performance monitor"
• "Help me with performance monitor"

Use cases:

• When the task matches this skill's domain expertise

When NOT to use:

• For tasks outside this skill's scope

/performance-monitor start skill-name

Get report

/performance-monitor report skill-name --days 7

Compare skills

/performance-monitor compare skill1 skill2 --metric success_rate

### Features

- Real-time metric collection
- Historical trend analysis
- Anomaly detection
- Performance regression alerts
- Cost tracking per skill


## When NOT to Use

- When the skill is stable and not changing
- For skills with fewer than 10 invocations (not enough data)
- When manual curation produces better results


## Overview

The Performance Monitor is a meta-skill that provides execution telemetry for every skill in the ecosystem. It collects, stores, and analyzes key performance indicators — execution latency, success rates, accuracy against expected outcomes, resource consumption (tokens, memory, wall time), and invocation frequency — giving both the agent and its operator visibility into how skills actually perform in production.

The lifecycle of performance monitoring spans five phases: **instrumentation** (wrapping skill execution to capture raw data), **collection** (aggregating and normalizing metrics into a structured store), **analysis** (computing percentiles, trends, and anomaly scores), **alerting** (firing notifications when metrics cross configured thresholds), and **improvement** (feeding insights back into skill versioning to guide optimization). Each phase is independently configurable, so a skill can graduate from simple timer-based tracking to full token accounting as it matures.

Measurement granularity matters. The monitor captures per-invocation data — each call to a skill produces a record with latency, outcome status (success, failure, timeout, or error), error type when applicable, estimated token consumption, and a timestamp. These raw samples feed into rolling window statistics: p50/p95/p99 latency, success rate over the last 1000 invocations, daily active skill counts, and token burn rate per skill per session.

The design follows the principle that monitoring must never become the bottleneck. The metric writer operates with a write-behind buffer and batch-inserts to SQLite (the default backend) at sub-millisecond overhead per invocation. For larger deployments, the same interface can back Prometheus counters or TimescaleDB hypertables without changing instrumentation code.


## Workflow

1. **Instrument the skill** — Apply the `@track_performance` decorator to the skill's entry-point function, or register a global hook via the 1ai-skills hook system that wraps every skill invocation automatically. The instrumentation layer captures start time, function arguments (sanitized), and agent session metadata.

2. **Execute and capture** — As the skill runs, the monitor records elapsed wall time, exit status (success or exception with error type), and estimated token consumption. Failures are captured in the `finally` block so that even crashing skills produce a metric record.

3. **Persist to the metrics store** — Each invocation sample is buffered and batch-inserted into the configured backend (SQLite by default). The schema stores skill_name, latency_ms, status, error_type, tokens_estimated, and recorded_at. The batch interval is configurable to balance write overhead against data freshness.

4. **Aggregate and compute trends** — A periodic aggregation job reads raw samples and computes rolling statistics: success rate over sliding windows (100 / 1K / 10K invocations), latency percentiles (p50, p95, p99), invocation frequency per hour, and token burn rate. Results are stored in a summary table for fast querying.

5. **Analyze for anomalies and regressions** — Compare the latest aggregated values against the trailing 14-day baseline. A z-score exceeding the configured threshold (default 3.0) triggers a regression flag. The analyzer also detects missing metrics (a skill that suddenly stopped reporting) and volume anomalies (unusual spike in invocations).

6. **Alert on threshold breaches** — When a regression flag is raised, the monitor dispatches a notification through the configured channel (Slack webhook, Telegram bot, or log file). The alert includes the skill name, metric that crossed the threshold, baseline vs current value, and a link to the trend chart.

7. **Review and optimize** — Weekly trend reports are generated automatically. Skill maintainers review the report, correlate metric shifts with code changes, and prioritize optimization. When a skill version is updated, the pre- and post-change metrics are compared to confirm the fix and update the baseline.

## Architecture

- **Input layer** — Receives and validates incoming requests
- **Processing layer** — Core logic for skill management
- **Output layer** — Formats and delivers results
- **State management** — Maintains context across invocations

## Code Examples

### Tracking Decorator

```python
import time, functools, sqlite3
from pathlib import Path
from datetime import datetime, timezone

DB_PATH = Path.home() / ".1ai" / "metrics.db"

def track_performance(skill_name: str):
    """Decorator that records latency, success, and token usage for a skill call."""
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            start = time.monotonic()
            try:
                result = fn(*args, **kwargs)
                status = "success"
                error_type = None
            except Exception as e:
                result = None
                status = "failure"
                error_type = type(e).__name__
                raise
            finally:
                elapsed_ms = (time.monotonic() - start) * 1000
                _write_metric(skill_name, elapsed_ms, status, error_type)
            return result
        return wrapper
    return decorator

def _write_metric(name, latency_ms, status, error_type):
    conn = sqlite3.connect(str(DB_PATH))
    try:
        conn.execute(
            "INSERT INTO skill_metrics (skill_name, latency_ms, status, error_type, recorded_at) "
            "VALUES (?, ?, ?, ?, ?)",
            (name, latency_ms, status, error_type, datetime.now(timezone.utc).isoformat()),
        )
        conn.commit()
    finally:
        conn.close()

Report Generator

def generate_report(skill_name: str, days: int = 7) -> dict:
    """Aggregate metrics and return summary statistics."""
    conn = sqlite3.connect(str(DB_PATH))
    conn.row_factory = sqlite3.Row
    rows = conn.execute(
        "SELECT latency_ms, status, error_type FROM skill_metrics "
        "WHERE skill_name = ? AND recorded_at >= datetime('now', ?)",
        (skill_name, f"-{days} days"),
    ).fetchall()
    conn.close()

    if not rows:
        return {"error": "no data for period"}

    latencies = [r["latency_ms"] for r in rows]
    successes = sum(1 for r in rows if r["status"] == "success")
    sorted_lat = sorted(latencies)
    n = len(latencies)

    return {
        "skill": skill_name,
        "period_days": days,
        "invocations": n,
        "success_rate": round(successes / n, 4),
        "latency_ms": {
            "p50": sorted_lat[n // 2] if n else 0,
            "p95": sorted_lat[int(n * 0.95)] if n else 0,
            "p99": sorted_lat[int(n * 0.99)] if n else 0,
            "avg": round(sum(latencies) / n, 1) if n else 0,
        },
        "errors": {e: sum(1 for r in rows if r["error_type"] == e)
                   for e in set(r["error_type"] for r in rows if r["error_type"])},
    }

Usage Example

# Apply the tracker to any skill function
@track_performance("content-seo-optimizer")
def run_seo_optimizer(text):
    # ... skill logic ...
    return results

# Generate a weekly report
report = generate_report("content-seo-optimizer", days=7)
print(f"Success rate: {report['success_rate']*100:.1f}%")
print(f"p95 latency: {report['latency_ms']['p95']:.0f}ms")

Configuration

• Metrics backend — SQLite file path (default: ~/.1ai/metrics.db), connection pool size, write-batch interval
• Instrumentation mode — decorator (wrap individual functions), hook (intercept all skill calls via hook system), or hybrid
• Latency thresholds — p50/p95/p99 targets in milliseconds; defaults: p50 < 500ms, p95 < 2_000ms, p99 < 5_000ms
• Token budget — per-skill token cap per invocation and per-session alert threshold
• Retention policy — raw sample TTL in days, aggregation interval (hourly → daily → weekly), compression method
• Alert channels — Slack webhook URL, Telegram bot token, or local log file path for regression notifications
• Anomaly detection — enable/disable toggle, z-score threshold (default: 3.0), baseline window in days (default: 14)

Integration

• Exposes standard interfaces for other skills to consume
• Supports event-driven and request-response patterns
• Compatible with the 1ai-skills hook system
• Logs metrics for the skill performance monitor

Common Issues / Troubleshooting

Anti-Rationalization Table

Monetization

Process

Preparation

• Instrument the target skill with the @track_performance decorator or context manager wrapper.
• Define acceptance thresholds for key metrics: latency (p50/p95/p99), success rate (≥99%), token budget per call.
• Choose a metrics backend — SQLite for single-agent setups, TimescaleDB or Prometheus for distributed deployments.
• Configure monitoring interval, data retention policy (default: 90 days raw samples, rolling aggregates permanent).

Execution

• Run the skill through its full range of invocations — success paths, edge cases, and expected failure modes.
• Verify that every call path produces a metric record: success, failure, timeout, or error classification.
• Monitor real-time dashboards during batch runs to spot regressions before they affect production.

Stewardship

• Review weekly trend reports and correlate metric shifts with recent skill version changes.
• Tune alert thresholds as the skill matures — tighter for production-critical skills, wider for experimental ones.
• Archive raw data beyond retention limits; compress old aggregates to the minimum-resolution tier.

Verification

• Instrumentation hooks registered and firing on each skill invocation
• Metric capture verified with at least one real invocation per tracked skill
• Latency, success status, token count, and error type written to metrics backend
• Anomaly detection triggers on regressions exceeding configured z-score threshold
• Reporting pipeline generates periodic trend output without errors
• Alert channel delivers notifications on cross-threshold events
• Cost-per-skill breakdown reconciles with actual usage logs