Kelly Agent Observability

Agent Fleet Observability Desk

Overview

Kelly Agent Observability is a Busabase Cloud App-in-Skill. Its canonical product surface is the AirApp in Busabase, not a separate local-data product. The same Hono source supports an explicitly requested local preview with OAuth connection bootstrap. It gives an operator a read-mostly dashboard over a MOCK fleet of LLM agents running behind a shared AI gateway for a generic organization: per-agent call volume, latency (p50/p95), error/failure rate, cost, an hourly call-volume sparkline, and a Trace Detail view that highlights exactly where a chain of tool calls broke.

This skill was never a live monitoring integration, even before this conversion — there is no real gateway, no real agents, and no real telemetry anywhere. The fleet snapshot (agents + metrics + traces) is generated by a deterministic, seeded function and written into Busabase by a trusted skill-root script (scripts/generate_fleet_data.mjs); the AirApp itself only ever reads whatever was last generated there. The only human action is narrow: append an "acknowledge" / "needs investigation" handoff note to an agent or trace — always a brand-new row in the handoffs Base, never a field update on the agent/trace's own record.

Default behavior is AirApp-first. Unless the user explicitly asks only for explanation, ensure the mock fleet exists (run the seed script below if the agents Base is empty) and give the user the clickable AirApp URL. Start localhost only when local preview/debugging is explicitly requested; it uses the same Busabase resources and never offers another data provider. Use chat-only mode only when the user says "纯聊天", "chat only", "no UI", or similar.

This is deliberately generic and brand-free: agent names are generic archetypes (Booking Assistant, Support Triage, etc.) for a generic organization ("your organization", "the fleet") — no real company or product appears anywhere in the code, config, or seed data.

Mandatory Dependencies

1. Read and follow $kelly-app-skill-creator for product behavior, visual quality, responsive layout, and the complete canonical content/kelly-agent-observability-app/ artifact.
2. Read and follow $busabase for connection, target Space, node discovery, ChangeRequests, review, and merge behavior.
3. Read and follow $busabase-app-creator for resource modeling, AirApp runtime limits, security, validation, and deployment.

If a dependency is unavailable, preserve this skill's artifact and product contracts, stop before the unavailable Busabase operation, and report the exact missing dependency. Do not invent a second data backend.

App UI Screenshots

Boundary

• Read-mostly dashboard over a generated mock fleet. NEVER contact any real agent gateway, LLM provider, or third-party API. There is no live monitoring integration in this skill by design.
• Agents and traces are never created or edited by the AirApp; they enter Busabase only through the trusted scripts/generate_fleet_data.mjs seed script, the same way kelly-portfolio-health's contracts enter through an external sync process the app doesn't control.
• The only Base the AirApp itself ever writes to is handoffs, and only ever by creating a brand-new row (an acknowledge / needs-investigation note) — never a field update on an agent or trace record.
• Never reference any real company, brand, or production system. Agent names are generic archetypes for a generic organization ("your organization", "the fleet").

Busabase Resources

Four Bases under one application Folder (kelly-agent-observability), declared in content/kelly-agent-observability-app/app/js/config.js and the generated template sidecars under content/:

• agents: one row per mock agent archetype (8 rows) — identity plus the latest rolled-up health metrics (status, call volume, p50/p95 latency, error rate, cost) and a 48h hourly call/error series (JSON array). Written only by the trusted generator script.
• traces: one row per mock trace (ordered tool-call chain), capped to fit under the 100-record read limit (8 agents × 10 traces = 80 by default). steps (JSON array) carries the ordered timeline; broke-at-step-id flags exactly where a broken chain stopped. Written only by the trusted generator script.
• handoffs: append-only human-in-the-loop log — an acknowledge / needs-investigation note against one agent or trace. The only Base the AirApp itself ever writes to.
• settings: sanitized fleet-generation metadata (schema version, seed, generated-at), one row keyed by kind.

Resources provision lazily through an idempotent Busabase ChangeRequest the first time the app runs in a Space; see references/fleet-schema.md for exact field shapes.

First Run And Onboarding

On invocation, check the agents Base. If it's empty, run the trusted seed script to generate the mock fleet:

node skills/kelly-agent-observability/scripts/generate_fleet_data.mjs --apply

There are no credentials to collect — this skill never calls an external system, so onboarding is just running the seed script once.

Local App

Default behavior is AirApp-first — give the user the clickable AirApp URL. Start pnpm --dir content/kelly-agent-observability-app dev only when local preview/debugging is explicitly requested.

Required app views (hash routes):

• #/overview: total calls (24h), total cost (today), degraded/critical/ healthy agent counts, and a per-agent card grid with a call-volume sparkline.
• #/agents: sortable Agent Health table (call volume, p50/p95 latency, error rate, cost, status badge).
• #/agents/<agent_id>: agent detail — metrics, sparkline, recent traces, and a handoff form (acknowledge / needs investigation).
• #/traces/<trace_id>: trace detail — ordered step timeline with the chain-break step visually flagged, and a handoff form.
• #/handoffs: history of recorded handoffs.
• #/settings: data provider and status-threshold summary.

Demo Mode

• ?demo=1 opens a deterministic, fully offline mock fleet (8 agents, 48h of hourly buckets, 16 traces per agent) for documentation and screenshots. It never reads or writes Busabase and never claims a real connection; demo handoff submissions only update the in-memory list already rendered.
• ?demo=agents and ?demo=trace select named mock scenes (agent health table, a broken trace).
• lang=en or lang=zh forces UI chrome language for screenshots.

UI language: support English and Chinese chrome with Auto default.

Fleet Generation Model

content/kelly-agent-observability-app/app/js/fleet-model.js (generateFleetData/summarizeFleet) is ported verbatim from the retired lib/generate.ts:

• Per-agent profile — a fixed "personality" (volume, latency base/jitter, error tendency, cost per call) per agent archetype, so the fleet reads as varied, not random noise.
• Hourly buckets — 48 hours of calls/errors per agent on a business-hours-ish curve, seeded per agent.
• Status — statusFor(error_rate_pct, p95_latency_ms): critical at error rate ≥ 8% or p95 ≥ 8000ms, degraded at ≥ 3% or ≥ 4000ms, else healthy.
• Traces — an ordered tool-call chain per trace; a trace can "break" mid chain (status: error, broke_at_step_id set to the last step), with the break probability tied to the agent's error tendency.

Same seed + same "now" + same tracesPerAgent always produces bit-identical output — this is the property scripts/generate_fleet_data.mjs and the demo provider both rely on.

Handoffs

The one human action — acknowledge or flag "needs investigation" on an agent or trace, with a free-text note — writes a brand-new row into the handoffs Base via busabase-sdk, never a field update on the agent/trace record itself:

• From a standalone local preview the write merges immediately (trusted operator).
• From the deployed AirApp it creates a pending ChangeRequest for the trusted process to merge, per the AirApp boundary in $busabase-app-creator.

Safety

• Never invent a real incident or real company data; this is illustrative mock telemetry only.
• Keep all writes scoped to the handoffs Base. No external network calls anywhere in this skill.
• Redact nothing sensitive is collected here — there are no credentials in this skill.

Useful Commands

node skills/kelly-agent-observability/scripts/generate_fleet_data.mjs --apply
pnpm --dir skills/kelly-agent-observability/content/kelly-agent-observability-app dev

Execution reports

Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.