Agent Observability Skill
Query structured JSONL event logs to understand agent behavior, debug failures, and analyze performance.
Triggers
| Trigger Phrase |
Operation |
query agent logs |
Run query_logs.py with filters |
find slow tool calls |
Run with --slow threshold |
show agent errors |
Run with --errors-only |
summarize session performance |
Run with --output summary-sessions |
analyze tool usage |
Run with --output summary-tools |
When to Use
Use this skill when:
- Debugging why an agent chose a particular tool or approach
- Finding slow tool calls that degrade agent performance
- Analyzing error patterns across agent sessions
- Comparing tool usage across sessions or agents
- Tracing decisions from orchestrator through sub-agents
Use direct log file inspection instead when:
- Checking a single known event in a small log
- The log file has fewer than 10 events
Event Schema
Logs use JSONL format (one JSON object per line). See schema.json for the full JSON Schema.
Event Types
| Type |
Purpose |
Key Fields |
| session_start |
Agent invocation begins |
agent, session_id |
| session_end |
Agent invocation completes |
agent, session_id |
| tool_call |
Tool invocation with timing |
tool.name, tool.duration_ms, tool.success |
| decision |
Reasoning captured alongside action |
decision.action, decision.reasoning |
| metric |
Numeric measurement |
metric.name, metric.value, metric.unit |
| error |
Error occurrence |
error.message, error.category, error.recoverable |
Example Events
{"timestamp":"2026-03-30T10:00:00Z","event_type":"session_start","session_id":"sess-001","agent":"implementer","message":"Session started"}
{"timestamp":"2026-03-30T10:00:01Z","event_type":"tool_call","session_id":"sess-001","agent":"implementer","level":"INFO","tool":{"name":"Read","duration_ms":45,"success":true,"input_summary":"src/main.py"},"message":"Read source file"}
{"timestamp":"2026-03-30T10:00:02Z","event_type":"decision","session_id":"sess-001","agent":"implementer","level":"INFO","decision":{"action":"Edit existing function","reasoning":"Function exists, modifying is safer than rewriting","alternatives_considered":["Rewrite from scratch","Create wrapper"]}}
{"timestamp":"2026-03-30T10:00:10Z","event_type":"error","session_id":"sess-001","agent":"implementer","level":"ERROR","error":{"message":"Test failed: assertion error in test_parse","category":"test_failure","recoverable":true}}
Log File Location
Agent event logs are stored at:
.agents/
logs/
{session-id}.jsonl # Per-session event log
Process
- Identify the log file to query (by session ID or date)
- Run query_logs.py with appropriate filters
- Review output for patterns, errors, or performance issues
- Use summary modes for high-level analysis across many events
Anti-Patterns
| Avoid |
Why |
Instead |
| Reading raw JSONL manually for large logs |
Slow, error-prone |
Use query_logs.py with filters |
| Ignoring decision events |
Loses the "why" behind agent actions |
Filter by --event-type decision |
| Checking only errors |
Misses slow degradation patterns |
Use --slow to find latency issues |
| Analyzing without session context |
Events lack meaning without grouping |
Use --output summary-sessions |
Scripts
| Script |
Platform |
Usage |
scripts/query_logs.py |
Python 3.8+ |
Cross-platform |
| Exit Code |
Meaning |
| 0 |
Query completed and results output |
| 1 |
File not found or invalid arguments |
| 2 |
Invalid JSONL format |
Quick Start
# Show all events in a session log
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl
# Find errors only
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --errors-only
# Find tool calls slower than 500ms
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --slow 500
# Filter by agent
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --agent implementer
# Session summary as JSON
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-sessions
# Tool usage summary
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-tools
# Filter by time range
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \
--since 2026-03-30T10:00:00Z --until 2026-03-30T11:00:00Z
# JSON output for automation
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \
--output json --event-type tool_call
Verification
After execution:
References
Domain knowledge for observability analysis:
| File |
Content |
| three-pillars-reference.md |
Logs, metrics, traces definitions, correlation matrix, OpenTelemetry |
| prometheus-recording-rules.md |
Recording rule patterns, CPU throttling, disk I/O, PSI, TCP metrics |
| otel-migration-reference.md |
IFx to OTel migration phases, instrumentation best practices, standard metrics |
Related Documents
1---2name: observability3version: 1.0.04model: haiku5model-rationale: cost. The 'haiku' rolling alias resolves via the platform model_tiers map to a tier priced below the sonnet-tier harness default; this unit is routing/mechanical work where the cheaper tier suffices (ADR-080 rule 3).6description: Use when you say `query agent logs`, `find slow tool calls`, or `show agent errors`. Query and analyze agent JSONL event logs for debugging, performance analysis, and decision tracing.7license: MIT8metadata:9 source: https://skillmd.com/skills/rjmurillo/observability10---1112# Agent Observability Skill1314Query structured JSONL event logs to understand agent behavior, debug failures, and analyze performance.1516## Triggers1718| Trigger Phrase | Operation |19|----------------|-----------|20| `query agent logs` | Run query_logs.py with filters |21| `find slow tool calls` | Run with --slow threshold |22| `show agent errors` | Run with --errors-only |23| `summarize session performance` | Run with --output summary-sessions |24| `analyze tool usage` | Run with --output summary-tools |2526## When to Use2728Use this skill when:2930- Debugging why an agent chose a particular tool or approach31- Finding slow tool calls that degrade agent performance32- Analyzing error patterns across agent sessions33- Comparing tool usage across sessions or agents34- Tracing decisions from orchestrator through sub-agents3536Use direct log file inspection instead when:3738- Checking a single known event in a small log39- The log file has fewer than 10 events4041## Event Schema4243Logs use JSONL format (one JSON object per line). See `schema.json` for the full JSON Schema.4445### Event Types4647| Type | Purpose | Key Fields |48|------|---------|------------|49| session_start | Agent invocation begins | agent, session_id |50| session_end | Agent invocation completes | agent, session_id |51| tool_call | Tool invocation with timing | tool.name, tool.duration_ms, tool.success |52| decision | Reasoning captured alongside action | decision.action, decision.reasoning |53| metric | Numeric measurement | metric.name, metric.value, metric.unit |54| error | Error occurrence | error.message, error.category, error.recoverable |5556### Example Events5758```jsonl59{"timestamp":"2026-03-30T10:00:00Z","event_type":"session_start","session_id":"sess-001","agent":"implementer","message":"Session started"}60{"timestamp":"2026-03-30T10:00:01Z","event_type":"tool_call","session_id":"sess-001","agent":"implementer","level":"INFO","tool":{"name":"Read","duration_ms":45,"success":true,"input_summary":"src/main.py"},"message":"Read source file"}61{"timestamp":"2026-03-30T10:00:02Z","event_type":"decision","session_id":"sess-001","agent":"implementer","level":"INFO","decision":{"action":"Edit existing function","reasoning":"Function exists, modifying is safer than rewriting","alternatives_considered":["Rewrite from scratch","Create wrapper"]}}62{"timestamp":"2026-03-30T10:00:10Z","event_type":"error","session_id":"sess-001","agent":"implementer","level":"ERROR","error":{"message":"Test failed: assertion error in test_parse","category":"test_failure","recoverable":true}}63```6465## Log File Location6667Agent event logs are stored at:6869```text70.agents/71 logs/72 {session-id}.jsonl # Per-session event log73```7475## Process76771. Identify the log file to query (by session ID or date)782. Run query_logs.py with appropriate filters793. Review output for patterns, errors, or performance issues804. Use summary modes for high-level analysis across many events8182## Anti-Patterns8384| Avoid | Why | Instead |85|-------|-----|---------|86| Reading raw JSONL manually for large logs | Slow, error-prone | Use query_logs.py with filters |87| Ignoring decision events | Loses the "why" behind agent actions | Filter by --event-type decision |88| Checking only errors | Misses slow degradation patterns | Use --slow to find latency issues |89| Analyzing without session context | Events lack meaning without grouping | Use --output summary-sessions |9091## Scripts9293| Script | Platform | Usage |94|--------|----------|-------|95| `scripts/query_logs.py` | Python 3.8+ | Cross-platform |9697| Exit Code | Meaning |98|-----------|---------|99| 0 | Query completed and results output |100| 1 | File not found or invalid arguments |101| 2 | Invalid JSONL format |102103## Quick Start104105```bash106# Show all events in a session log107python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl108109# Find errors only110python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --errors-only111112# Find tool calls slower than 500ms113python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --slow 500114115# Filter by agent116python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --agent implementer117118# Session summary as JSON119python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-sessions120121# Tool usage summary122python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-tools123124# Filter by time range125python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \126 --since 2026-03-30T10:00:00Z --until 2026-03-30T11:00:00Z127128# JSON output for automation129python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \130 --output json --event-type tool_call131```132133## Verification134135After execution:136137- [ ] Script exits with code 0138- [ ] Output format matches requested mode (table, json, summary-sessions, summary-tools)139- [ ] Filters reduce event count as expected140- [ ] Session summaries include all sessions present in the log141142## References143144Domain knowledge for observability analysis:145146| File | Content |147|------|---------|148| [three-pillars-reference.md](references/three-pillars-reference.md) | Logs, metrics, traces definitions, correlation matrix, OpenTelemetry |149| [prometheus-recording-rules.md](references/prometheus-recording-rules.md) | Recording rule patterns, CPU throttling, disk I/O, PSI, TCP metrics |150| [otel-migration-reference.md](references/otel-migration-reference.md) | IFx to OTel migration phases, instrumentation best practices, standard metrics |151152## Related Documents153154- [Event Schema](schema.json)155- [Agent Metrics Skill](../metrics/SKILL.md)156- [Issue #1301](https://github.com/rjmurillo/ai-agents/issues/1301)