Agent Observability Skill
Query structured JSONL event logs to understand agent behavior, debug failures, and analyze performance.
Triggers
| Trigger Phrase |
Operation |
query agent logs |
Run query_logs.py with filters |
find slow tool calls |
Run with --slow threshold |
show agent errors |
Run with --errors-only |
summarize session performance |
Run with --output summary-sessions |
analyze tool usage |
Run with --output summary-tools |
When to Use
Use this skill when:
- Debugging why an agent chose a particular tool or approach
- Finding slow tool calls that degrade agent performance
- Analyzing error patterns across agent sessions
- Comparing tool usage across sessions or agents
- Tracing decisions from orchestrator through sub-agents
Use direct log file inspection instead when:
- Checking a single known event in a small log
- The log file has fewer than 10 events
Event Schema
Logs use JSONL format (one JSON object per line). See schema.json for the full JSON Schema.
Event Types
| Type |
Purpose |
Key Fields |
| session_start |
Agent invocation begins |
agent, session_id |
| session_end |
Agent invocation completes |
agent, session_id |
| tool_call |
Tool invocation with timing |
tool.name, tool.duration_ms, tool.success |
| decision |
Reasoning captured alongside action |
decision.action, decision.reasoning |
| metric |
Numeric measurement |
metric.name, metric.value, metric.unit |
| error |
Error occurrence |
error.message, error.category, error.recoverable |
Example Events
{"timestamp":"2026-03-30T10:00:00Z","event_type":"session_start","session_id":"sess-001","agent":"implementer","message":"Session started"}
{"timestamp":"2026-03-30T10:00:01Z","event_type":"tool_call","session_id":"sess-001","agent":"implementer","level":"INFO","tool":{"name":"Read","duration_ms":45,"success":true,"input_summary":"src/main.py"},"message":"Read source file"}
{"timestamp":"2026-03-30T10:00:02Z","event_type":"decision","session_id":"sess-001","agent":"implementer","level":"INFO","decision":{"action":"Edit existing function","reasoning":"Function exists, modifying is safer than rewriting","alternatives_considered":["Rewrite from scratch","Create wrapper"]}}
{"timestamp":"2026-03-30T10:00:10Z","event_type":"error","session_id":"sess-001","agent":"implementer","level":"ERROR","error":{"message":"Test failed: assertion error in test_parse","category":"test_failure","recoverable":true}}
Log File Location
Agent event logs are stored at:
.agents/
logs/
{session-id}.jsonl # Per-session event log
Process
- Identify the log file to query (by session ID or date)
- Run query_logs.py with appropriate filters
- Review output for patterns, errors, or performance issues
- Use summary modes for high-level analysis across many events
Anti-Patterns
| Avoid |
Why |
Instead |
| Reading raw JSONL manually for large logs |
Slow, error-prone |
Use query_logs.py with filters |
| Ignoring decision events |
Loses the "why" behind agent actions |
Filter by --event-type decision |
| Checking only errors |
Misses slow degradation patterns |
Use --slow to find latency issues |
| Analyzing without session context |
Events lack meaning without grouping |
Use --output summary-sessions |
Scripts
| Script |
Platform |
Usage |
scripts/query_logs.py |
Python 3.8+ |
Cross-platform |
| Exit Code |
Meaning |
| 0 |
Query completed and results output |
| 1 |
File not found or invalid arguments |
| 2 |
Invalid JSONL format |
Quick Start
# Show all events in a session log
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl
# Find errors only
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --errors-only
# Find tool calls slower than 500ms
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --slow 500
# Filter by agent
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --agent implementer
# Session summary as JSON
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-sessions
# Tool usage summary
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-tools
# Filter by time range
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \
--since 2026-03-30T10:00:00Z --until 2026-03-30T11:00:00Z
# JSON output for automation
python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \
--output json --event-type tool_call
Verification
After execution:
References
Domain knowledge for observability analysis:
| File |
Content |
| three-pillars-reference.md |
Logs, metrics, traces definitions, correlation matrix, OpenTelemetry |
| prometheus-recording-rules.md |
Recording rule patterns, CPU throttling, disk I/O, PSI, TCP metrics |
| otel-migration-reference.md |
IFx to OTel migration phases, instrumentation best practices, standard metrics |
Related Documents
1---2name: observability3description: Use when you say `query agent logs`, `find slow tool calls`, or `show agent errors`. Query and analyze agent JSONL event logs for debugging, performance analysis, and decision tracing.4license: MIT5---67# Agent Observability Skill89Query structured JSONL event logs to understand agent behavior, debug failures, and analyze performance.1011## Triggers1213| Trigger Phrase | Operation |14|----------------|-----------|15| `query agent logs` | Run query_logs.py with filters |16| `find slow tool calls` | Run with --slow threshold |17| `show agent errors` | Run with --errors-only |18| `summarize session performance` | Run with --output summary-sessions |19| `analyze tool usage` | Run with --output summary-tools |2021## When to Use2223Use this skill when:2425- Debugging why an agent chose a particular tool or approach26- Finding slow tool calls that degrade agent performance27- Analyzing error patterns across agent sessions28- Comparing tool usage across sessions or agents29- Tracing decisions from orchestrator through sub-agents3031Use direct log file inspection instead when:3233- Checking a single known event in a small log34- The log file has fewer than 10 events3536## Event Schema3738Logs use JSONL format (one JSON object per line). See `schema.json` for the full JSON Schema.3940### Event Types4142| Type | Purpose | Key Fields |43|------|---------|------------|44| session_start | Agent invocation begins | agent, session_id |45| session_end | Agent invocation completes | agent, session_id |46| tool_call | Tool invocation with timing | tool.name, tool.duration_ms, tool.success |47| decision | Reasoning captured alongside action | decision.action, decision.reasoning |48| metric | Numeric measurement | metric.name, metric.value, metric.unit |49| error | Error occurrence | error.message, error.category, error.recoverable |5051### Example Events5253```jsonl54{"timestamp":"2026-03-30T10:00:00Z","event_type":"session_start","session_id":"sess-001","agent":"implementer","message":"Session started"}55{"timestamp":"2026-03-30T10:00:01Z","event_type":"tool_call","session_id":"sess-001","agent":"implementer","level":"INFO","tool":{"name":"Read","duration_ms":45,"success":true,"input_summary":"src/main.py"},"message":"Read source file"}56{"timestamp":"2026-03-30T10:00:02Z","event_type":"decision","session_id":"sess-001","agent":"implementer","level":"INFO","decision":{"action":"Edit existing function","reasoning":"Function exists, modifying is safer than rewriting","alternatives_considered":["Rewrite from scratch","Create wrapper"]}}57{"timestamp":"2026-03-30T10:00:10Z","event_type":"error","session_id":"sess-001","agent":"implementer","level":"ERROR","error":{"message":"Test failed: assertion error in test_parse","category":"test_failure","recoverable":true}}58```5960## Log File Location6162Agent event logs are stored at:6364```text65.agents/66 logs/67 {session-id}.jsonl # Per-session event log68```6970## Process71721. Identify the log file to query (by session ID or date)732. Run query_logs.py with appropriate filters743. Review output for patterns, errors, or performance issues754. Use summary modes for high-level analysis across many events7677## Anti-Patterns7879| Avoid | Why | Instead |80|-------|-----|---------|81| Reading raw JSONL manually for large logs | Slow, error-prone | Use query_logs.py with filters |82| Ignoring decision events | Loses the "why" behind agent actions | Filter by --event-type decision |83| Checking only errors | Misses slow degradation patterns | Use --slow to find latency issues |84| Analyzing without session context | Events lack meaning without grouping | Use --output summary-sessions |8586## Scripts8788| Script | Platform | Usage |89|--------|----------|-------|90| `scripts/query_logs.py` | Python 3.8+ | Cross-platform |9192| Exit Code | Meaning |93|-----------|---------|94| 0 | Query completed and results output |95| 1 | File not found or invalid arguments |96| 2 | Invalid JSONL format |9798## Quick Start99100```bash101# Show all events in a session log102python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl103104# Find errors only105python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --errors-only106107# Find tool calls slower than 500ms108python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --slow 500109110# Filter by agent111python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --agent implementer112113# Session summary as JSON114python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-sessions115116# Tool usage summary117python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl --output summary-tools118119# Filter by time range120python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \121 --since 2026-03-30T10:00:00Z --until 2026-03-30T11:00:00Z122123# JSON output for automation124python .claude/skills/observability/scripts/query_logs.py .agents/logs/sess-001.jsonl \125 --output json --event-type tool_call126```127128## Verification129130After execution:131132- [ ] Script exits with code 0133- [ ] Output format matches requested mode (table, json, summary-sessions, summary-tools)134- [ ] Filters reduce event count as expected135- [ ] Session summaries include all sessions present in the log136137## References138139Domain knowledge for observability analysis:140141| File | Content |142|------|---------|143| [three-pillars-reference.md](references/three-pillars-reference.md) | Logs, metrics, traces definitions, correlation matrix, OpenTelemetry |144| [prometheus-recording-rules.md](references/prometheus-recording-rules.md) | Recording rule patterns, CPU throttling, disk I/O, PSI, TCP metrics |145| [otel-migration-reference.md](references/otel-migration-reference.md) | IFx to OTel migration phases, instrumentation best practices, standard metrics |146147## Related Documents148149- [Event Schema](schema.json)150- [Agent Metrics Skill](../metrics/SKILL.md)151- [Issue #1301](https://github.com/rjmurillo/ai-agents/issues/1301)
rjmurillo