Agent Observability Skill
Query structured JSONL event logs to understand agent behavior, debug failures, and analyze performance.
Triggers
| Trigger Phrase |
Operation |
query agent logs |
Run query_logs.py with filters |
find slow tool calls |
Run with --slow threshold |
show agent errors |
Run with --errors-only |
summarize session performance |
Run with --output summary-sessions |
analyze tool usage |
Run with --output summary-tools |
When to Use
Use this skill when:
- Debugging why an agent chose a particular tool or approach
- Finding slow tool calls that degrade agent performance
- Analyzing error patterns across agent sessions
- Comparing tool usage across sessions or agents
- Tracing decisions from orchestrator through sub-agents
- Deciding which operational events new code must emit, and which it must leave out (read the event-logging contract below)
Use direct log file inspection instead when:
- Checking a single known event in a small log
- The log file has fewer than 10 events
Event Schema
Logs use JSONL format (one JSON object per line). See schema.json for the full JSON Schema.
Event Types
| Type |
Purpose |
Key Fields |
| session_start |
Agent invocation begins |
agent, session_id |
| session_end |
Agent invocation completes |
agent, session_id |
| tool_call |
Tool invocation with timing |
tool.name, tool.duration_ms, tool.success |
| decision |
Reasoning captured alongside action |
decision.action, decision.reasoning |
| metric |
Numeric measurement |
metric.name, metric.value, metric.unit |
| error |
Error occurrence |
error.message, error.category, error.recoverable |
Example Events
{"timestamp":"2026-03-30T10:00:00Z","event_type":"session_start","session_id":"sess-001","agent":"implementer","message":"Session started"}
{"timestamp":"2026-03-30T10:00:01Z","event_type":"tool_call","session_id":"sess-001","agent":"implementer","level":"INFO","tool":{"name":"Read","duration_ms":45,"success":true,"input_summary":"src/main.py"},"message":"Read source file"}
{"timestamp":"2026-03-30T10:00:02Z","event_type":"decision","session_id":"sess-001","agent":"implementer","level":"INFO","decision":{"action":"Edit existing function","reasoning":"Function exists, modifying is safer than rewriting","alternatives_considered":["Rewrite from scratch","Create wrapper"]}}
{"timestamp":"2026-03-30T10:00:10Z","event_type":"error","session_id":"sess-001","agent":"implementer","level":"ERROR","error":{"message":"Test failed: assertion error in test_parse","category":"test_failure","recoverable":true}}
Log File Location
Agent event logs are stored at:
.agents/
logs/
{session-id}.jsonl # Per-session event log
Process
- Identify the log file to query (by session ID or date)
- Run query_logs.py with appropriate filters
- Review output for patterns, errors, or performance issues
- Use summary modes for high-level analysis across many events
Anti-Patterns
| Avoid |
Why |
Instead |
| Reading raw JSONL manually for large logs |
Slow, error-prone |
Use query_logs.py with filters |
| Ignoring decision events |
Loses the "why" behind agent actions |
Filter by --event-type decision |
| Checking only errors |
Misses slow degradation patterns |
Use --slow to find latency issues |
| Analyzing without session context |
Events lack meaning without grouping |
Use --output summary-sessions |
Scripts
| Script |
Platform |
Usage |
scripts/query_logs.py |
Python 3.8+ |
Cross-platform |
check_event_logging.py |
Python 3.8+ |
Check a scenario against the event-logging contract |
| Exit Code |
Meaning |
| 0 |
Query completed and results output (query) or scenario compliant (check) |
| 1 |
File not found or invalid arguments (query) or scenario non-compliant (check) |
| 2 |
Invalid JSONL format (query) or malformed scenario (check) |
Quick Start
# Show all events in a session log
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl
# Check an emission scenario against the event-logging contract
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/check_event_logging.py" scenario.json
# Find errors only
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --errors-only
# Find tool calls slower than 500ms
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --slow 500
# Filter by agent
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --agent implementer
# Session summary as JSON
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --output summary-sessions
# Tool usage summary
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --output summary-tools
# Filter by time range
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl \
--since 2026-03-30T10:00:00Z --until 2026-03-30T11:00:00Z
# JSON output for automation
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl \
--output json --event-type tool_call
Verification
After execution:
References
Domain knowledge for observability analysis:
| File |
Content |
| event-logging-contract.md |
Canonical contract for which operational events code emits and omits: decision test, MUST and MUST NOT tables, log vs metric vs trace, worked example |
| three-pillars-reference.md |
Logs, metrics, traces definitions, correlation matrix, OpenTelemetry |
| prometheus-recording-rules.md |
Recording rule patterns, CPU throttling, disk I/O, PSI, TCP metrics |
| otel-migration-reference.md |
IFx to OTel migration phases, instrumentation best practices, standard metrics |
Related Documents
1---2name: observability-33description: Use when you say `query agent logs`, `find slow tool calls`, `show agent errors`, or `what to log`. Query and analyze agent JSONL event logs for debugging, performance analysis, and decision tracing.4metadata:5 source: https://skillmd.com/skills/rjmurillo/observability-36---78# Agent Observability Skill910Query structured JSONL event logs to understand agent behavior, debug failures, and analyze performance.1112## Triggers1314| Trigger Phrase | Operation |15|----------------|-----------|16| `query agent logs` | Run query_logs.py with filters |17| `find slow tool calls` | Run with --slow threshold |18| `show agent errors` | Run with --errors-only |19| `summarize session performance` | Run with --output summary-sessions |20| `analyze tool usage` | Run with --output summary-tools |2122## When to Use2324Use this skill when:2526- Debugging why an agent chose a particular tool or approach27- Finding slow tool calls that degrade agent performance28- Analyzing error patterns across agent sessions29- Comparing tool usage across sessions or agents30- Tracing decisions from orchestrator through sub-agents31- Deciding which operational events new code must emit, and which it must leave out (read the event-logging contract below)3233Use direct log file inspection instead when:3435- Checking a single known event in a small log36- The log file has fewer than 10 events3738## Event Schema3940Logs use JSONL format (one JSON object per line). See `schema.json` for the full JSON Schema.4142### Event Types4344| Type | Purpose | Key Fields |45|------|---------|------------|46| session_start | Agent invocation begins | agent, session_id |47| session_end | Agent invocation completes | agent, session_id |48| tool_call | Tool invocation with timing | tool.name, tool.duration_ms, tool.success |49| decision | Reasoning captured alongside action | decision.action, decision.reasoning |50| metric | Numeric measurement | metric.name, metric.value, metric.unit |51| error | Error occurrence | error.message, error.category, error.recoverable |5253### Example Events5455```jsonl56{"timestamp":"2026-03-30T10:00:00Z","event_type":"session_start","session_id":"sess-001","agent":"implementer","message":"Session started"}57{"timestamp":"2026-03-30T10:00:01Z","event_type":"tool_call","session_id":"sess-001","agent":"implementer","level":"INFO","tool":{"name":"Read","duration_ms":45,"success":true,"input_summary":"src/main.py"},"message":"Read source file"}58{"timestamp":"2026-03-30T10:00:02Z","event_type":"decision","session_id":"sess-001","agent":"implementer","level":"INFO","decision":{"action":"Edit existing function","reasoning":"Function exists, modifying is safer than rewriting","alternatives_considered":["Rewrite from scratch","Create wrapper"]}}59{"timestamp":"2026-03-30T10:00:10Z","event_type":"error","session_id":"sess-001","agent":"implementer","level":"ERROR","error":{"message":"Test failed: assertion error in test_parse","category":"test_failure","recoverable":true}}60```6162## Log File Location6364Agent event logs are stored at:6566```text67.agents/68 logs/69 {session-id}.jsonl # Per-session event log70```7172## Process73741. Identify the log file to query (by session ID or date)752. Run query_logs.py with appropriate filters763. Review output for patterns, errors, or performance issues774. Use summary modes for high-level analysis across many events7879## Anti-Patterns8081| Avoid | Why | Instead |82|-------|-----|---------|83| Reading raw JSONL manually for large logs | Slow, error-prone | Use query_logs.py with filters |84| Ignoring decision events | Loses the "why" behind agent actions | Filter by --event-type decision |85| Checking only errors | Misses slow degradation patterns | Use --slow to find latency issues |86| Analyzing without session context | Events lack meaning without grouping | Use --output summary-sessions |8788## Scripts8990| Script | Platform | Usage |91|--------|----------|-------|92| `scripts/query_logs.py` | Python 3.8+ | Cross-platform |93| `check_event_logging.py` | Python 3.8+ | Check a scenario against the event-logging contract |9495| Exit Code | Meaning |96|-----------|---------|97| 0 | Query completed and results output (query) or scenario compliant (check) |98| 1 | File not found or invalid arguments (query) or scenario non-compliant (check) |99| 2 | Invalid JSONL format (query) or malformed scenario (check) |100101## Quick Start102103```bash104# Show all events in a session log105python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl106107# Check an emission scenario against the event-logging contract108python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/check_event_logging.py" scenario.json109110# Find errors only111python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --errors-only112113# Find tool calls slower than 500ms114python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --slow 500115116# Filter by agent117python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --agent implementer118119# Session summary as JSON120python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --output summary-sessions121122# Tool usage summary123python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl --output summary-tools124125# Filter by time range126python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl \127 --since 2026-03-30T10:00:00Z --until 2026-03-30T11:00:00Z128129# JSON output for automation130python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/observability/scripts/query_logs.py" .project-toolkit/logs/sess-001.jsonl \131 --output json --event-type tool_call132```133134## Verification135136After execution:137138- [ ] Script exits with code 0139- [ ] Output format matches requested mode (table, json, summary-sessions, summary-tools)140- [ ] Filters reduce event count as expected141- [ ] Session summaries include all sessions present in the log142143## References144145Domain knowledge for observability analysis:146147| File | Content |148|------|---------|149| [event-logging-contract.md](references/event-logging-contract.md) | Canonical contract for which operational events code emits and omits: decision test, MUST and MUST NOT tables, log vs metric vs trace, worked example |150| [three-pillars-reference.md](references/three-pillars-reference.md) | Logs, metrics, traces definitions, correlation matrix, OpenTelemetry |151| [prometheus-recording-rules.md](references/prometheus-recording-rules.md) | Recording rule patterns, CPU throttling, disk I/O, PSI, TCP metrics |152| [otel-migration-reference.md](references/otel-migration-reference.md) | IFx to OTel migration phases, instrumentation best practices, standard metrics |153154## Related Documents155156- [Event Schema](schema.json)157- [Agent Metrics Skill](../metrics/SKILL.md)