Observability & Tracing for Multi-Agent Systems
One trace = one run. One JSONL line = one event. Log once; view many times without re-running.
Data model
- Trace — the complete record of one pipeline run, keyed by
trace_id.
- Span — a logical unit of work within a trace (agent turn, tool call, sub-task). Spans nest via
parent.
- Event — one JSONL line: a boundary crossing (
trace_start, step_complete, tool_call, error, …).
See references/event-schema.md for the full field list, types, and a safe JSONL example.
Storage convention
.context/traces/<trace_id>.jsonl ← one file per trace, append-only
Add .context/traces/ to .gitignore. Rotate or archive files older than your retention window — see references/usage.md.
Viewing a trace
python scripts/trace_view.py <trace_id> # timeline + summary
python scripts/trace_view.py --latest # most recently modified trace
python scripts/trace_view.py --list # enumerate available traces
python scripts/trace_view.py --help # all options
scripts/trace_view.py is self-contained stdlib Python — run it directly. references/usage.md explains output columns and integration patterns (read, don't run).
Integration: start / step / end
import json, datetime, pathlib, sys
TRACES_DIR = pathlib.Path(".context/traces")
def trace_log(trace_id: str, **fields) -> None:
"""Append one event; swallow all errors so logging never breaks the pipeline."""
try:
TRACES_DIR.mkdir(parents=True, exist_ok=True)
event = {"trace_id": trace_id, "ts": datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z"), **fields}
with open(TRACES_DIR / f"{trace_id}.jsonl", "a") as f:
f.write(json.dumps(event) + "\n")
except Exception as exc:
print(f"[trace] warn: {exc}", file=sys.stderr)
# Call 1 — at pipeline start
trace_log(tid, kind="trace_start", span="root", step=0,
agent=ORCHESTRATOR_NAME, status="started",
input_tokens=0, output_tokens=0, parent=None)
# Call 2 — after each agent step (repeat for every turn)
trace_log(tid, kind="step_complete", span=span_id, step=step_n,
agent=agent_name, tool=tool_used_or_none, status="ok",
input_tokens=in_t, output_tokens=out_t, parent=parent_span)
# Call 3 — at pipeline end (or on abort)
trace_log(tid, kind="trace_end", span="root", step=final_step_n,
agent=ORCHESTRATOR_NAME, status="ok",
input_tokens=0, output_tokens=0, parent=None)
Anti-patterns
| Anti-pattern |
Risk |
| Logging prompt text or user messages verbatim |
PII / secrets leakage |
| Logging API keys, tokens, or env vars |
Direct secret exposure |
Raising exceptions inside trace_log |
Breaks the pipeline; observability must fail open |
| Appending all runs to one file |
Destroys per-trace isolation; --latest becomes meaningless |
| Logging response payloads > a few KB |
Trace files bloat; slow to render and archive |
Reference files
references/event-schema.md — READ — field-by-field schema, hierarchy diagram, annotated examplereferences/usage.md — READ — storage layout, rotation, output format, run-vs-read guide
Script
scripts/trace_view.py — RUN — self-contained timeline viewer; run --help before reading sourceexamples/selftest.sh — RUN — renders the shipped demo trace and checks --list discovery
Related skills in this bundle
eval-harness — scores the traces this skill records (the demo trace is shared between the two).prompt-contracts — when a trace shows the same output rejected turn after turn, that skill finds which contract layer drifted.registry-ssot — the routing truth a healthy trace should reflect.
1---2name: observability-tracing3description: Instrument a multi-agent pipeline with structured per-trace JSONL logs and render them as a human-readable timeline, token summary, and error report — without ever breaking the pipeline. Use this skill when you need to see what a multi-step or multi-agent run actually did, debug why a pipeline stalled or produced wrong output, audit token spend across agents and tools, or produce a durable audit trail that survives fan-out across runtimes. Trigger on any of: add observability, tracing, pipeline logging, debug agents, audit trail, trace viewer, why did the agent do X.4license: Apache-2.05---67# Observability & Tracing for Multi-Agent Systems89One trace = one run. One JSONL line = one event. Log once; view many times without re-running.1011## Data model1213- **Trace** — the complete record of one pipeline run, keyed by `trace_id`.14- **Span** — a logical unit of work within a trace (agent turn, tool call, sub-task). Spans nest via `parent`.15- **Event** — one JSONL line: a boundary crossing (`trace_start`, `step_complete`, `tool_call`, `error`, …).1617See `references/event-schema.md` for the full field list, types, and a safe JSONL example.1819## Storage convention2021```22.context/traces/<trace_id>.jsonl ← one file per trace, append-only23```2425Add `.context/traces/` to `.gitignore`. Rotate or archive files older than your retention window — see `references/usage.md`.2627## Viewing a trace2829```bash30python scripts/trace_view.py <trace_id> # timeline + summary31python scripts/trace_view.py --latest # most recently modified trace32python scripts/trace_view.py --list # enumerate available traces33python scripts/trace_view.py --help # all options34```3536`scripts/trace_view.py` is **self-contained stdlib Python** — run it directly. `references/usage.md` explains output columns and integration patterns (read, don't run).3738## Integration: start / step / end3940```python41import json, datetime, pathlib, sys4243TRACES_DIR = pathlib.Path(".context/traces")4445def trace_log(trace_id: str, **fields) -> None:46 """Append one event; swallow all errors so logging never breaks the pipeline."""47 try:48 TRACES_DIR.mkdir(parents=True, exist_ok=True)49 event = {"trace_id": trace_id, "ts": datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z"), **fields}50 with open(TRACES_DIR / f"{trace_id}.jsonl", "a") as f:51 f.write(json.dumps(event) + "\n")52 except Exception as exc:53 print(f"[trace] warn: {exc}", file=sys.stderr)5455# Call 1 — at pipeline start56trace_log(tid, kind="trace_start", span="root", step=0,57 agent=ORCHESTRATOR_NAME, status="started",58 input_tokens=0, output_tokens=0, parent=None)5960# Call 2 — after each agent step (repeat for every turn)61trace_log(tid, kind="step_complete", span=span_id, step=step_n,62 agent=agent_name, tool=tool_used_or_none, status="ok",63 input_tokens=in_t, output_tokens=out_t, parent=parent_span)6465# Call 3 — at pipeline end (or on abort)66trace_log(tid, kind="trace_end", span="root", step=final_step_n,67 agent=ORCHESTRATOR_NAME, status="ok",68 input_tokens=0, output_tokens=0, parent=None)69```7071## Anti-patterns7273| Anti-pattern | Risk |74|---|---|75| Logging prompt text or user messages verbatim | PII / secrets leakage |76| Logging API keys, tokens, or env vars | Direct secret exposure |77| Raising exceptions inside `trace_log` | Breaks the pipeline; observability must fail open |78| Appending all runs to one file | Destroys per-trace isolation; `--latest` becomes meaningless |79| Logging response payloads > a few KB | Trace files bloat; slow to render and archive |8081## Reference files8283- `references/event-schema.md` — **READ** — field-by-field schema, hierarchy diagram, annotated example84- `references/usage.md` — **READ** — storage layout, rotation, output format, run-vs-read guide8586## Script8788- `scripts/trace_view.py` — **RUN** — self-contained timeline viewer; run `--help` before reading source89- `examples/selftest.sh` — **RUN** — renders the shipped demo trace and checks `--list` discovery9091## Related skills in this bundle9293- `eval-harness` — scores the traces this skill records (the demo trace is shared between the two).94- `prompt-contracts` — when a trace shows the same output rejected turn after turn, that skill finds which contract layer drifted.95- `registry-ssot` — the routing truth a healthy trace should reflect.
sharp-skills