Agent Observability

Agent Observability

You instrument the user's product agents — not the coding agent — so every run leaves a trace that can be inspected, scored, and learned from. You explain everything in plain language: the user is non-technical.

Plain-Language Primer (teach this first, once)

• A trace is the full story of one request: every model call, tool call, and retrieval step, in order, with timing, tokens, and cost attached.
• A span is one step inside that story (one LLM call, one tool call).
• Observability means you can answer "why did this output happen?" and "what did this run cost?" after the fact, without guessing.
• The backend is a separate managed service. Your product sends trace data to it over HTTPS as a side effect of running. You do NOT host it yourself in the common case, and it never runs on your laptop.
• Why it matters: an agent can return a confident wrong answer with HTTP 200. Without traces you cannot debug it, cannot measure quality, and cannot run any learning loop (runtime-learning-loop hard-fails without this).

Hard Rules

Never log secrets, API keys, or user PII into trace payloads — configure redaction/masking before production traffic, not after. Never propose self-hosting on the user's machine. Managed free tier first; self-host-in-cloud is a clearly-labeled last resort. Always instrument against OpenTelemetry GenAI conventions (gen_ai.*) or OpenInference — never a proprietary-only SDK — so the backend can be swapped later by config, not a rewrite. Always verify the first trace end-to-end (run once, open the trace, see spans + token counts) before calling the setup done. Always set a sampling/cost policy before production scale — keep failures at 100%, sample successes.

Workflow

Step 1 — Scope

Identify: which agent flows matter most (start with ONE), expected volume/month, whether data is sensitive (PII/health/finance), and which framework the product uses (Google ADK, LangGraph/LangChain, OpenAI/Anthropic SDK direct, custom).

Step 2 — Choose a backend

Read references/backends.md (decision table, free tiers, tradeoffs — verify current pricing at decision time; tiers change). Defaults: Langfuse Cloud free tier for general use; Phoenix for agent-heavy/ADK products (deepest multi-step agent views); LangSmith only if the product is LangChain/LangGraph-first. Sensitive data → check the backend's redaction options FIRST; most redact nothing by default.

Step 3 — Instrument

Prefer auto-instrumentation for the detected framework (one register()/instrument() call + env vars for keys/endpoint). Put API keys in environment variables — never in code. Wrap only the entry point; let the instrumentor capture nested spans. Tag spans with user_id/session_id/feature name so cost and quality can be sliced later.

Step 4 — Verify first trace

Run one real request. Open the backend UI. Confirm: full span tree visible, prompts/completions captured, token counts + cost present, no secrets/PII in payloads. If any check fails, fix before proceeding.

Step 5 — Cost and sampling policy

Async exporters add negligible latency; the real costs are storage and noise. Policy: 100% of failed runs, sampled successes (start 10–25% at scale), retention matched to the free tier's window. Set a monthly span budget aligned to the free-tier cap.

Step 6 — Wire to quality loops

Point eval-pipeline at production traces (score samples with eval-judge); this trace+eval pair is the precondition runtime-learning-loop checks. Record backend choice + config in docs/observability/OBSERVABILITY.md and log it to docs/skill-outputs/SKILL-OUTPUTS.md.

Output Format

=== Observability Plan: [product] ===
Flow instrumented: [flow] | Framework: [ADK|LangGraph|SDK|custom]
Backend: [name + tier] — why: [1 line] | Est. volume: [spans/mo] vs free cap: [cap]
Redaction: [what is masked] | Sampling: [failures 100%, successes N%]
First trace verified: [yes/no] | Saved: docs/observability/OBSERVABILITY.md
Next: eval-pipeline wiring → runtime-learning-loop precondition met

Gotchas

• Traces are for the shipped product; run-trace is for the coding agent. Different data, different store — never mix them.
• SaaS backends receive your prompts. That is user data leaving your infrastructure — check redaction defaults (usually OFF) before pointing production at them.
• Instrument to the standard, not the vendor. All major backends ingest OTel GenAI gen_ai.* spans; the backend is a 2-year decision, the instrumentation is forever.
• Free-tier caps are per month and small (e.g. tens of thousands of spans). One agent request can emit 5–30 spans. Do the multiplication before promising "free".
• A trace that omits tool calls is half a trace. Multi-agent debugging lives in the tool/handoff spans, not the final completion.

Example

Read references/examples.md for full walkthroughs (Langfuse path, redaction setup, backend swap).

Common Rationalizations

Verification

• First real trace opened in backend UI with full span tree + token counts
• No secrets or PII visible in any span payload (spot-check 3 traces)
• Sampling + retention policy written into OBSERVABILITY.md
• Instrumentation is OTel GenAI / OpenInference (backend swappable by config)
• docs/skill-outputs/SKILL-OUTPUTS.md appended

Red Flags

• Backend chosen before checking framework auto-instrumentation support
• Production traffic flowing with default (empty) redaction config
• Proprietary SDK wrapped around every call site instead of one instrumentor
• "Observability done" claimed without opening a single real trace
• Coding-agent activity and product traces mixed in one store

Impact Report

Observability set up for: [product/flow]
Backend: [name + tier] | Framework instrumented: [name]
First trace verified: [yes/no] | Redaction: [configured/not needed]
Sampling: [policy] | Files: docs/observability/OBSERVABILITY.md
Ready for: eval-pipeline wiring, runtime-learning-loop precondition