ADK Observability Guide
Iron Law
NEVER ship a production ADK agent without Cloud Trace enabled. Set otel_to_cloud=True in the FastAPI app and LOGS_BUCKET_NAME env var before any production deployment.
Reference Files
| File |
Contents |
reference/cloud-trace-and-logging.md |
Cloud Trace setup, prompt-response logging infrastructure, environment variables, enabling/disabling locally, verification commands |
reference/bigquery-agent-analytics.md |
BigQuery Agent Analytics plugin — enabling, key features, tool provenance tracking |
reference/slo-alerting.md |
SLO-based burn-rate alerting (vs threshold alerting), multi-window pattern, error budget tracking, PromQL examples for availability/latency/safety SLOs |
Observability Tiers
| Tier |
What It Does |
Default State |
Best For |
| Tier 1: Cloud Trace |
Distributed tracing — execution flow, latency, errors via OpenTelemetry spans |
Always-on |
Debugging latency, understanding agent execution flow |
| Tier 2: Prompt-Response Logging |
GenAI interactions exported to GCS, BigQuery, and Cloud Logging |
Disabled locally; enabled when deployed |
Auditing LLM interactions, compliance |
| Tier 3: BigQuery Agent Analytics |
Structured agent events (LLM calls, tool use, outcomes) to BigQuery |
Opt-in (--bq-analytics at scaffold time) |
Conversational analytics, custom dashboards, LLM-as-judge evals |
| Tier 4: SLO Alerting |
SLO-based burn-rate alerts (availability, latency, safety) via Prometheus — fires only when error budget is genuinely at risk |
Opt-in — requires Prometheus + Alertmanager stack |
Production alerting for agentic AI services; replaces naive threshold alerts |
Ask the user which tier(s) they need — they can be combined. Tier 1 is mandatory; Tiers 2, 3, and 4 are additive.
Quick Setup
Tier 1: Cloud Trace (Always-On)
# In your FastAPI app — scaffolded projects have this pre-configured
from google.adk.telemetry import setup_telemetry
setup_telemetry(otel_to_cloud=True)
View traces: Cloud Console > Trace > Trace explorer
Tier 2: Prompt-Response Logging
# Enable locally
export LOGS_BUCKET_NAME="your-bucket-name"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="NO_CONTENT"
See reference/cloud-trace-and-logging.md for full infrastructure list, env vars, and verification commands.
Tier 3: BigQuery Agent Analytics
# Enable at scaffold time
uvx agent-starter-pack create . --bq-analytics
See reference/bigquery-agent-analytics.md for post-scaffold manual setup and key features.
Process
- Confirm tier(s) — Ask which tiers the user needs before configuring anything
- Read reference files — See
reference/cloud-trace-and-logging.md (Tier 1 & 2) or reference/bigquery-agent-analytics.md (Tier 3) before writing any configuration
- Check env vars — Verify all required environment variables are set (table in
reference/cloud-trace-and-logging.md)
- Verify with commands — Use the verification commands in
reference/cloud-trace-and-logging.md to confirm telemetry is flowing
- Check costs — High telemetry volume? Switch to
NO_CONTENT mode; reduce BigQuery retention; disable unused tiers
Troubleshooting
| Issue |
Solution |
| No traces in Cloud Trace |
Verify otel_to_cloud=True in FastAPI app; check SA has cloudtrace.agent role |
| Prompt-response data missing |
Check LOGS_BUCKET_NAME is set; verify SA has storage.objectCreator; check app logs |
| Privacy mode misconfigured |
Check OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT — use NO_CONTENT for metadata-only |
| BigQuery Analytics not logging |
Verify plugin in app/agent.py; check BQ_ANALYTICS_DATASET_ID env var |
| Traces missing tool spans |
Tool spans appear under execute_tool — check trace explorer filters |
Documentation Sources
| Topic |
URL |
| Observability overview |
https://google.github.io/adk-docs/observability/index.md |
| Agent activity logging |
https://google.github.io/adk-docs/observability/logging/index.md |
| Cloud Trace integration |
https://google.github.io/adk-docs/integrations/cloud-trace/index.md |
| BigQuery Agent Analytics |
https://google.github.io/adk-docs/integrations/bigquery-agent-analytics/index.md |
1---2name: adk-observability-guide3description: ADK observability, Cloud Trace, prompt logging, agent analytics, BigQuery agent logs, ADK monitoring, ADK telemetry. Use when configuring tracing, logging, or analytics for ADK agents, debugging production agent behavior, or setting up monitoring for a deployed agent.4---56# ADK Observability Guide78## Iron Law910**NEVER ship a production ADK agent without Cloud Trace enabled. Set `otel_to_cloud=True` in the FastAPI app and `LOGS_BUCKET_NAME` env var before any production deployment.**1112## Reference Files1314| File | Contents |15|------|----------|16| `reference/cloud-trace-and-logging.md` | Cloud Trace setup, prompt-response logging infrastructure, environment variables, enabling/disabling locally, verification commands |17| `reference/bigquery-agent-analytics.md` | BigQuery Agent Analytics plugin — enabling, key features, tool provenance tracking |18| `reference/slo-alerting.md` | SLO-based burn-rate alerting (vs threshold alerting), multi-window pattern, error budget tracking, PromQL examples for availability/latency/safety SLOs | Setting up production alerts for agentic AI services |1920---2122## Observability Tiers2324| Tier | What It Does | Default State | Best For |25|------|-------------|---------------|----------|26| **Tier 1: Cloud Trace** | Distributed tracing — execution flow, latency, errors via OpenTelemetry spans | Always-on | Debugging latency, understanding agent execution flow |27| **Tier 2: Prompt-Response Logging** | GenAI interactions exported to GCS, BigQuery, and Cloud Logging | Disabled locally; enabled when deployed | Auditing LLM interactions, compliance |28| **Tier 3: BigQuery Agent Analytics** | Structured agent events (LLM calls, tool use, outcomes) to BigQuery | Opt-in (`--bq-analytics` at scaffold time) | Conversational analytics, custom dashboards, LLM-as-judge evals |29| **Tier 4: SLO Alerting** | SLO-based burn-rate alerts (availability, latency, safety) via Prometheus — fires only when error budget is genuinely at risk | Opt-in — requires Prometheus + Alertmanager stack | Production alerting for agentic AI services; replaces naive threshold alerts |3031Ask the user which tier(s) they need — they can be combined. Tier 1 is mandatory; Tiers 2, 3, and 4 are additive.3233---3435## Quick Setup3637### Tier 1: Cloud Trace (Always-On)3839```python40# In your FastAPI app — scaffolded projects have this pre-configured41from google.adk.telemetry import setup_telemetry42setup_telemetry(otel_to_cloud=True)43```4445View traces: **Cloud Console > Trace > Trace explorer**4647### Tier 2: Prompt-Response Logging4849```bash50# Enable locally51export LOGS_BUCKET_NAME="your-bucket-name"52export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="NO_CONTENT"53```5455See `reference/cloud-trace-and-logging.md` for full infrastructure list, env vars, and verification commands.5657### Tier 3: BigQuery Agent Analytics5859```bash60# Enable at scaffold time61uvx agent-starter-pack create . --bq-analytics62```6364See `reference/bigquery-agent-analytics.md` for post-scaffold manual setup and key features.6566---6768## Process69701. **Confirm tier(s)** — Ask which tiers the user needs before configuring anything712. **Read reference files** — See `reference/cloud-trace-and-logging.md` (Tier 1 & 2) or `reference/bigquery-agent-analytics.md` (Tier 3) before writing any configuration723. **Check env vars** — Verify all required environment variables are set (table in `reference/cloud-trace-and-logging.md`)734. **Verify with commands** — Use the verification commands in `reference/cloud-trace-and-logging.md` to confirm telemetry is flowing745. **Check costs** — High telemetry volume? Switch to `NO_CONTENT` mode; reduce BigQuery retention; disable unused tiers7576---7778## Troubleshooting7980| Issue | Solution |81|-------|----------|82| No traces in Cloud Trace | Verify `otel_to_cloud=True` in FastAPI app; check SA has `cloudtrace.agent` role |83| Prompt-response data missing | Check `LOGS_BUCKET_NAME` is set; verify SA has `storage.objectCreator`; check app logs |84| Privacy mode misconfigured | Check `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` — use `NO_CONTENT` for metadata-only |85| BigQuery Analytics not logging | Verify plugin in `app/agent.py`; check `BQ_ANALYTICS_DATASET_ID` env var |86| Traces missing tool spans | Tool spans appear under `execute_tool` — check trace explorer filters |8788---8990## Documentation Sources9192| Topic | URL |93|-------|-----|94| Observability overview | `https://google.github.io/adk-docs/observability/index.md` |95| Agent activity logging | `https://google.github.io/adk-docs/observability/logging/index.md` |96| Cloud Trace integration | `https://google.github.io/adk-docs/integrations/cloud-trace/index.md` |97| BigQuery Agent Analytics | `https://google.github.io/adk-docs/integrations/bigquery-agent-analytics/index.md` |
kumaran-is