Observability Patterns

Observability Patterns Skill

This skill provides comprehensive templates and configurations for implementing observability in Google ADK agents. Includes logging, tracing, BigQuery analytics, Cloud Trace integration, and third-party observability platforms.

Overview

Google ADK supports multiple observability approaches for monitoring, debugging, and analyzing agent behavior:

1. Cloud Trace - Google Cloud native tracing with OpenTelemetry
2. BigQuery Agent Analytics - Comprehensive event logging and analysis
3. AgentOps - Session replays and unified tracing analytics
4. Phoenix (Arize) - Open-source observability with self-hosted control
5. Weave (W&B) - Weights & Biases platform for tracking and visualization

This skill covers production-ready observability implementations with security and scalability.

Available Scripts

1. Setup Cloud Trace

Script: scripts/setup-cloud-trace.sh <project-id>

Purpose: Configures Cloud Trace integration for ADK agents

Parameters:

• project-id - Google Cloud project ID (required)

Usage:

# Setup Cloud Trace for local development
./scripts/setup-cloud-trace.sh my-project-id

# Setup with ADK CLI deployment
adk deploy agent_engine --project=my-project-id --trace_to_cloud ./agent

Environment Variables:

• GOOGLE_CLOUD_PROJECT - Project ID for Cloud Trace
• GOOGLE_APPLICATION_CREDENTIALS - Path to service account key

Output: Cloud Trace enabled, traces visible in console.cloud.google.com

2. Setup BigQuery Agent Analytics

Script: scripts/setup-bigquery-analytics.sh <project-id> <dataset-id> [bucket-name]

Purpose: Configures BigQuery Agent Analytics plugin for comprehensive event logging

Parameters:

• project-id - Google Cloud project ID (required)
• dataset-id - BigQuery dataset name (required)
• bucket-name - GCS bucket for multimodal content (optional)

Usage:

# Setup basic BigQuery analytics
./scripts/setup-bigquery-analytics.sh my-project agent-analytics

# Setup with GCS for multimodal content
./scripts/setup-bigquery-analytics.sh my-project agent-analytics my-content-bucket

# Create dataset and table
bq mk --dataset my-project:agent-analytics
bq mk --table agent-analytics.agent_events_v2 templates/bigquery-schema.json

IAM Requirements:

• roles/bigquery.jobUser - Required for BigQuery operations
• roles/bigquery.dataEditor - Required for writing data
• roles/storage.objectCreator - Required if using GCS offloading

Output: BigQuery table created, events streaming to dataset

3. Setup AgentOps

Script: scripts/setup-agentops.sh

Purpose: Configures AgentOps integration for session replays and metrics

Usage:

# Install AgentOps
pip install -U agentops

# Setup with API key
AGENTOPS_API_KEY=your_api_key_here ./scripts/setup-agentops.sh

# Verify setup
python -c "import agentops; agentops.init(); print('AgentOps ready')"

Environment Variables:

• AGENTOPS_API_KEY - AgentOps API key from app.agentops.ai/settings/projects

Output: AgentOps initialized, sessions visible in dashboard

4. Setup Phoenix

Script: scripts/setup-phoenix.sh

Purpose: Configures Phoenix (Arize) integration for open-source observability

Usage:

# Install Phoenix packages
pip install openinference-instrumentation-google-adk arize-phoenix-otel

# Setup Phoenix with API key
PHOENIX_API_KEY=your_key_here \
PHOENIX_COLLECTOR_ENDPOINT=https://app.phoenix.arize.com/s/your-space \
./scripts/setup-phoenix.sh

# Verify Phoenix connection
python scripts/verify-phoenix.py

Environment Variables:

• PHOENIX_API_KEY - Phoenix API key from phoenix.arize.com
• PHOENIX_COLLECTOR_ENDPOINT - Phoenix collector endpoint URL

Output: Phoenix tracer initialized, traces visible in Phoenix dashboard

5. Setup Weave

Script: scripts/setup-weave.sh <entity> <project>

Purpose: Configures Weave (W&B) integration for observability

Parameters:

• entity - W&B entity name (visible in Teams sidebar)
• project - W&B project name

Usage:

# Install Weave dependencies
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# Setup Weave with API key
WANDB_API_KEY=your_wandb_key_here ./scripts/setup-weave.sh my-team my-project

# Verify Weave connection
python scripts/verify-weave.py

Environment Variables:

• WANDB_API_KEY - W&B API key from wandb.ai/authorize

Output: Weave tracer initialized, traces visible in Weave dashboard

6. Validate Observability Setup

Script: scripts/validate-observability.sh

Purpose: Validates observability configuration and connectivity

Checks:

• Cloud Trace connectivity
• BigQuery dataset and table existence
• AgentOps initialization
• Phoenix endpoint reachability
• Weave endpoint reachability
• IAM permissions
• Environment variables set

Usage:

# Validate all observability configurations
./scripts/validate-observability.sh

# Validate specific tool
./scripts/validate-observability.sh --tool=bigquery
./scripts/validate-observability.sh --tool=cloud-trace
./scripts/validate-observability.sh --tool=agentops

Exit Codes:

• 0 - All checks passed
• 1 - Configuration missing
• 2 - Connectivity failed
• 3 - Permission issues

Available Templates

1. Cloud Trace Configuration

Template: templates/cloud-trace-config.py

Purpose: Cloud Trace integration for ADK agents

Features:

• OpenTelemetry configuration
• Automatic span creation for agent runs
• LLM and tool call tracing
• Error and latency tracking

Usage:

# Enable Cloud Trace via ADK CLI
adk deploy agent_engine --project=$GOOGLE_CLOUD_PROJECT --trace_to_cloud ./agent

# Or via Python SDK
from google.adk.app import AdkApp

app = AdkApp(
    agent=my_agent,
    enable_tracing=True
)

Span Labels:

• invocation - Top-level agent invocation
• agent_run - Individual agent execution
• call_llm - LLM API calls
• execute_tool - Tool executions

2. BigQuery Analytics Configuration

Template: templates/bigquery-analytics-config.py

Purpose: Complete BigQuery Agent Analytics plugin configuration

Features:

• Asynchronous event logging
• Multimodal content with GCS offloading
• OpenTelemetry-style tracing (trace_id, span_id)
• Event filtering and batching
• Custom content formatting

Usage:

from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin, BigQueryLoggerConfig
)

bq_config = BigQueryLoggerConfig(
    enabled=True,
    gcs_bucket_name="your-bucket-name",
    max_content_length=500 * 1024,  # 500KB inline limit
    batch_size=1,  # Low latency
    event_allowlist=["LLM_RESPONSE", "TOOL_COMPLETED"]
)

plugin = BigQueryAgentAnalyticsPlugin(
    project_id="your-project-id",
    dataset_id="your-dataset-id",
    config=bq_config
)

app = App(root_agent=agent, plugins=[plugin])

Configuration Options:

• enabled - Toggle logging on/off
• gcs_bucket_name - GCS bucket for large content
• max_content_length - Inline text limit (default 500KB)
• batch_size - Events per write (default 1)
• event_allowlist - Whitelist specific event types
• event_denylist - Blacklist specific event types
• content_formatter - Custom formatting function

3. BigQuery Schema

Template: templates/bigquery-schema.json

Purpose: BigQuery table schema for agent_events_v2

Schema Fields:

• timestamp - Event recording time
• event_type - Event category (LLM_REQUEST, TOOL_STARTING, etc.)
• content - Event-specific JSON payload
• content_parts - Structured multimodal data
• trace_id - OpenTelemetry trace ID
• span_id - OpenTelemetry span ID
• agent - Agent name
• user_id - User identifier

Partitioning: By DATE(timestamp) for cost optimization

Clustering: By event_type, agent, user_id for query performance

4. AgentOps Configuration

Template: templates/agentops-config.py

Purpose: AgentOps integration for session replays

Features:

• Minimal two-line integration
• Hierarchical span visualization
• LLM call tracking with prompts and completions
• Token count and latency metrics
• Cost tracking

Usage:

import agentops

# Initialize AgentOps (before ADK imports)
agentops.init()

# Your ADK agent code
from google.adk.app import App
app = App(root_agent=my_agent)

Span Hierarchy:

• Agent spans: Named adk.agent.{AgentName}
• LLM spans: Capture prompts, completions, tokens
• Tool spans: Record parameters and results

5. Phoenix Configuration

Template: templates/phoenix-config.py

Purpose: Phoenix (Arize) integration for open-source observability

Features:

• Self-hosted data control
• OpenInference instrumentation
• Trace evaluation
• Performance debugging
• Custom evaluators

Usage:

import os
from phoenix.otel import register

# Set Phoenix credentials
os.environ["PHOENIX_API_KEY"] = "your_api_key_here"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com/s/your-space"

# Register Phoenix tracer
tracer_provider = register(
    project_name="my-adk-agent",
    auto_instrument=True
)

# Your ADK agent code (Phoenix auto-captures traces)
from google.adk.app import App
app = App(root_agent=my_agent)

Auto-Instrumentation: Phoenix automatically traces all ADK operations

6. Weave Configuration

Template: templates/weave-config.py

Purpose: Weave (W&B) integration for observability

Features:

• Timeline of agent calls
• Tool invocation tracking
• Reasoning process analysis
• Span hierarchy visualization
• Dashboard integration

Usage:

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import base64

# Setup Weave exporter
wandb_api_key = os.environ["WANDB_API_KEY"]
entity = "your-entity"
project = "your-project"

auth_string = f"api:{wandb_api_key}"
encoded_auth = base64.b64encode(auth_string.encode()).decode()

exporter = OTLPSpanExporter(
    endpoint="https://trace.wandb.ai/otel/v1/traces",
    headers={
        "Authorization": f"Basic {encoded_auth}",
        "project_id": f"{entity}/{project}"
    }
)

# Configure tracer provider (BEFORE ADK imports)
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Your ADK agent code
from google.adk.app import App
app = App(root_agent=my_agent)

Critical: Set tracer provider before importing ADK components

Available Examples

1. Complete Observability Setup

Example: examples/complete-observability.md

Covers:

• Multi-tool observability setup
• Cloud Trace + BigQuery combination
• Third-party tool integration
• Production deployment patterns
• Cost optimization strategies

Step-by-Step Guide:

1. Enable Cloud Trace for distributed tracing
2. Configure BigQuery for event logging
3. Add AgentOps for session replays
4. Optional: Phoenix or Weave for additional insights
5. Validate all configurations
6. Deploy to production

Production Checklist:

• Cloud Trace enabled in production
• BigQuery dataset created with proper IAM
• GCS bucket configured for multimodal content
• Event filtering configured to control costs
• Alert rules defined for error rates
• Dashboard created for key metrics
• Retention policies set for cost control

2. BigQuery Analytics Queries

Example: examples/bigquery-queries.md

Covers:

• Conversation trace retrieval
• Token usage analysis
• Error rate tracking
• Tool usage statistics
• Performance metrics
• Cost analysis

Query Examples:

-- Retrieve conversation traces
SELECT timestamp, event_type, JSON_VALUE(content, '$.response')
FROM agent_events_v2
WHERE trace_id = 'your-trace-id'
ORDER BY timestamp ASC;

-- Token usage by agent
SELECT
  agent,
  AVG(CAST(JSON_VALUE(content, '$.usage.total') AS INT64)) as avg_tokens,
  SUM(CAST(JSON_VALUE(content, '$.usage.total') AS INT64)) as total_tokens
FROM agent_events_v2
WHERE event_type = 'LLM_RESPONSE'
GROUP BY agent;

-- Error rate by event type
SELECT
  event_type,
  COUNT(*) as error_count,
  DATE(timestamp) as day
FROM agent_events_v2
WHERE event_type LIKE '%ERROR%'
GROUP BY event_type, day
ORDER BY day DESC, error_count DESC;

-- Tool usage frequency
SELECT
  JSON_VALUE(content, '$.tool_name') as tool,
  COUNT(*) as usage_count
FROM agent_events_v2
WHERE event_type = 'TOOL_COMPLETED'
GROUP BY tool
ORDER BY usage_count DESC;

-- Access multimodal content from GCS
SELECT
  part.mime_type,
  part.object_ref.uri as gcs_uri
FROM agent_events_v2,
UNNEST(content_parts) AS part
WHERE part.storage_mode = 'GCS_REFERENCE';

3. Multi-Tool Integration

Example: examples/multi-tool-integration.md

Covers:

• Using multiple observability tools together
• Cloud Trace + BigQuery + AgentOps
• Data correlation across platforms
• Tool selection criteria
• Cost vs. insight tradeoffs

Integration Patterns:

Pattern 1: Google Cloud Native

• Cloud Trace for distributed tracing
• BigQuery for detailed event analysis
• Best for: GCP-centric deployments

Pattern 2: Comprehensive Monitoring

• Cloud Trace for infrastructure tracing
• AgentOps for session replays
• BigQuery for analytics
• Best for: Production monitoring with detailed debugging

Pattern 3: Open Source

• Phoenix for self-hosted observability
• BigQuery for long-term storage
• Best for: Data sovereignty requirements

Pattern 4: ML-Focused

• Weave for experiment tracking
• BigQuery for analytics
• Best for: Research and experimentation

4. Production Deployment

Example: examples/production-deployment.md

Covers:

• Production-ready observability configuration
• IAM role setup
• Cost optimization
• Alert configuration
• Dashboard creation
• Incident response

Production Setup:

1. IAM Configuration:

   • Service account with minimal permissions
   • Separate dev/staging/prod credentials
   • Workload Identity for GKE deployments

2. Cost Controls:

   • Event filtering to reduce BigQuery writes
   • GCS lifecycle policies for multimodal content
   • Table partitioning and clustering
   • Retention policies (30-90 days)

3. Monitoring:

   • Cloud Monitoring alerts for error rates
   • BigQuery query dashboard in Looker Studio
   • AgentOps session replay for debugging
   • Trace analysis for performance issues

4. Security:

   • No credentials in code (environment variables only)
   • VPC Service Controls for data protection
   • Customer-managed encryption keys (CMEK)
   • Audit logging for compliance

Security Compliance

CRITICAL: This skill follows strict security rules:

❌ NEVER hardcode:

• API keys (AgentOps, Phoenix, Weave, W&B)
• Google Cloud credentials
• Service account keys
• OAuth tokens
• BigQuery connection strings

✅ ALWAYS:

• Use environment variables for secrets
• Generate .env.example with placeholders
• Add .env* to .gitignore
• Use Google Application Default Credentials
• Document credential acquisition process
• Use IAM roles instead of service account keys when possible

Placeholder format:

# .env.example
GOOGLE_CLOUD_PROJECT=your-project-id
AGENTOPS_API_KEY=your_agentops_key_here
PHOENIX_API_KEY=your_phoenix_key_here
PHOENIX_COLLECTOR_ENDPOINT=https://app.phoenix.arize.com/s/your-space
WANDB_API_KEY=your_wandb_key_here

Progressive Disclosure

This skill provides immediate setup guidance with references to detailed documentation:

• Quick Start: Use setup scripts for immediate configuration
• Production: Reference production-deployment.md for complete guide
• Analytics: Use bigquery-queries.md for query templates
• Integration: Reference multi-tool-integration.md for advanced patterns

Load additional files only when specific customization is needed.

Common Workflows

1. Local Development Setup

# Enable Cloud Trace for local debugging
export GOOGLE_CLOUD_PROJECT=your-project-id
./scripts/setup-cloud-trace.sh your-project-id

# Start agent with tracing
python my_agent.py
# View traces at console.cloud.google.com/traces

2. Production Deployment with BigQuery

# 1. Create BigQuery dataset
bq mk --dataset my-project:agent-analytics

# 2. Create events table
bq mk --table agent-analytics.agent_events_v2 templates/bigquery-schema.json

# 3. Create GCS bucket for multimodal content
gsutil mb gs://my-agent-content/

# 4. Setup BigQuery analytics
./scripts/setup-bigquery-analytics.sh my-project agent-analytics my-agent-content

# 5. Deploy agent
adk deploy agent_engine --project=my-project ./agent

# 6. Validate setup
./scripts/validate-observability.sh --tool=bigquery

3. Multi-Tool Integration

# 1. Setup Cloud Trace
export GOOGLE_CLOUD_PROJECT=your-project-id
./scripts/setup-cloud-trace.sh your-project-id

# 2. Setup BigQuery Analytics
./scripts/setup-bigquery-analytics.sh your-project agent-analytics my-bucket

# 3. Setup AgentOps
export AGENTOPS_API_KEY=your_key_here
./scripts/setup-agentops.sh

# 4. Validate all configurations
./scripts/validate-observability.sh

Troubleshooting

Cloud Trace Not Showing Traces

Check:

• GOOGLE_CLOUD_PROJECT environment variable is set
• Cloud Trace API is enabled
• Service account has roles/cloudtrace.agent
• Tracer initialized before ADK imports

Debug:

# Check Cloud Trace API status
gcloud services list --enabled | grep cloudtrace

# Enable Cloud Trace API
gcloud services enable cloudtrace.googleapis.com

# Test trace export
python scripts/test-cloud-trace.py

BigQuery Events Not Appearing

Check:

• Dataset and table exist
• Service account has correct IAM roles
• BigQuery API is enabled
• Plugin configuration is correct
• No event filtering blocking events

Debug:

# Check dataset exists
bq ls my-project:

# Check table schema
bq show --schema agent-analytics.agent_events_v2

# Check IAM permissions
gcloud projects get-iam-policy my-project \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:YOUR_SA_EMAIL"

# Test plugin manually
python scripts/test-bigquery-plugin.py

AgentOps Not Capturing Traces

Check:

• AgentOps initialized before ADK imports
• API key is valid
• Network connectivity to app.agentops.ai
• AgentOps package version is latest

Fix:

# Update AgentOps
pip install -U agentops

# Test initialization
python -c "import agentops; agentops.init(); print('Success')"

# Check for conflicts with other tracers
# Ensure AgentOps is initialized first

Phoenix Connection Failed

Check:

• Phoenix API key is valid
• Collector endpoint URL is correct
• Network access to Phoenix endpoint
• Required packages installed

Debug:

# Test Phoenix endpoint
curl -H "Authorization: Bearer YOUR_KEY" \
  https://app.phoenix.arize.com/s/YOUR_SPACE

# Verify package versions
pip list | grep -E "(openinference|phoenix)"

# Run verification script
python scripts/verify-phoenix.py

Weave Traces Not Appearing

Check:

• Tracer provider set BEFORE ADK imports
• W&B API key is valid
• Entity and project names are correct
• OTEL exporter configured properly

Fix:

# Verify initialization order
# 1. Import OTEL packages
# 2. Configure and set tracer provider
# 3. THEN import ADK

# Correct order:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
trace.set_tracer_provider(TracerProvider())  # FIRST

from google.adk.app import App  # THEN

Dependencies

Required:

• google-adk>=1.21.0 - ADK framework (version 1.21.0+ for full BigQuery features)
• google-cloud-trace>=1.13.0 - Cloud Trace client (optional)
• google-cloud-bigquery>=3.0.0 - BigQuery client (optional)

Optional (Third-party tools):

• agentops>=0.3.0 - AgentOps integration
• openinference-instrumentation-google-adk>=0.1.0 - Phoenix instrumentation
• arize-phoenix-otel>=0.1.0 - Phoenix OTEL exporter
• opentelemetry-sdk>=1.20.0 - OpenTelemetry SDK for Weave
• opentelemetry-exporter-otlp-proto-http>=1.20.0 - OTLP exporter for Weave

Installation:

# Core ADK with Cloud Trace
pip install google-adk google-cloud-trace

# With BigQuery Analytics
pip install google-adk google-cloud-bigquery

# With AgentOps
pip install google-adk agentops

# With Phoenix
pip install google-adk openinference-instrumentation-google-adk arize-phoenix-otel

# With Weave
pip install google-adk opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# All observability tools
pip install google-adk google-cloud-trace google-cloud-bigquery agentops \
  openinference-instrumentation-google-adk arize-phoenix-otel \
  opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Best Practices

1. Multi-Layer Observability: Use Cloud Trace for infrastructure, BigQuery for analytics, and AgentOps for debugging
2. Cost Control: Implement event filtering and retention policies to manage BigQuery costs
3. Security: Never hardcode credentials; use environment variables and IAM roles
4. Progressive Rollout: Start with Cloud Trace, add BigQuery when analytics needed
5. Tool Selection: Choose tools based on requirements (open-source vs. managed, cost vs. features)
6. Data Correlation: Use trace_id across all tools for unified debugging
7. Alert Configuration: Set up alerts for error rates, latency spikes, and cost anomalies
8. Dashboard Creation: Build custom dashboards in Looker Studio, Grafana, or tool-native UIs

Additional Resources

• Cloud Trace: https://cloud.google.com/trace/docs
• BigQuery Agent Analytics: https://google.github.io/adk-docs/observability/bigquery-agent-analytics/
• AgentOps: https://app.agentops.ai/
• Phoenix (Arize): https://arize.com/docs/phoenix/
• Weave (W&B): https://docs.wandb.ai/weave/
• ADK Observability Guide: https://google.github.io/adk-docs/observability/
• OpenTelemetry: https://opentelemetry.io/docs/

Tool Comparison