Sf AI Agentforce Observability

sf-ai-agentforce-observability: Agentforce Session Tracing Extraction & Analysis

Expert in extracting and analyzing Agentforce session tracing data from Salesforce Data 360. Supports high-volume data extraction (1-10M records/day), Parquet storage, and Polars-based analysis for debugging agent behavior.

Core Responsibilities

1. Session Extraction: Extract STDM (Session Tracing Data Model) data via Data 360 Query API
2. Data Storage: Write to Parquet format with PyArrow for efficient storage
3. Analysis: Polars-based lazy evaluation for memory-efficient analysis
4. Debugging: Session timeline reconstruction for troubleshooting agent issues
5. Cross-Skill Integration: Works with sf-connected-apps for auth, sf-ai-agentscript for fixes

Document Map

CRITICAL: Prerequisites Checklist

Before extracting session data, verify:

Official Setup Guide: Set Up Agentforce Session Tracing

Auth Setup (via sf-connected-apps)

# 1. Create key directory
mkdir -p ~/.sf/jwt

# 2. Generate certificate (naming convention: {org}-agentforce-observability)
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout ~/.sf/jwt/myorg-agentforce-observability.key \
  -out ~/.sf/jwt/myorg-agentforce-observability.crt \
  -subj "/CN=AgentforceObservability/O=MyOrg"

# 3. Secure the private key
chmod 600 ~/.sf/jwt/myorg-agentforce-observability.key

# 4. Create External Client App in Salesforce (see docs/auth-setup.md)
# Required scopes: cdp_query_api, refresh_token/offline_access

Key Path Resolution Order:

1. Explicit --key-path argument
2. App-specific: ~/.sf/jwt/{org}-agentforce-observability.key
3. Generic fallback: ~/.sf/jwt/{org}.key

See docs/auth-setup.md for detailed instructions.

T6 Live API Discovery Summary

Validated: January 30, 2026 | 24 DMOs Found | 260+ Test Points

Key Discoveries:

• Field naming: API uses AiAgent (lowercase 'i'), not AIAgent
• Agent name location: Stored on Moment, not Session
• Channel types: E & O, Builder, SCRT2 - EmbeddedMessaging, Voice, NGC
• Participant roles: USER, AGENT (not Owner/Observer)

Session Tracing Data Model (STDM)

See resources/data-model-reference.md for the complete field-level schema of all 5 core DMOs + 13 GenAI Audit DMOs.

5 Core DMOs — all field names use AiAgent prefix (lowercase 'i'):

13 GenAI Trust Layer DMOs — detectors for toxicity, PII, prompt defense, instruction adherence:

Workflow (5-Phase Pattern)

Phase 1: Requirements Gathering

Use AskUserQuestion to gather:

Phase 2: Auth Configuration

Verify JWT auth is configured:

from scripts.auth import Data360Auth

auth = Data360Auth(
    org_alias="myorg",
    consumer_key="YOUR_CONSUMER_KEY"
)

# Test authentication
token = auth.get_token()
print(f"Auth successful: {token[:20]}...")

If auth fails, invoke:

Skill(skill="sf-connected-apps", args="Setup JWT Bearer for Data 360")

Phase 3: Extraction

Basic Extraction (last 7 days):

python3 scripts/cli.py extract \
  --org prod \
  --days 7 \
  --output ./stdm_data

Filtered Extraction:

python3 scripts/cli.py extract \
  --org prod \
  --since 2026-01-01 \
  --until 2026-01-28 \
  --agent Customer_Support_Agent \
  --output ./stdm_data

Session Tree (specific session):

python3 scripts/cli.py extract-tree \
  --org prod \
  --session-id "a0x..." \
  --output ./debug_session

Phase 4: Analysis

Session Summary:

from scripts.analyzer import STDMAnalyzer
from pathlib import Path

analyzer = STDMAnalyzer(Path("./stdm_data"))

# High-level summary
summary = analyzer.session_summary()
print(summary)

# Step distribution by agent
steps = analyzer.step_distribution(agent_name="Customer_Support_Agent")
print(steps)

# Topic routing analysis
topics = analyzer.topic_analysis()
print(topics)

Debug Specific Session:

python3 scripts/cli.py debug-session \
  --data-dir ./stdm_data \
  --session-id "a0x..."

Phase 5: Integration & Next Steps

Based on analysis findings:

CLI Quick Reference

Extraction Commands

Analysis Commands

Common Flags

See docs/cli-reference.md for complete documentation.

Analysis Examples

See references/analysis-examples.md for full session summary and debug timeline output examples.

Session Summary: Shows sessions by agent (count, avg turns, avg duration) and end type distribution (completed/escalated/abandoned).

Debug Timeline: Reconstructs a session step-by-step — input → topic routing → LLM steps → action steps → output — with timestamps and I/O payloads.

Cross-Skill Integration

Key Insights

Billing & Common Issues

See references/billing-and-troubleshooting.md for credit consumption details and error resolution.

Quick reference: Session Tracing consumes Data 360 credits. ~24 records per LLM round-trip. 1,000 sessions/day × 4 turns × 24 = ~96K records/day. Use Digital Wallet for consumption tracking.

Output Directory Structure

stdm_data/
├── sessions/          # date=YYYY-MM-DD/part-0000.parquet
├── interactions/      # date=YYYY-MM-DD/part-0000.parquet
├── steps/             # date=YYYY-MM-DD/part-0000.parquet
├── messages/          # date=YYYY-MM-DD/part-0000.parquet
└── metadata/          # extraction.json + watermark.json (incremental)

Dependencies

Python 3.10+: polars>=1.0, pyarrow>=15.0, pyjwt>=2.8, cryptography>=42.0, httpx>=0.27, rich>=13.0, click>=8.1, pydantic>=2.6

Install: pip install -r requirements.txt