Sf AI Agentforce Observability

sf-ai-agentforce-observability: Agentforce Session Tracing Extraction & Analysis

Expert in extracting and analyzing Agentforce session tracing data from Salesforce Data 360. Supports high-volume data extraction (1-10M records/day), Parquet storage, and Polars-based analysis for debugging agent behavior.

Core Responsibilities

1. Session Extraction: Extract STDM (Session Tracing Data Model) data via Data 360 Query API
2. Data Storage: Write to Parquet format with PyArrow for efficient storage
3. Analysis: Polars-based lazy evaluation for memory-efficient analysis
4. Debugging: Session timeline reconstruction for troubleshooting agent issues
5. Cross-Skill Integration: Works with sf-connected-apps for auth, sf-ai-agentscript for fixes

Document Map

Quick Links:

• Data Model Overview
• CLI Quick Reference
• Analysis Examples
• Cross-Skill Integration

CRITICAL: Prerequisites Checklist

Before extracting session data, verify:

Official Setup Guide: Set Up Agentforce Session Tracing

Auth Setup (via sf-connected-apps)

# 1. Create key directory
mkdir -p ~/.sf/jwt

# 2. Generate certificate (naming convention: {org}-agentforce-observability)
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout ~/.sf/jwt/myorg-agentforce-observability.key \
  -out ~/.sf/jwt/myorg-agentforce-observability.crt \
  -subj "/CN=AgentforceObservability/O=MyOrg"

# 3. Secure the private key
chmod 600 ~/.sf/jwt/myorg-agentforce-observability.key

# 4. Create External Client App in Salesforce (see docs/auth-setup.md)
# Required scopes: cdp_query_api, refresh_token/offline_access

Key Path Resolution Order:

1. Explicit --key-path argument
2. App-specific: ~/.sf/jwt/{org}-agentforce-observability.key
3. Generic fallback: ~/.sf/jwt/{org}.key

See docs/auth-setup.md for detailed instructions.

T6 Live API Discovery Summary ✅

Validated: January 30, 2026 | 24 DMOs Found | 260+ Test Points

Key Discoveries:

• Field naming: API uses AiAgent (lowercase 'i'), not AIAgent
• Agent name location: Stored on Moment, not Session
• Channel types: E & O, Builder, SCRT2 - EmbeddedMessaging, Voice, NGC, Builder: Voice Preview
• Agent types: EinsteinServiceAgent, AgentforceEmployeeAgent, AgentforceServiceAgent, Employee
• Participant roles: USER, AGENT (not Owner/Observer)
• GenAI detectors: TOXICITY (9 categories), PII (4 types), PROMPT_DEFENSE, InstructionAdherence

Session Tracing Data Model (STDM)

The STDM consists of 5 core DMOs plus 13 GenAI Audit DMOs. Important: Field names use AiAgent (lowercase 'i'), not AIAgent.

ssot__AIAgentSession__dlm (SESSION)
├── ssot__Id__c                          # Session ID (UUID)
├── ssot__StartTimestamp__c              # Session start (TimestampTZ)
├── ssot__EndTimestamp__c                # Session end (TimestampTZ)
├── ssot__AiAgentSessionEndType__c       # End type (Completed, Abandoned, etc.)
├── ssot__AiAgentChannelType__c          # Channel (PSTN, Messaging, etc.)
├── ssot__RelatedMessagingSessionId__c   # Linked messaging session
├── ssot__RelatedVoiceCallId__c          # Linked voice call
├── ssot__SessionOwnerId__c              # Owner ID
├── ssot__IndividualId__c                # Data Cloud individual
└── ssot__InternalOrganizationId__c      # Org ID

    └── ssot__AIAgentInteraction__dlm (TURN)  [1:N]
        ├── ssot__Id__c                          # Interaction ID
        ├── ssot__AiAgentSessionId__c            # FK to Session
        ├── ssot__AiAgentInteractionType__c      # TURN or SESSION_END
        ├── ssot__TopicApiName__c                # Topic that handled this turn
        ├── ssot__StartTimestamp__c              # Turn start
        ├── ssot__EndTimestamp__c                # Turn end
        ├── ssot__TelemetryTraceId__c            # Trace ID for debugging
        └── ssot__TelemetryTraceSpanId__c        # Span ID for debugging

            └── ssot__AIAgentInteractionStep__dlm (STEP)  [1:N]
                ├── ssot__Id__c                          # Step ID
                ├── ssot__AiAgentInteractionId__c        # FK to Interaction
                ├── ssot__AiAgentInteractionStepType__c  # LLM_STEP or ACTION_STEP
                ├── ssot__Name__c                        # Action/step name
                ├── ssot__InputValueText__c              # Input to step (JSON)
                ├── ssot__OutputValueText__c             # Output from step (JSON)
                ├── ssot__ErrorMessageText__c            # Error if step failed
                ├── ssot__PreStepVariableText__c         # Variables before
                ├── ssot__PostStepVariableText__c        # Variables after
                ├── ssot__GenerationId__c                # LLM generation ID
                └── ssot__GenAiGatewayRequestId__c       # GenAI Gateway request

ssot__AIAgentMoment__dlm (MOMENT - links to Session, not Interaction)
├── ssot__Id__c                          # Moment ID
├── ssot__AiAgentSessionId__c            # FK to Session (NOT interaction!)
├── ssot__AiAgentApiName__c              # Agent API name (lives here!)
├── ssot__AiAgentVersionApiName__c       # Agent version
├── ssot__RequestSummaryText__c          # User request summary
├── ssot__ResponseSummaryText__c         # Agent response summary
├── ssot__StartTimestamp__c              # Moment start
└── ssot__EndTimestamp__c                # Moment end

Key Schema Notes:

• Agent API name is in AIAgentMoment, not AIAgentSession
• Moments link to sessions via AiAgentSessionId, not interactions
• All field names use AiAgent prefix (lowercase 'i')

GenAI Trust Layer DMOs (13) ✅ T6 Verified

GenAIGatewayRequest__dlm (30 fields) - LLM request details
├── gatewayRequestId__c, prompt__c, maskedPrompt__c
├── model__c, provider__c, temperature__c
├── promptTokens__c, completionTokens__c, totalTokens__c
├── enableInputSafetyScoring__c, enableOutputSafetyScoring__c, enablePiiMasking__c
└── sessionId__c, userId__c, appType__c, feature__c

GenAIGeneration__dlm (11 fields) - LLM output
├── generationId__c (FK for Steps)
├── responseText__c, maskedResponseText__c
└── Links to: AIAgentInteractionStep.ssot__GenerationId__c

GenAIContentQuality__dlm (10 fields) - Trust Layer assessment
└── isToxicityDetected__c, parent__c (FK to Generation)

GenAIContentCategory__dlm (10 fields) - Detector results
├── detectorType__c: TOXICITY | PII | PROMPT_DEFENSE | InstructionAdherence
├── category__c: hate, identity, CREDIT_CARD, EMAIL_ADDRESS, High, Low, etc.
└── value__c: Confidence score (0.0-1.0)

GenAIFeedback__dlm (16 fields) - User feedback
├── feedback__c: GOOD | BAD
└── GenAIFeedbackDetail__dlm (10 fields) - Free-text comments

Detector Categories (Live API Verified):

See resources/data-model-reference.md for full field documentation.

Workflow (5-Phase Pattern)

Phase 1: Requirements Gathering

Use AskUserQuestion to gather:

Phase 2: Auth Configuration

Verify JWT auth is configured:

from scripts.auth import Data360Auth

auth = Data360Auth(
    org_alias="myorg",
    consumer_key="YOUR_CONSUMER_KEY"
)

# Test authentication
token = auth.get_token()
print(f"Auth successful: {token[:20]}...")

If auth fails, invoke:

Skill(skill="sf-connected-apps", args="Setup JWT Bearer for Data 360")

Phase 3: Extraction

Basic Extraction (last 7 days):

python3 scripts/cli.py extract \
  --org prod \
  --days 7 \
  --output ./stdm_data

Filtered Extraction:

python3 scripts/cli.py extract \
  --org prod \
  --since 2026-01-01 \
  --until 2026-01-28 \
  --agent Customer_Support_Agent \
  --output ./stdm_data

Session Tree (specific session):

python3 scripts/cli.py extract-tree \
  --org prod \
  --session-id "a0x..." \
  --output ./debug_session

Phase 4: Analysis

Session Summary:

from scripts.analyzer import STDMAnalyzer
from pathlib import Path

analyzer = STDMAnalyzer(Path("./stdm_data"))

# High-level summary
summary = analyzer.session_summary()
print(summary)

# Step distribution by agent
steps = analyzer.step_distribution(agent_name="Customer_Support_Agent")
print(steps)

# Topic routing analysis
topics = analyzer.topic_analysis()
print(topics)

Debug Specific Session:

python3 scripts/cli.py debug-session \
  --data-dir ./stdm_data \
  --session-id "a0x..."

Phase 5: Integration & Next Steps

Based on analysis findings:

CLI Quick Reference

Extraction Commands

Analysis Commands

Common Flags

See docs/cli-reference.md for complete documentation.

Analysis Examples

Session Summary

📊 SESSION SUMMARY
════════════════════════════════════════════════════════════════

Period: 2026-01-21 to 2026-01-28
Total Sessions: 15,234
Unique Agents: 3

SESSIONS BY AGENT
────────────────────────────────────────────────────────────────
Agent                          │ Sessions │ Avg Turns │ Avg Duration
───────────────────────────────┼──────────┼───────────┼─────────────
Customer_Support_Agent         │   8,502  │    4.2    │     3m 15s
Order_Tracking_Agent           │   4,128  │    2.8    │     1m 45s
Product_FAQ_Agent              │   2,604  │    1.9    │       45s

END TYPE DISTRIBUTION
────────────────────────────────────────────────────────────────
✅ Completed:    12,890 (84.6%)
🔄 Escalated:     1,523 (10.0%)
❌ Abandoned:       821 (5.4%)

Debug Session Timeline

🔍 SESSION DEBUG: a0x1234567890ABC
════════════════════════════════════════════════════════════════

Agent: Customer_Support_Agent
Started: 2026-01-28 10:15:23 UTC
Duration: 4m 32s
End Type: Completed
Turns: 5

TIMELINE
────────────────────────────────────────────────────────────────
10:15:23 │ [INPUT]  "I need help with my order #12345"
10:15:24 │ [TOPIC]  → Order_Tracking (confidence: 0.95)
10:15:24 │ [STEP]   LLM_STEP: Identify intent
10:15:25 │ [STEP]   ACTION_STEP: Get_Order_Status
         │          Input: {"orderId": "12345"}
         │          Output: {"status": "Shipped", "eta": "2026-01-30"}
10:15:26 │ [OUTPUT] "Your order #12345 has shipped and will arrive by Jan 30."

10:16:01 │ [INPUT]  "Can I change the delivery address?"
10:16:02 │ [TOPIC]  → Order_Tracking (same topic)
10:16:02 │ [STEP]   LLM_STEP: Clarify request
10:16:03 │ [STEP]   ACTION_STEP: Check_Modification_Eligibility
         │          Input: {"orderId": "12345", "type": "address_change"}
         │          Output: {"eligible": false, "reason": "Already shipped"}
10:16:04 │ [OUTPUT] "I'm sorry, the order has already shipped..."

Cross-Skill Integration

Prerequisite Skills

Follow-up Skills

Commonly Used With

Key Insights

Billing Considerations

Reference: Billing Considerations for Agentforce Session Tracing

Agentforce Session Tracing consumes Data 360 credits for ingestion, storage, and processing.

Credit Consumption

Cost Estimation

Records per session ≈ Turns × 24 (avg per LLM call)
Daily records ≈ Sessions/day × Avg turns × 24

Example:
  1,000 sessions/day × 4 turns × 24 = 96,000 records/day ingested

Tip: Use Digital Wallet for near real-time consumption tracking.

Common Issues & Fixes

See resources/troubleshooting.md for detailed solutions.

Output Directory Structure

After extraction:

stdm_data/
├── sessions/
│   └── date=2026-01-28/
│       └── part-0000.parquet
├── interactions/
│   └── date=2026-01-28/
│       └── part-0000.parquet
├── steps/
│   └── date=2026-01-28/
│       └── part-0000.parquet
├── messages/
│   └── date=2026-01-28/
│       └── part-0000.parquet
└── metadata/
    ├── extraction.json      # Extraction parameters
    └── watermark.json       # For incremental extraction

Dependencies

Python 3.10+ with:

polars>=1.0.0           # DataFrame library (lazy evaluation)
pyarrow>=15.0.0         # Parquet support
pyjwt>=2.8.0            # JWT generation
cryptography>=42.0.0    # Certificate handling
httpx>=0.27.0           # HTTP client
rich>=13.0.0            # CLI progress bars
click>=8.1.0            # CLI framework
pydantic>=2.6.0         # Data validation

Install: pip install -r requirements.txt

License

MIT License. See LICENSE file. Copyright (c) 2024-2026 Jag Valaiyapathy