Debugging & Observability Guide

Debugging & Observability Guide

Find the Leak Before It Finds You

The Debugging Workflow

┌─────────────────────┐      ┌─────────────────────┐      ┌─────────────────────┐
│ 📋 Interaction      │  →   │ 📊 Trace            │  →   │ ⚙️ Find the         │
│    Details          │      │    Waterfall        │      │    Leak             │
└─────────────────────┘      └─────────────────────┘      └─────────────────────┘

The 4 Debugging Views

View 1: Interaction Details (Summary)

Purpose: High-level chronological list with AI-generated summaries

Shows:

• ✅ Input received from user
• ✅ Reasoning decisions made
• ✅ Actions executed
• ✅ Output evaluation results

Best For: Quickly understanding WHAT happened

View 2: Trace Waterfall (Technical)

Purpose: Granular view showing every internal step

Shows:

• ✅ Exact prompt sent to the LLM
• ✅ Latency for each span (milliseconds)
• ✅ Raw JSON input/output for every tool call
• ✅ Variable state at each step

Best For: Understanding WHY something happened

View 3: Variable State Tracking

Purpose: Real-time table of variable Entry vs Exit values

Shows:

• ✅ Which variables changed during each span
• ✅ Critical security variables (verified, customer_id)
• ✅ Values LLM used vs values it should have used

Best For: Finding when LLM ignored variable state ("goal drift")

View 4: Script View with Linting

Purpose: Agent Script code with real-time syntax validation

Shows:

• ✅ Block ordering errors
• ✅ Indentation issues
• ✅ Missing required fields
• ✅ Invalid resource references

Best For: Catching errors before deployment

View Selection Guide

The 6 Span Types

Reading a Trace Waterfall

Example Timeline

SPAN                    DURATION    TIMELINE
────────────────────────────────────────────────────────────────────
➡️ Topic Enter          15ms        ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▶ before_reasoning      850ms       ████████████████████░░░░░░░░░░░░
🧠 reasoning            1250ms      ██████████████████████████████░░
⚡ Action Call          450ms       ██████████████░░░░░░░░░░░░░░░░░░

Latency Benchmarks

Variable State Analysis

Entry vs Exit Pattern

KEY INSIGHT: If a critical variable like is_verified was False when an action executed, you've found your leak point.

Common Debug Patterns

Pattern 1: Wrong Policy Applied

Symptom: Customer received wrong regional policy

Trace Analysis:

1. Check Variable State → CustomerCountry was empty at filter step
2. Check variable declaration → mutable string = ""
3. Root Cause: Should be linked string with source: @session.Country

Fix:

# Wrong
CustomerCountry: mutable string = ""

# Correct
CustomerCountry: linked string
  source: @session.Country

Pattern 2: Action Executed Without Authorization

Symptom: Refund processed without identity verification

Trace Analysis:

1. Check reasoning span → LLM selected process_refund
2. Check action definition → No available when guard
3. Root Cause: LLM could see and select unguarded action

Fix:

# Wrong - no guard
process_refund: @actions.process_refund
  description: "Issue refund"

# Correct - guarded
process_refund: @actions.process_refund
  description: "Issue refund"
  available when @variables.customer_verified == True

Pattern 3: Post-Action Logic Didn't Run

Symptom: CRM case wasn't created after refund approval

Trace Analysis:

1. Check instruction resolution order → Post-action check at bottom
2. Check transition → Topic transitioned before check could run
3. Root Cause: Post-action check must be at TOP

Fix:

# Wrong - check at bottom
instructions: ->
  | Help with refund.
  transition to @topic.next

  if @variables.refund_done:  # Never reaches here!
    run @actions.log_refund

# Correct - check at TOP
instructions: ->
  if @variables.refund_done:
    run @actions.log_refund
    transition to @topic.success

  | Help with refund.

Pattern 4: Infinite Loop

Symptom: Agent keeps returning to same topic

Trace Analysis:

1. Check transitions → topic_enter repeating for same topic
2. Check conditions → No exit condition defined
3. Root Cause: Missing state change or exit condition

Fix:

# Wrong - no exit condition
instructions: ->
  | Continue processing.

# Correct - track state, add exit
instructions: ->
  if @variables.processing_complete == True:
    transition to @topic.done

  | Continue processing.
  set @variables.step_count = @variables.step_count + 1

Pattern 5: LLM Ignores Variable State (Goal Drift)

Symptom: LLM makes decision contradicting variable value

Trace Analysis:

1. Check Variable State → Variable had correct value
2. Check resolved instructions → Condition should have pruned text
3. Root Cause: Using pipe syntax (|) instead of arrow (->)

Fix:

# Wrong - pipe doesn't support conditionals
instructions: |
  if @variables.verified:
    Help the user.
  # This is literal text, not a condition!

# Correct - arrow enables conditionals
instructions: ->
  if @variables.verified:
    | Help the user.
  else:
    | Please verify first.

Diagnostic Checklist

Quick Triage

Deep Investigation

The Big Picture

"Prompts are suggestions. Guards are guarantees."

The LLM might ignore your instructions. The only way to truly prevent unwanted behavior is through deterministic guards like available when.

When you remove an action from the LLM's toolkit, it literally cannot invoke it. That's not a suggestion - that's enforcement.

Key Takeaways

Planner Engine Differences

Salesforce has multiple planner engines. Behavior differs between them, which affects debugging.

Identifying Your Planner

Check Setup > Agentforce > Agent Settings to see which planner engine is active. Behavior differences between planners are the most common source of "it worked in dev but not in prod" issues.

Debugging Tips by Planner

Java Planner:

• More verbose trace output — look for detailed span data
• Lightning components render — test rich UI interactions
• Citations appear in responses — verify citation accuracy

Atlas/Daisy Planner:

• Trace output may be more concise — focus on variable state changes
• Lightning components won't render — test text-based fallbacks
• Context variables may behave differently — verify $Context access patterns