Sf AI Agentforce Testing

sf-ai-agentforce-testing: Agentforce Test Execution & Coverage Analysis

Expert testing engineer specializing in Agentforce agent testing, topic/action coverage analysis, and agentic fix loops. Execute agent tests, analyze failures, and automatically fix issues via sf-ai-agentforce.

Core Responsibilities

1. Test Execution: Run agent tests via sf agent test run with coverage analysis
2. Test Spec Generation: Create YAML test specifications for agents
3. Coverage Analysis: Track topic selection accuracy, action invocation rates
4. Preview Testing: Interactive simulated and live agent testing
5. Agentic Fix Loop: Automatically fix failing agents and re-test
6. Cross-Skill Orchestration: Delegate fixes to sf-ai-agentforce, data to sf-data

📚 Document Map

⚡ Quick Links:

• Scoring System - 5-category validation
• CLI Command Reference - Essential commands
• Agentic Fix Loop - Auto-fix workflow
• Test Spec Reference - Complete YAML format guide
• Automated Testing - Python scripts and workflows

⚠️ CRITICAL: Orchestration Order

sf-metadata → sf-apex → sf-flow → sf-deploy → sf-ai-agentforce → sf-deploy → sf-ai-agentforce-testing (you are here)

Why testing is LAST:

1. Agent must be published before running automated tests
2. Agent must be activated for preview mode
3. All dependencies (Flows, Apex) must be deployed first
4. Test data (via sf-data) should exist before testing actions

⚠️ MANDATORY Delegation:

• Fixes: ALWAYS use Skill(skill="sf-ai-agentforce") for agent script fixes
• Test Data: Use Skill(skill="sf-data") for action test data
• OAuth Setup: Use Skill(skill="sf-connected-apps") for live preview

⚠️ CRITICAL: Org Requirements (Agent Testing Center)

Agent testing requires the Agent Testing Center feature, which is NOT enabled by default in all orgs.

Check if Agent Testing Center is Enabled

# This will fail if Agent Testing Center is not enabled
sf agent test list --target-org [alias]

# Expected errors if NOT enabled:
# "Not available for deploy for this organization"
# "INVALID_TYPE: Cannot use: AiEvaluationDefinition in this organization"

Orgs WITHOUT Agent Testing Center

Enabling Agent Testing Center

1. Scratch Org - Add to scratch-def.json:

   {
     "features": ["AgentTestingCenter", "EinsteinGPTForSalesforce"]
   }
   

2. Production/Sandbox - Contact Salesforce to enable the feature

3. Fallback - Use sf agent preview for manual testing (see Automated Testing Guide)

⚠️ CRITICAL: Prerequisites Checklist

Before running agent tests, verify:

Workflow (6-Phase Pattern)

Phase 1: Prerequisites

Use AskUserQuestion to gather:

• Agent name/API name
• Target org alias
• Test mode (simulated vs live)
• Coverage threshold (default: 80%)
• Enable agentic fix loop?

Then:

1. Verify agent is published and activated
2. Check for existing test specs: Glob: **/*.yaml, Glob: **/tests/*.yaml
3. Create TodoWrite tasks

Phase 2: Test Spec Creation

Option A: Interactive Generation (no automation available)

# Interactive test spec generation
sf agent generate test-spec --output-file ./tests/agent-spec.yaml

# ⚠️ NOTE: There is NO --api-name flag! The command is interactive-only.

Option B: Automated Generation (Python script)

# Generate from agent file
python3 hooks/scripts/generate-test-spec.py \
  --agent-file /path/to/Agent.agent \
  --output tests/agent-spec.yaml \
  --verbose

See Test Spec Reference for complete YAML format guide.

Create Test in Org:

sf agent test create --spec ./tests/agent-spec.yaml --api-name MyAgentTest --target-org [alias]

Phase 3: Test Execution

Automated Tests:

sf agent test run --api-name MyAgentTest --wait 10 --result-format json --target-org [alias]

Interactive Preview (Simulated):

sf agent preview --api-name AgentName --output-dir ./logs --target-org [alias]

Interactive Preview (Live):

sf agent preview --api-name AgentName --use-live-actions --client-app AppName --apex-debug --target-org [alias]

Phase 4: Results Analysis

Parse test results JSON and display formatted summary:

📊 AGENT TEST RESULTS
════════════════════════════════════════════════════════════════

Agent: Customer_Support_Agent
Org: my-sandbox
Duration: 45.2s
Mode: Simulated

SUMMARY
───────────────────────────────────────────────────────────────
✅ Passed:    18
❌ Failed:    2
⏭️ Skipped:   0
📈 Topic Selection: 95%
🎯 Action Invocation: 90%

FAILED TESTS
───────────────────────────────────────────────────────────────
❌ test_complex_order_inquiry
   Utterance: "What's the status of orders 12345 and 67890?"
   Expected: get_order_status invoked 2 times
   Actual: get_order_status invoked 1 time
   Category: ACTION_INVOCATION_COUNT_MISMATCH

COVERAGE SUMMARY
───────────────────────────────────────────────────────────────
Topics Tested:       4/5 (80%) ⚠️
Actions Tested:      6/8 (75%) ⚠️
Guardrails Tested:   3/3 (100%) ✅

Phase 5: Agentic Fix Loop

When tests fail, automatically fix via sf-ai-agentforce:

Auto-Fix Command Example:

Skill(skill="sf-ai-agentforce", args="Fix agent [AgentName] - Error: [category] - [details]")

See Agentic Fix Loops Guide for:

• Complete decision tree
• Detailed fix strategies for each error type
• Cross-skill orchestration workflow
• Python scripts for automated testing
• Example fix loop executions

Phase 6: Coverage Improvement

If coverage < threshold:

1. Identify untested topics/actions from results
2. Add test cases to spec YAML
3. Update test: sf agent test create --spec ./tests/agent-spec.yaml --force-overwrite
4. Re-run: sf agent test run --api-name MyAgentTest --wait 10

Scoring System (100 Points)

Scoring Thresholds:

⭐⭐⭐⭐⭐ 90-100 pts → Production Ready
⭐⭐⭐⭐   80-89 pts → Good, minor improvements
⭐⭐⭐    70-79 pts → Acceptable, needs work
⭐⭐      60-69 pts → Below standard
⭐        <60 pts  → BLOCKED - Major issues

⛔ TESTING GUARDRAILS (MANDATORY)

BEFORE running tests, verify:

NEVER do these:

CLI Command Reference

Test Lifecycle Commands

Preview Commands

Result Formats

Test Spec Quick Reference

Basic Template:

subjectType: AGENT
subjectName: <Agent_Name>

testCases:
  # Topic routing
  - utterance: "What's on your menu?"
    expectation:
      topic: product_faq
      actionSequence: []

  # Action invocation
  - utterance: "Search for Harry Potter books"
    expectation:
      topic: book_search
      actionSequence:
        - search_catalog

  # Edge case
  - utterance: ""
    expectation:
      graceful_handling: true

For complete YAML format reference, see Test Spec Reference

Cross-Skill Integration

Required Delegations:

For complete orchestration workflow, see Agentic Fix Loops

Automated Testing (Python Scripts)

This skill includes Python scripts for fully automated agent testing:

Quick Usage:

# Generate test spec from agent file
python3 hooks/scripts/generate-test-spec.py \
  --agent-file /path/to/Agent.agent \
  --output specs/Agent-tests.yaml

# Run full automated workflow
python3 hooks/scripts/run-automated-tests.py \
  --agent-name MyAgent \
  --agent-dir /path/to/project \
  --target-org dev

For complete documentation, see Agentic Fix Loops Guide

Templates Reference

🐛 Known Issues & CLI Bugs

Last Updated: 2026-01-04 | Tested With: sf CLI v2.118.16

CRITICAL: sf agent test create MasterLabel Bug

Status: 🔴 BLOCKING - Prevents YAML-based test creation

Error:

Error (SfError): Required fields are missing: [MasterLabel]

Root Cause: The CLI generates XML from YAML but doesn't include the required name element (MasterLabel) in the AiEvaluationDefinition metadata.

Workarounds:

1. Use sf agent generate test-spec (interactive mode) - this works correctly
2. Create tests via Salesforce Testing Center UI
3. Deploy XML metadata directly (bypassing YAML conversion)

Reproduction:

# Create minimal test spec
cat > test.yaml << 'EOF'
subjectType: AGENT
subjectName: My_Agent
testCases:
  - utterance: "Hello"
    expectation:
      topic: Welcome
      actionSequence: []
EOF

# This will fail:
sf agent test create --spec test.yaml --api-name MyTest --target-org dev
# Error: Required fields are missing: [MasterLabel]

YAML vs XML Format Discrepancy

Issue: YAML test specs use different field names than the actual XML metadata.

Impact: Tests created via YAML may have empty or mismatched expectations.

Topic Expectation Not Populated

Issue: When YAML includes topic: TopicName, the generated XML has empty expectedValue:

<!-- Generated (BROKEN) -->
<expectation>
    <name>topic_assertion</name>
    <!-- Missing expectedValue! -->
</expectation>

<!-- Should be -->
<expectation>
    <name>topic_assertion</name>
    <expectedValue>TopicName</expectedValue>
</expectation>

Agent Metadata Structure

Understanding agent components - Agent data is distributed across multiple metadata types:

Bot (Agent Definition)
 └── BotVersion
      └── genAiPlannerName → GenAiPlannerBundle
                              └── (references GenAiFunction for topics)

Retrieve all components:
  sf project retrieve start --metadata "Bot:AgentName"
  sf project retrieve start --metadata "GenAiPlannerBundle:AgentName_v1"

Note: BotDefinition is NOT queryable via Tooling API SOQL. Use metadata retrieval instead.

💡 Key Insights

🔄 Two Fix Strategies

When agent tests fail, there are TWO valid approaches:

Decision Flow:

Test Failed → Can you modify the agent?
                    │
          ┌────────┴────────┐
          ↓                 ↓
         YES                NO
          ↓                 ↓
    Fix Agent          Fix Test Spec
    (sf-ai-agentforce)  (update YAML)

Example: Fixing Test Expectations

# BEFORE (wrong expectations)
expectedTopic: GeneralCRM
expectedActions:
  - IdentifyRecordByName
  - GetRecordDetails

# AFTER (matches actual behavior)
expectedTopic: MigrationDefaultTopic
expectedActions:
  - IdentifyRecordByName
  - QueryRecords

🔄 Automated Test-Fix Loop

NEW in v1.1.0 | Claude Code can now orchestrate fully automated test-fix cycles

Overview

The test-fix loop enables Claude Code to:

1. Run tests → sf agent test run with JSON output
2. Analyze failures → Parse results and categorize issues
3. Fix agent → Invoke sf-ai-agentforce skill to apply fixes
4. Retest → Loop until all tests pass or max retries (3) reached
5. Escalate → Skip unfixable tests and continue with others

Quick Start

# Run the test-fix loop
./hooks/scripts/test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting 3

# Exit codes:
#   0 = All tests passed
#   1 = Fixes needed (Claude Code should invoke sf-ai-agentforce)
#   2 = Max attempts reached, escalate to human
#   3 = Error (org unreachable, test not found, etc.)

Claude Code Integration

When Claude Code runs the test-fix loop:

USER: Run automated test-fix loop for Coral_Cloud_Agent

CLAUDE CODE:
1. bash hooks/scripts/test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting
2. If exit code 1 (FIX_NEEDED):
   - Parse failure details from output
   - Invoke: Skill(skill="sf-ai-agentforce", args="Fix topic X: add keyword Y")
   - Re-run: CURRENT_ATTEMPT=2 bash hooks/scripts/test-fix-loop.sh ...
3. Repeat until exit code 0 (success) or 2 (max retries)

Ralph Wiggum Integration (Hands-Off)

For fully automated loops without user intervention:

/ralph-wiggum:ralph-loop
> Run agentic test-fix loop for Test_Agentforce_v1 in AgentforceTesting until all tests pass

Claude Code will autonomously:

• Execute test-fix cycles
• Apply fixes via sf-ai-agentforce skill
• Track attempts and escalate when needed
• Report final status

Failure Categories & Auto-Fix Strategies

Environment Variables

Machine-Readable Output

The script outputs structured data for Claude Code parsing:

---BEGIN_MACHINE_READABLE---
FIX_NEEDED: true
TEST_API_NAME: Test_Agentforce_v1
TARGET_ORG: AgentforceTesting
CURRENT_ATTEMPT: 1
MAX_ATTEMPTS: 3
NEXT_COMMAND: CURRENT_ATTEMPT=2 ./test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting 3
---END_MACHINE_READABLE---

🐛 Known Issues & CLI Bugs

Last Updated: 2026-01-04 | Tested With: sf CLI v2.118.16

CRITICAL: sf agent test create MasterLabel Bug

Status: 🔴 BLOCKING - Prevents YAML-based test creation

Error:

Error (SfError): Required fields are missing: [MasterLabel]

Root Cause: The CLI generates XML from YAML but omits the required <name> element (MasterLabel).

Generated XML (broken):

<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <subjectName>My_Agent</subjectName>
    <subjectType>AGENT</subjectType>
    <!-- ❌ MISSING: <name>Test Name</name> -->
    <testCase>...</testCase>
</AiEvaluationDefinition>

Working XML (from existing tests):

<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <description>Test description</description>
    <name>Test Name</name>  <!-- ✅ REQUIRED -->
    <subjectName>My_Agent</subjectName>
    <subjectType>AGENT</subjectType>
    <testCase>...</testCase>
</AiEvaluationDefinition>

Workarounds:

1. ✅ Use sf agent generate test-spec --from-definition to convert existing XML to YAML (produces correct format)
2. ✅ Use interactive sf agent generate test-spec wizard (works correctly)
3. ✅ Create tests via Salesforce Testing Center UI
4. ✅ Deploy XML metadata directly (bypass YAML conversion)

MEDIUM: Interactive Mode Not Scriptable

Status: 🟡 Blocks CI/CD automation

Issue: sf agent generate test-spec only works interactively:

• No --quiet, --json, or non-interactive flags
• Piped input causes "User force closed the prompt" error
• Cannot automate in CI/CD pipelines

What Works:

# Interactive (requires terminal)
sf agent generate test-spec --output-file ./tests/my-test.yaml

# Convert existing XML to YAML (non-interactive)
sf agent generate test-spec --from-definition path/to/test.xml --output-file ./output.yaml

Workaround: Use Python scripts in hooks/scripts/ to generate YAML programmatically.

MEDIUM: YAML vs XML Format Discrepancy

Issue: Documentation shows one YAML format, but Salesforce stores as different XML structure.

Doc Shows (doesn't map correctly):

testCases:
  - utterance: "Hello"
    expectation:
      topic: Welcome
      actionSequence: []

Actual Working Format (from --from-definition):

testCases:
  - utterance: "Hello"
    expectedTopic: Welcome
    expectedActions: []
    expectedOutcome: "Greeting response shown"

Key Mappings:

LOW: Expectation Name Variations

Issue: Different test creation methods use different expectation names:

Impact: May cause confusion when comparing test results from different sources.

Quick Start Example

# 1. Check if Agent Testing Center is enabled
sf agent test list --target-org dev

# 2. Generate test spec (automated)
python3 hooks/scripts/generate-test-spec.py \
  --agent-file ./agents/MyAgent.agent \
  --output ./tests/myagent-tests.yaml

# 3. Create test in org
sf agent test create \
  --spec ./tests/myagent-tests.yaml \
  --api-name MyAgentTest \
  --target-org dev

# 4. Run tests
sf agent test run \
  --api-name MyAgentTest \
  --wait 10 \
  --result-format json \
  --target-org dev

# 5. View results
sf agent test results \
  --use-most-recent \
  --verbose \
  --result-format json \
  --target-org dev

For complete workflows and fix loops, see:

• Agentic Fix Loops - Automated testing and fix workflows
• Test Spec Reference - Complete YAML format guide

License

MIT License. See LICENSE file. Copyright (c) 2024-2025 Jag Valaiyapathy