Agent Testing And Evaluation

Agent Testing and Evaluation

Use this skill when the work is validating that an Agentforce agent routes correctly, produces quality responses, and continues to behave as expected after configuration changes. This skill covers the full testing lifecycle: interactive conversation testing in Agent Builder, structured utterance and topic tests defined in AiEvaluationDefinition metadata, programmatic test execution via the Testing API (Connect API), evaluation metrics interpretation, and regression testing patterns across the DevOps lifecycle. It does not cover how to create an agent, design subagents, or build actions — those are covered by the sibling skills listed in Related Skills.

Agentforce testing sits at the intersection of deterministic validation (did the agent fire the right action?) and probabilistic quality assessment (did the response satisfy the customer?). Both dimensions matter. Ignoring either produces false confidence.

Terminology. Agent topics were renamed subagents in April 2026. The change is vocabulary only — there are no changes to functionality, and the API surface did not rename. This skill therefore leads with subagent in prose but deliberately keeps topic everywhere the platform kept it: the expectTopicName and actualTopicName fields, the topic_sequence_match expectation, the "topic test" assertion category, the "topic activation accuracy" metric, and the search keywords readers still arrive with.

Before Starting

Gather this context before working on anything in this domain:

• Is the agent in Active state? Testing API can test an Active agent in any org. The Conversation Preview panel in Agentforce Builder works on Draft agents too — use it during development.
• Which environment is being tested? Sandbox is the correct place for pre-production automated test suites. Data Cloud is generally not available in Developer Edition orgs by default; confirm Einstein is enabled and any Data Cloud grounding sources are seeded if tests depend on knowledge retrieval.
• What subagents exist and what are their classification descriptions? Utterance test design requires knowing the intended scope boundaries between subagents — ambiguous boundaries are the #1 source of routing failures.
• Do test cases need to invoke real external actions (callouts, record operations)? By default, test runs via the Testing API execute the agent's reasoning engine and subagent/action routing but do not submit DML or callouts to external systems. Plan mock data accordingly.
• Has a baseline evaluation run been captured? Regression testing requires a saved baseline. Capture one before any subagent, instruction, or action change.

Core Concepts

Agentforce Testing Center and AiEvaluationDefinition

Agentforce provides two testing surfaces that share the same underlying evaluation engine:

• Testing Center UI (Setup > Agentforce > Testing Center) — browser-based tool for creating, organizing, and running test suites against a named agent. Best for iterative development and ad-hoc validation.
• Testing API — programmatic interface combining AiEvaluationDefinition Metadata API types (for test definition) and Connect API endpoints (for test execution and result retrieval). Best for CI/CD pipeline integration and regression automation.

AiEvaluationDefinition is the canonical metadata type for a test suite. It defines the agent under test and a set of test cases. Each test case specifies:

Test definitions deploy alongside the agent metadata, so test suites are version-controlled and environment-promotable.

Agentforce DX CLI — the sf agent test Command Family

Agentforce DX adds a Salesforce CLI surface on top of the same AiEvaluationDefinition / Connect API machinery. For anyone wiring agent tests into a script or CI pipeline, these commands are the idiomatic path — raw curl against the Connect API (Pattern 1) still works but is no longer necessary:

Two supporting notes:

• sf agent test run accepts --result-format json|tap|junit plus --output-dir to write result files to disk — JUnit output is what most CI systems parse natively for pass/fail gating.
• The generic sf api request rest command can call the same Connect API endpoints (including Get Test Results by runId) when you need an endpoint the sf agent test family doesn't wrap.

Deploying the AiEvaluationDefinition metadata itself remains a separate step via standard sf project deploy start — deployment installs the definition; only sf agent test run (or the Connect API POST) executes it.

Three Test Types

The platform evaluates three distinct properties per test case:

1. Topic test — did the agent classify the utterance to the expected subagent? This is a deterministic pass/fail. A failing topic test means the classification descriptions need refinement or the utterance is ambiguous.
2. Action test — did the agent invoke the expected action or action sequence? Validates that the reasoning engine's plan matches design intent. Multi-step action sequences can be tested by specifying an ordered list.
3. Instruction adherence test — how well did the generated response follow the subagent's instructions? Evaluated by a secondary LLM judge on a pass/fail basis with configurable criteria. Useful for tone, constraint, and persona compliance checks.

You can combine all three expectations on a single test case or use them independently.

Utterance Coverage

A subagent is not proven to work from a single utterance. Adequate coverage requires:

A coverage matrix (subagent × utterance type) makes gaps visible. Without this, teams discover routing failures in production from real customer sessions.

Evaluation Metrics

Post-deploy and ongoing quality measurement uses a set of standard operational metrics:

No single metric is sufficient. Containment without CSAT can mask an agent that contains by frustrating customers. CSAT without containment rate masks a well-liked but expensive agent.

Common Patterns

Pattern 1: Pre-Launch Subagent Coverage Validation

When to use: Before activating a new agent or a significantly revised subagent set.

How it works:

1. Build a coverage matrix: list every subagent and define at least 5 utterances per subagent — 2 happy-path, 2 edge-case, 1 boundary utterance near an adjacent subagent.

2. Create one AiEvaluationDefinition file per subagent group (or one combined file). Each test case sets expectTopicName to the intended subagent.

3. Deploy the AiEvaluationDefinition to the sandbox via sf project deploy start.

4. Execute via the Testing API. Note the shape of these paths: the resource sits directly under /services/data/vXX.0/einstein/ — there is no connect/ segment — and every operation goes through the /runs collection. Detailed per-test-case output lives on a further /results sub-resource:

   Operation	Method and path
   Start a run	POST /services/data/v63.0/einstein/ai-evaluations/runs
   Poll status	GET /services/data/v63.0/einstein/ai-evaluations/runs/{runId}
   Retrieve results	GET /services/data/v63.0/einstein/ai-evaluations/runs/{runId}/results

# Start the evaluation run
curl -X POST \
  https://ORG_DOMAIN.my.salesforce.com/services/data/v63.0/einstein/ai-evaluations/runs \
  -H "Authorization: Bearer SESSION_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "aiEvaluationDefinitionName": "OrderAgentTopicTests"
  }'
# Response:
# {
#   "id": "0Xx000000000001AAA",
#   "status": "IN_PROGRESS",
#   "startTime": "2026-04-10T14:30:00.000Z"
# }

5. Poll the returned job ID until status: Completed:

curl https://ORG_DOMAIN.my.salesforce.com/services/data/v63.0/einstein/ai-evaluations/runs/0Xx000000000001AAA \
  -H "Authorization: Bearer SESSION_ID"
# Response when complete:
# {
#   "id": "0Xx000000000001AAA",
#   "status": "COMPLETED",
#   "summary": {
#     "totalTestCases": 12,
#     "passed": 10,
#     "failed": 2
#   },
#   "testCaseResults": [
#     {
#       "testCaseIndex": 0,
#       "utterance": "Where is my order?",
#       "expectations": { "expectTopicName": "OrderStatus" },
#       "actuals": { "actualTopicName": "OrderStatus" },
#       "result": "PASS"
#     },
#     {
#       "testCaseIndex": 3,
#       "utterance": "I was charged twice and want my money back",
#       "expectations": { "expectTopicName": "ReturnRequest" },
#       "actuals": { "actualTopicName": "BillingInquiry" },
#       "result": "FAIL"
#     }
#   ]
# }

6. Retrieve the per-test-case detail from the /results sub-resource. The status poll returns the run summary; it is /results that carries the full test-case report, so a pipeline that never calls it cannot explain why a run failed:

curl https://ORG_DOMAIN.my.salesforce.com/services/data/v63.0/einstein/ai-evaluations/runs/0Xx000000000001AAA/results \
  -H "Authorization: Bearer ACCESS_TOKEN"

6. Review results: any FAIL on a topic test means the utterance routed elsewhere. Inspect the actualTopicName in the result payload and tune the classification description of either the intended or the competing subagent.
7. Re-run until all topic tests pass. Document the final pass results as the baseline.

Why not just use Conversation Preview manually: Manual preview is valuable for exploratory testing but does not produce repeatable, trackable results. It cannot catch regressions after a future change.

CLI alternative: steps 4–5 (execute + poll) collapse into a single sf agent test run --api-name OrderAgentTopicTests --wait 10 — see Pattern 4. The raw Connect API calls above remain useful when the calling system is not a Salesforce CLI environment.

Pattern 2: Regression Suite After Subagent or Action Changes

When to use: Any time a subagent's classification description, instructions, or action set is modified.

How it works:

1. Before making changes, capture the current test run results as a baseline (save the JSON result payload from the Connect API or export from Testing Center).
2. Apply the subagent or action change in the sandbox.
3. Re-run the existing AiEvaluationDefinition test suite against the modified agent.
4. Diff the new results against the baseline: look for test cases that were previously PASS and are now FAIL (newly broken) and test cases that were previously FAIL and are now PASS (intentional improvements or coincidental fixes).
5. Investigate and resolve all newly broken cases before promoting the change.
6. Update the baseline after deliberate improvements are confirmed.

Why not skip the baseline: Without a baseline, you cannot distinguish a regression from a pre-existing bug. Teams that skip baselining end up unable to tell whether a failing test is caused by the current change or was always broken.

Pattern 3: Multi-Turn Conversation Testing

When to use: When the agent handles conversations that require context from prior turns (e.g., "change my order" after the agent has already retrieved the order details, or disambiguation flows).

How it works:

1. Identify conversation flows that have meaningful context dependency — where the correct agent behavior in turn N depends on what happened in turn N-1 or earlier.
2. In AiEvaluationDefinition, include a conversationHistory array in the test case input. Each element in the array is a prior turn with role (user or agent) and content.
3. The agent evaluation engine replays the conversation including the provided history, then evaluates only the final user utterance against the expectations.
4. Write separate test cases for different conversational states to test the full flow matrix, not just the terminal turn.

Pattern 4: CLI-Native Test Pipeline with Agentforce DX

When to use: Wiring agent tests into a CI/CD pipeline, or authoring tests from a DX project instead of hand-writing AiEvaluationDefinition XML.

How it works:

1. Scaffold the test spec YAML locally — the authoring precursor to any metadata:

sf agent generate test-spec

2. Create the test in the dev org from the spec. This creates the AiEvaluationDefinition in the org and automatically syncs the resulting metadata back into the DX project source, so the test is version-controlled without a manual retrieve:

sf agent test create   # pass the generated test spec YAML file

3. In the pipeline, run synchronously and emit CI-native output:

sf agent test run --api-name Order_Agent_Tests --target-org ci-sandbox \
  --wait 10 --result-format junit --output-dir ./agent-test-results

--wait <minutes> blocks until completion so the pipeline step's exit status can gate promotion. --result-format junit (or json / tap) plus --output-dir writes result files most CI systems parse natively — no custom JSON-diffing script required for the pass/fail gate itself (you still persist the JSON payload for regression baselining, Pattern 2).

4. For long suites, run asynchronously instead: omit --wait, capture the printed sf agent test resume command, and fetch results in a later pipeline stage with sf agent test resume or sf agent test results --job-id <id>.
5. For scripts targeting multiple orgs, discover or validate test API names first with sf agent test list against the target org.

Why not raw curl: The Connect API calls in Pattern 1 work, but they require you to manage session tokens, polling loops, and result parsing yourself. The sf agent test family reuses the CLI's org auth, handles polling, and produces JUnit output for free. Keep curl (or sf api request rest) for endpoints the command family doesn't wrap.

Decision Guidance

Recommended Workflow

Step-by-step instructions for an AI agent or practitioner working on agent testing:

1. Map subagent coverage — list all subagents, identify utterance gaps, and produce a coverage matrix before writing any test cases. Prioritize boundary utterances between adjacent subagents.
2. Author test cases — either scaffold a test spec YAML with sf agent generate test-spec and create the org test (plus auto-synced metadata) with sf agent test create, or hand-author AiEvaluationDefinition metadata directly. Either way: structured test cases with utterances, expected subagent names, expected action sequences, and instruction adherence expectations. Include multi-turn conversation history for context-dependent flows.
3. Deploy and execute in sandbox — deploy the test definition via sf project deploy start, then execute with sf agent test run --api-name <name> --wait <minutes> (or the Testing API Connect endpoint POST /connect/einstein/ai-evaluations with manual polling when the CLI is not available).
4. Review and iterate — inspect FAIL results, identify whether the failure is in subagent routing (tune classification descriptions), action sequencing (revise subagent instructions or action order), or instruction adherence (tighten subagent instructions). Re-run until all cases pass.
5. Capture baseline — save the passing test run results as the regression baseline before any further changes.
6. Integrate into DevOps pipeline — add an sf agent test run --wait <minutes> --result-format junit --output-dir <dir> step to the promotion pipeline from sandbox to production and let the CI runner's JUnit parsing block promotion on test failures. Use sf agent test list in scripts that need to discover or validate test API names in the target org first.
7. Monitor post-deploy — track containment rate, escalation rate, and CSAT via Enhanced Event Logs reports. Treat anomalies as signals to add new test cases for the conversation patterns causing failures.

Review Checklist

Run through these before marking testing work complete:

• Coverage matrix created: at least 5 utterances per subagent (happy path, edge case, boundary, out-of-scope).
• AiEvaluationDefinition metadata authored and version-controlled alongside agent metadata.
• All topic tests passing (100% correct topic activation across the test suite).
• Action sequence tests passing for all primary action flows.
• Instruction adherence tests included for subagents with strict tone or constraint requirements.
• Multi-turn conversation tests cover all context-dependent flows.
• Baseline test results saved before any subagent or action change.
• Regression diff reviewed after each change — no newly broken cases remain.
• Test execution integrated into the sandbox-to-production promotion pipeline (sf agent test run --wait --result-format junit, or Connect API when the CLI is unavailable).
• Enhanced Event Logs enabled on the production agent for post-deploy monitoring.
• Containment rate, escalation rate, and CSAT dashboards configured or scheduled.

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

1. Test conversations do not invoke real DML or callouts — Testing API evaluates the agent's reasoning and routing but action execution is simulated. Tests can pass 100% while a misconfigured action (wrong API endpoint, missing field mapping) fails only in a live session. Always run at least one live conversation test in sandbox with real data before promoting to production.
2. AiEvaluationDefinition deploys but Testing API requires a separate execute call — deploying the metadata does not run the tests. Teams sometimes deploy the definition and assume tests passed. You must explicitly POST to the Connect API execute endpoint and wait for the job to complete before checking results.
3. Subagent classification is probabilistic — the same utterance can route differently on repeated runs — the reasoning engine has inherent non-determinism. An utterance that sits on a subagent boundary may alternate between two subagents across test runs. Test suites with too many boundary utterances in the happy-path tier will produce flaky results. Move genuinely ambiguous utterances to a dedicated "boundary" tier and evaluate the routing distribution rather than expecting 100% consistency on them.
4. Enhanced Event Logs only capture production conversations, not Testing API runs — test results are returned in the Testing API response payload, not in Enhanced Event Logs. Teams expecting to find test run failures in the Event Log will find nothing. Use Event Logs only for post-deploy monitoring of real user sessions.
5. Instruction adherence evaluation uses a secondary LLM judge and can be inconsistent at low test volumes — the instruction adherence score is produced by a separate model evaluation, not a rule-based check. On very short or edge-case responses it can produce inconsistent pass/fail results across repeated runs of the same test. Use it as a trend signal, not a binary gate, until you have sufficient test volume to trust the distribution.
6. sf agent test run is asynchronous by default — a CI step that omits --wait exits before any results exist — without --wait, the command only prints the sf agent test resume command and returns immediately. A pipeline that greps that step's output for pass/fail will gate on nothing. Add --wait <minutes> for synchronous gating, or split into an async run stage plus a later sf agent test results --job-id stage.
7. The VS Code Agent Tests panel requires the AiEvaluationDefinition in a local package directory — tests run from VS Code execute in the same development org as sf agent test run, but VS Code additionally requires the test's AiEvaluationDefinition component to physically exist in a package directory of the DX project, not just in the org. sf agent test create satisfies this automatically because it syncs the metadata back; tests created only in the Testing Center UI won't appear until retrieved into source.

Output Artifacts

Related Skills

• agentforce/agentforce-agent-creation — use for standing up a new agent, channel assignment, and activation. Testing assumes the agent already exists.
• agentforce/agent-topic-design — use when subagent classification failures in tests indicate subagent boundary or description problems.
• agentforce/agent-actions — use when action sequence test failures indicate action configuration or contract issues.
• devops/scratch-org-management — use when the agent testing pipeline is part of a scratch org–based CI workflow.