Validation History: sf-ai-agentscript

Validation History: sf-ai-agentscript

This file tracks the validation history of the sf-ai-agentscript skill. Validation agents are deployed to a test org to verify that documented patterns still work with current Salesforce releases.

Latest Validation

Validation Agent Results

Tier 1: Original Agents (13) — Re-validated

Tier 2: New Agents (3) — v1.9.0 Patterns

Total Duration: ~183s (16 agents — Tier 1+2 only)

Tier 3: Flow/Apex Integration Agents (6) — v2.1.0 Patterns

Apex Test Classes Deployed: BareOutputAction.cls (bare List return), DateTimeAction.cls (DateTime with @InvocableVariable wrappers)

Tier 4: Action Metadata Properties Agents (2) — v2.2.0 Patterns

Root Cause of v1.3.0 False Negatives: Val_Action_Properties only tested properties on @utils.transition (utility actions without targets). These properties ARE valid on action definitions with target: (Level 1 actions). The skill incorrectly generalized "not valid on transitions" → "not valid anywhere."

TDD Findings (v2.2.0)

Finding 10: label: is valid on action definitions, topics, and I/O fields ✅

• Pattern: label: property on topic block, action definition, input fields, and output fields
• Result: All compile and publish successfully on apex://TestApexAction target
• Impact: Corrected "Features NOT Valid" table in SKILL.md → renamed to "Feature Validity by Context"
• Scope: Valid on target-backed actions; still NOT valid on @utils.transition (Val_Action_Properties confirms)

Finding 11: require_user_confirmation, include_in_progress_indicator, progress_indicator_message valid on target-backed actions ✅

• Pattern: All three properties on an action definition with target: "apex://TestApexAction"
• Result: Compile and publish successfully
• Impact: Updated Feature Validity table: valid on actions with targets, not on @utils.transition
• Note: require_user_confirmation compiles but runtime confirmation dialog does not trigger (Issue 6 still OPEN)

Finding 12: is_required and is_user_input valid as input metadata properties ✅

• Pattern: is_required: True and is_user_input: True on action input definitions
• Result: Both compile and publish successfully
• Impact: Added Input Properties table to actions-reference.md; clarified reserved field names in SKILL.md

Finding 13: is_displayable and is_used_by_planner valid as output metadata properties ✅

• Pattern: is_displayable: False and is_used_by_planner: True on action output definitions
• Result: Both compile and publish successfully
• Impact: is_displayable confirmed as standard name for filter_from_agent (alias relationship documented)

TDD Findings (v2.1.0)

Finding 4: Bare @InvocableMethod is INCOMPATIBLE with Agent Script ❌

• Pattern: Apex class returning List<String> directly (no @InvocableVariable wrapper classes)
• Rounds Tested: 3 iterations — input name name (invalid), input name input (invalid), no inputs block (Internal Error)
• Root Cause: The framework requires @InvocableVariable annotations to discover bindable parameter names. Without wrapper classes, no I/O name can be matched.
• Impact: Added "Bare @InvocableMethod Pattern (NOT Compatible)" warning to SKILL.md and actions-reference.md
• Rule: Always use @InvocableVariable wrapper classes when targeting Apex from Agent Script

Finding 5: datetime type works in action I/O ✅

• Pattern: startDateTime: datetime and endDateTime: datetime in action inputs/outputs targeting Apex DateTime params
• Result: Publishes successfully; maps to lightning__dateTimeStringType in Lightning type system
• Impact: Added datetime → lightning__dateTimeStringType to SKILL.md type mapping tables

Finding 6: I/O names must exactly match @InvocableVariable field names ❌

• Pattern: Output name wrong_name when Apex class has @InvocableVariable public String outputText
• Error: "invalid output 'wrong_name'" at server-side compilation
• Impact: Added I/O Name Matching Rules section to SKILL.md and actions-reference.md with exact-match table

Finding 7: Partial output declaration is valid ✅

• Pattern: Declaring only outputText: string when Apex class also has inputText: string as output
• Result: Publishes successfully — subset of outputs is valid
• Impact: Added "Partial Output Pattern" documentation to SKILL.md

Finding 8: outputs: block specifically required (not just "I/O") ❌

• Pattern: Action with inputs: block but no outputs: block
• Error: "Internal Error, try again later" at server-side compilation
• Impact: Updated Issue 15 in known-issues.md — outputs: block is specifically required. inputs: alone is NOT sufficient.

Finding 9: Level 1 without Level 2 is valid ✅

• Pattern: Action definition in topic actions: block with target:, inputs:, outputs: but NO @actions.X invocation in reasoning.actions:
• Result: Publishes successfully — action definitions exist for the planner without explicit invocation
• Impact: Added "Level 1 Without Level 2 Is Valid" note to SKILL.md Two-Level Action System section

TDD Findings (v1.9.0)

Finding 1: else: + nested if does NOT compile ❌

• Pattern: else: block containing a nested if statement (SKILL.md Approach 3)
• Error: Unexpected 'if' at the nested if line, cascading Unexpected '@utils', Missing colon, Invalid syntax
• Impact: SKILL.md Approach 3 must be removed or marked INVALID
• Workaround: Use compound conditions (if A and B:) or sequential ifs
• Note: Local LSP validator PASSES this pattern — only server-side compiler rejects it

Finding 2: <> not-equal operator does NOT compile ❌

• Pattern: if @variables.x <> "value": using <> as not-equal
• Error: Unexpected '>' 44:29 — parser does not recognize <> as a token
• Impact: Remove <> from SKILL.md operator table; only != is valid
• Workaround: Use != operator (already validated in Val_Comparison_Ops)

Finding 3: Multiple available when clauses ARE valid ✅

• Pattern: Two available when clauses on the same action
• Result: Compiles and publishes successfully
• Impact: SKILL.md constraint "One available when per action" is WRONG — remove it
• Impact: Common Issues entry "Duplicate 'available when' clause" is WRONG — remove it
• Impact: multi-step-workflow template pattern is CORRECT as-is

Patterns Validated

Each validation agent tests specific patterns documented in SKILL.md:

1. Val_Minimal_Syntax → Core Syntax

   • config block with agent_name, agent_label, default_agent_user
   • system block with messages and instructions
   • start_agent block with reasoning and actions
   • topic block with description and reasoning

2. Val_Arithmetic_Ops → Expression Operators

   • Addition: @variables.counter + 1
   • Subtraction: @variables.counter - 1
   • Note: *, /, % are NOT supported

3. Val_Comparison_Ops → Expression Operators

   • Equality: ==, not-equal: !=
   • Comparisons: <, <=, >, >=
   • Logical: and, or, not
   • Note: <> is NOT valid (see Finding 2)

4. Val_Variable_Scopes → Variable Namespaces

   • mutable string with default value
   • mutable number with default value
   • mutable boolean with default value
   • set @variables.x = value assignment

5. Val_Topic_Transitions → Topic Transitions

   • @utils.transition to @topic.X (permanent handoff)
   • Multi-step topic chains

6. Val_Latch_Pattern → Production Gotchas

   • Boolean flag initialization
   • Setting latch on entry
   • Checking latch in topic selector
   • Clearing latch on completion

7. Val_Loop_Guard → Production Gotchas

   • Iteration counter pattern
   • available when guard clause
   • Exit condition on max iterations

8. Val_Interpolation → Instruction Syntax

   • Basic interpolation: {!@variables.x}
   • Multiple variables in string
   • Conditional interpolation: {!value if condition else alt}

9. Val_Action_Properties → Action Properties on @utils.transition (NEGATIVE)

   • CONFIRMED NOT VALID on @utils.transition: require_user_confirmation, include_in_progress_indicator, output_instructions, progress_indicator_message
   • CONFIRMED VALID: description only (on @utils.transition)
   • CORRECTED (v2.2.0): These properties ARE valid on action definitions with target: — see Val_Action_Meta_Props (#23)

10. Val_Before_Reasoning → Lifecycle Hooks

    • Content directly under before_reasoning: block
    • NO instructions: wrapper required

11. Val_After_Reasoning → Lifecycle Hooks

    • Content directly under after_reasoning: block
    • NO instructions: wrapper required

12. Val_Label_Property → label: on @utils.transition (NEGATIVE, scope-limited)

    • CONFIRMED NOT VALID on @utils.transition actions (v1.3.0)
    • CORRECTED (v2.2.0): label: IS valid on target-backed action definitions, topics, and I/O fields — see Val_Action_Meta_Props (#23)

13. Val_Always_Expect_Input → Unimplemented Features (NEGATIVE)

    • CONFIRMED NOT VALID: always_expect_input: property

14. Val_Else_Nested_If → if/else Nesting (NEGATIVE)

    • CONFIRMED NOT VALID: else: block with nested if statement
    • CONFIRMED VALID: Compound conditions and sequential ifs as alternatives

15. Val_Step_Guard → Topic Re-Entry Protection

    • Step counter in start_agent guards topic selector
    • workflow_active boolean flag pattern
    • available when on step completion

16. Val_Multiple_Available_When → Action Guards

    • CONFIRMED VALID: Multiple available when clauses on same action
    • DISPROVES: "One available when per action" constraint in SKILL.md

17. Val_Apex_Bare_Output → Bare @InvocableMethod (NEGATIVE)

    • CONFIRMED NOT COMPATIBLE: Bare List<String> return without @InvocableVariable wrappers
    • 3 rounds tested: input name name (invalid), input (invalid), no inputs (Internal Error)
    • Rule: Always use @InvocableVariable wrapper classes for Agent Script actions

18. Val_Datetime_IO → DateTime Type in Action I/O

    • CONFIRMED VALID: datetime type in action inputs/outputs
    • Maps to lightning__dateTimeStringType in Lightning type system
    • Apex target: DateTimeAction.cls with DateTime wrapper params

19. Val_Name_Mismatch → I/O Name Matching (NEGATIVE)

    • CONFIRMED: I/O names must exactly match @InvocableVariable field names
    • Output name wrong_name vs Apex field outputText → "invalid output 'wrong_name'"

20. Val_Partial_Output → Partial Output Declaration

    • CONFIRMED VALID: Declaring subset of target outputs is acceptable
    • Declared outputText only from TestApexAction (which has both input and output wrappers)

21. Val_No_Outputs → Inputs-Only Action (NEGATIVE)

    • CONFIRMED: outputs: block is specifically required for publish
    • Action with inputs: but no outputs: → "Internal Error, try again later"
    • Validates Issue 15 nuance: outputs specifically required, not just "I/O"

22. Val_Level1_Only → Level 1 Without Level 2

    • CONFIRMED VALID: Action definition in topic actions: block without @actions.X in reasoning
    • Level 1 definitions (target + I/O) without Level 2 invocations publish successfully

23. Val_Action_Meta_Props → Action Metadata Properties (POSITIVE)

    • CONFIRMED VALID: label: on action definitions, topics, and I/O fields
    • CONFIRMED VALID: require_user_confirmation: True on target-backed actions (runtime Issue 6)
    • CONFIRMED VALID: include_in_progress_indicator: True on target-backed actions
    • CONFIRMED VALID: progress_indicator_message: on target-backed actions (both flow:// and apex://)
    • CORRECTS v1.3.0: Val_Action_Properties only tested @utils.transition (limited property support)

24. Val_IO_Meta_Props → I/O Metadata Properties (POSITIVE)

    • CONFIRMED VALID: is_required: True on action inputs
    • CONFIRMED VALID: is_user_input: True on action inputs (LLM extracts from conversation)
    • CONFIRMED VALID: is_displayable: False on action outputs (alias for filter_from_agent)
    • CONFIRMED VALID: is_used_by_planner: True on action outputs

Validation Command

# Navigate to validation directory
cd sf-ai-agentscript/validation

# Deploy metadata first
sf project deploy start \
  --source-dir validation-agents/force-app/main/default/aiAuthoringBundles \
  --target-org AgentforceTesting --json

# Publish each agent
for agent in Val_Minimal_Syntax Val_Arithmetic_Ops Val_Comparison_Ops Val_Variable_Scopes \
  Val_Topic_Transitions Val_Latch_Pattern Val_Loop_Guard Val_Interpolation \
  Val_Action_Properties Val_Before_Reasoning Val_After_Reasoning \
  Val_Label_Property Val_Always_Expect_Input Val_Else_Nested_If \
  Val_Step_Guard Val_Multiple_Available_When \
  Val_Apex_Bare_Output Val_Datetime_IO Val_Name_Mismatch \
  Val_Partial_Output Val_No_Outputs Val_Level1_Only \
  Val_Action_Meta_Props Val_IO_Meta_Props; do
  sf agent publish authoring-bundle --api-name "$agent" --target-org AgentforceTesting --json
done

Test Org Configuration

History

Next Validation Due

2026-03-19 (30 days from last validation)

Troubleshooting

If Validation Fails

1. Check the error message - Salesforce will indicate what syntax changed
2. Update SKILL.md - Document the new constraint or syntax requirement
3. Fix the validation agent - Update to use correct syntax
4. Re-run validation - Ensure all agents pass again
5. Update this file - Log the issue and resolution in History

Common Issues