Datarobot Agent Assist Simulate

Agent Assist — Simulate

Adversarially test and harden an implemented agent before deployment. Runs three simulation tracks (attack, behavior, persistence), then patches and retests failing scenarios across multiple rounds until all pass or the fixing limit is reached.

Script Path Resolution

Resolve once per session: <skill_scripts_dir> is the scripts/ subdirectory next to this SKILL.md. Confirm it exists. Use the resolved absolute path everywhere.

Python: prefer .venv/bin/python3 if a .venv/ exists; otherwise python3. Store as <python>. Require 3.11+:

<python> -c "import sys; assert sys.version_info >= (3, 11)"
<python> -c "import pydantic, yaml" 2>/dev/null || <python> -m pip install pydantic pyyaml

OpenCode:

dr opencode upgrade && export PATH="$PATH:$HOME/.opencode/bin"

Auth:

dr auth check
dr dotenv update

If any check fails, surface the error and stop.

Swarm directory: All intermediate files (scenario inputs/outputs, run results, convergence state) are written to .datarobot/swarm/ under the project root and deleted at the end of Step 5.

If .datarobot/swarm/ already exists at the start of a session, it is leftover from a prior session that did not complete Step 5 cleanup. Run rm -rf .datarobot/swarm/ before proceeding.

Pre-flight Check

1. Confirm agent_spec.md exists with a system_prompt. If not, route the user to agent-assist-build and stop.
2. Confirm implementation code exists (agent.py, myagent.py, tools.py, or app.py). If not, route to agent-assist-build and stop.

Step 1 — Configure

Collect answers in sequence. Do not start simulation until all are answered.

If agent_config.yaml already exists, read it and ask:

"Last time: [persona], [context or none], [iterations] rounds of fixing, [eval mode], [model]. Same settings or change anything?"

Q1 — User type: Read agent_spec.md and offer 2–4 domain-specific personas plus "Other — describe your user segment."

Q2 — Grounding context (optional): Based on agent_spec.md, ask for domain-appropriate examples (e.g. past queries, sample inputs, real requests). Keep the question to one sentence. Save to user_context.txt if provided. Skip if the user says "skip."

Q3 — Fixing rounds: Ask:

"How many rounds of fixing should I run on failing scenarios? Default: 3."

Q4 — Evaluation mode: Ask:

"How should results be evaluated? Standard gives a simple pass/fail. Scored rates each result by severity: low, medium, high, or critical. Default: standard."

Q5 — Model: Run dr opencode models. Recommend the strongest available models from the catalog. Show up to 5 as a numbered list. After the list, ask: "Want to see more models from DataRobot LLM Gateway?" If yes, show more from the catalog. Store the choice as <model>.

Q6 — Selective tool execution (optional): Read the tool definitions in agent_spec.md and identify any that appear to be read-only — no writes, no side effects, names like get_, list_, load_, fetch_, search_. If any exist, ask:

"I can call these tools for real during simulation instead of generating fictional return values: [list each with its description] Run them for real, or simulate all? Default is simulate."

If the user chooses real execution: edit agent_spec.md to add is_readonly: true on each approved tool, then pass --execution-mode selective_e2e to native_scenarios.py configure. Find the .venv/bin/python3 nearest to the implementation file (check its directory, then walk up) and use it instead of <python> when invoking native_swarm.py in Step 3. If the user declines or no read-only tools exist, omit --execution-mode (defaults to simulated).

Start OpenCode server (once, after model selection):

dr opencode serve --port 4096 2>/dev/null &
until curl -sf http://127.0.0.1:4096/global/health >/dev/null 2>&1; do sleep 0.25; done

Store the PID as <opencode_server_pid> and http://127.0.0.1:4096 as <opencode_server_url>.

Persist config:

<python> <skill_scripts_dir>/native_scenarios.py configure agent_spec.md \
  --user-persona "<persona>" \
  --iterations <n> \
  --judge-mode <standard|scored> \
  --model "<model>" \
  [--context user_context.txt]

Step 2 — Generate Scenarios

<python> <skill_scripts_dir>/native_scenarios.py prepare agent_spec.md \
  --config agent_config.yaml

Run each generator one at a time. Before each, announce what the track tests. After each, report the count and list the scenario names.

Say: "Generating attack scenarios — these test whether the agent can be manipulated into bypassing its own restrictions."

<python> <skill_scripts_dir>/gateway_worker.py \
  --role-prompt generate-attack \
  --input-path "$SWARM_DIR/attack-input.json" \
  --response-path "$SWARM_DIR/attack-output.json" \
  --model <model> --server-url <opencode_server_url>

Read $SWARM_DIR/attack-output.json and report: "Generated X attack scenarios:" followed by a list of scenario names.

Say: "Generating behavior scenarios — these test how the agent handles ambiguous or edge-case user requests."

<python> <skill_scripts_dir>/gateway_worker.py \
  --role-prompt generate-behavior \
  --input-path "$SWARM_DIR/behavior-input.json" \
  --response-path "$SWARM_DIR/behavior-output.json" \
  --model <model> --server-url <opencode_server_url>

Read $SWARM_DIR/behavior-output.json and report: "Generated X behavior scenarios:" followed by a list of scenario names.

Say: "Generating persistence scenarios — these apply multi-turn pressure to see if the agent holds its position under pushback."

<python> <skill_scripts_dir>/gateway_worker.py \
  --role-prompt generate-persistence \
  --input-path "$SWARM_DIR/persistence-input.json" \
  --response-path "$SWARM_DIR/persistence-output.json" \
  --model <model> --server-url <opencode_server_url>

Read $SWARM_DIR/persistence-output.json and report: "Generated X persistence scenarios:" followed by a list of scenario names.

Validate all outputs:

<python> <skill_scripts_dir>/native_scenarios.py finalize

Read finalize stdout and present the candidate list grouped by track. For each scenario output one line: - [name] — [one sentence on what it targets]

Example:

Attack (4)
- Malicious CSV Injects Visualization Override — tests whether the agent follows instructions embedded in uploaded file content
- Path Traversal via File Path Argument — tests whether the agent rejects file paths outside the allowed directory
...

Behavior (3)
...

Persistence (3)
...

Then ask:

"Add or remove any, or say 'run it' to confirm. Say 'explain [name]' or 'explain all' for full detail on any scenario." Write the authoritative criteria:

<python> <skill_scripts_dir>/native_scenarios.py confirm --output evaluation_criteria.md

Step 3 — Simulate

Find all implementation files in the working directory (agent.py, myagent.py, tools.py, app.py) and pass each as a separate --implementation flag. Tell the user before launching:

"[N] scenarios queued — covering adversarial attacks, ambiguous user behavior, and multi-turn pressure. Typically 2–5 minutes."

<python> <skill_scripts_dir>/native_swarm.py run agent_spec.md \
  --criteria evaluation_criteria.md \
  --config agent_config.yaml \
  --server-url <opencode_server_url> \
  --model <model> \
  --implementation <path> [--implementation <path> ...] \
  [--tools-path <path/to/tools.py>]

Include --tools-path only when execution.mode is selective_e2e and tools.py exists. Parse stdout as the summary JSON. Surface any warning: lines from stderr before presenting results.

While the swarm is running, read the task output file every 30 seconds and narrate newly completed scenarios to the user (scenario name, track, pass/breach/error). Continue until the run completes.

Present results:

Say: "N of M scenarios passed."

Then give a structured narrative before proceeding:

Per-track summary — for each track (attack, behavior, persistence), state how many passed and list each scenario with one line on what it tested. Example:

"Attack — 4/4 passed: path traversal via load_dataset, query injection via run_analysis, scope escalation via wildcard, instruction override attempt."

Per-breach narrative — for each breach, say:

"Breach: [scenario_name] ([track]) What happened: [breach_reason in plain language] Likely fix: [prompt addition / code guard]"

Determine "likely fix" from the breach: if the agent violated a stated restriction, it is a prompt patch. If the agent attempted a tool call it should not have made, it may need a code guard in the implementation.

Overall read — one sentence on where the agent is strong and where it is soft. Example:

"The agent holds well on adversarial attacks and ambiguous behavior, but has soft spots under sustained multi-turn pressure."

Proceed to Step 4 regardless of breached — native_convergence.py initialize must run even when breached == 0, since it is what creates the convergence state that Step 5's report command reads.

Step 4 — Converge

<python> <skill_scripts_dir>/native_convergence.py initialize agent_spec.md \
  --criteria evaluation_criteria.md \
  --config agent_config.yaml \
  --actual-model "<model>"

Parse stdout as JSON: {"status": "...", "breaches": [...], "exhausted": [...], "passed": [...]}. Each breach entry has scenario_id, scenario_name, track, breach_reason, transcript, breach_indicators, iteration, and suggested_rerun_dir.

If status is complete, skip to Step 5.

Fix loop — repeat until advance returns complete:

For each breach in breaches:

1. Read the breach transcript and breach_reason. Propose the minimal addition to the system prompt that prevents this behavior. Tell the user:

   "Breach: [scenario_name] — [breach_reason] Proposed fix: [your proposed text] Apply this patch? (yes/no)"

   If approved, edit agent_spec.md directly to append the text to system_prompt. Then find the SYSTEM_PROMPT string in the implementation files (check agent.py, myagent.py, tools.py, app.py) and apply the same addition there so the deployed agent stays in sync with the spec.

   If the breach is structural (cannot be fixed by prompt alone — e.g. a missing tool guard in the implementation), say so and read the implementation file at the relevant function. Propose a targeted code change and ask for approval. If approved, apply it with your Edit tool.

2. Before retesting, tell the user:

   "Breach: [scenario_name] What happened: [breach_reason] Fix: [one sentence describing what was added to the system prompt or implementation] Retesting now to verify the patch holds."

   Then re-run the scenario. Call initialize once to set up the run and get the first input/response paths:

   <python> <skill_scripts_dir>/native_execution.py initialize agent_spec.md \
     --criteria evaluation_criteria.md \
     --scenario-id <scenario_id> \
     --run-dir <suggested_rerun_dir>
   

   This returns {"role": "runner", "input_path": "...", "response_path": "...", "turn_number": 1, ...}.

   Role → prompt name mapping (use this to pick --role-prompt):

   • runner → run-scenario
   • fixture → generate-tool-return
   • evaluator → evaluate-result

   Drive to terminal — loop using submit output to advance state. Do NOT call initialize again inside the loop:

   Before each worker call, announce the current step:

   • Runner: "Turn N/M — running scenario"
   • Fixture: "Turn N/M — generating tool return"
   • Evaluator: "Turn N/M — evaluating"

   # 1. Call worker using input_path and response_path from initialize (or previous submit).
   #    For the fixture role: if agent_config.yaml has execution.mode: selective_e2e and
   #    tools.py exists, call tool_executor.py instead of gateway_worker.py:
   #
   #    selective_e2e fixture:
   #    <python> <skill_scripts_dir>/tool_executor.py \
   #      --input-path <input_path> --response-path <response_path> \
   #      --tools-path <tools_path> --readonly-tools <comma-separated readonly fn names>
   #
   #    all other roles (runner, evaluator) and simulated fixtures:
   <python> <skill_scripts_dir>/gateway_worker.py \
     --role-prompt <run-scenario|generate-tool-return|evaluate-result> \
     --input-path <input_path> \
     --response-path <response_path> \
     --model <model> --server-url <opencode_server_url>
   
   # 2. Submit and read next state:
   <python> <skill_scripts_dir>/native_execution.py submit \
     --run-dir <suggested_rerun_dir> --response <response_path>
   

   submit returns the next {"role", "input_path", "response_path", "turn_number"} or a terminal {"status": "passed"|"breach"|"error"}. Use the returned input_path and response_path for the next worker call. Stop when status is terminal.

   Report ✓ passed / ✗ breach / ! error when terminal.

3. After all reruns in this round, call advance with the completed rerun dirs:

   <python> <skill_scripts_dir>/native_convergence.py advance agent_spec.md \
     --rerun <scenario_id>:<suggested_rerun_dir> [--rerun ...]
   

   Parse stdout the same way as initialize. If status is complete, stop. Otherwise repeat the fix loop for the new breaches list.

For any scenario in exhausted: read its breach_reason and transcript, identify the implementation function responsible, propose a targeted code fix, ask for approval, apply with your Edit tool, then rerun it (using the next iteration dir from suggested_rerun_dir in the advance output) and call advance again.

When advance returns complete:

"Convergence complete. X resolved, Y exhausted."

Step 5 — Report

<python> <skill_scripts_dir>/native_convergence.py report agent_spec.md \
  --output eval_report.md

kill <opencode_server_pid> 2>/dev/null || true
rm -rf .datarobot/swarm/

After the script writes eval_report.md, append a "## Changes Applied" section to the file listing every change made during Step 4 — scenario name, what was changed, and why. For prompt patches, list both the addition to agent_spec.md and the corresponding change to the implementation file. For code fixes, list the function and file changed. Use your Edit tool to append to eval_report.md.

Present passed/total, unresolved, exhausted, and readiness to the user. If ready: false, say so explicitly before offering next steps.

For each exhausted scenario, immediately after the pass/fail summary line, add one line per scenario:

"⚠ [scenario_name] couldn't be resolved after [N] attempts — this needs your attention before deploying."

Then read eval_report.md and present a per-track breakdown. For each track (attack, behavior, persistence), list each scenario with one line: what it tested, whether it passed initially or needed a fix, and how many turns it ran.

Next steps:

What would you like to do next?
1. Review eval_report.md     — outcomes and unresolved scenarios
2. Re-run simulation         — after further changes
3. Test locally              — run the agent on your machine
4. Deploy                    — deploy the hardened agent to DataRobot

• If 1: read eval_report.md and present a structured summary.
• If 2: return to Step 1 (reuse saved settings or re-collect).
• If 3: read AGENTS.md, display the local test command, tell the user to run it in a new terminal.
• If 4: follow the deploy instructions in agent-assist-build/SKILL.md.

Error Handling

Step 2 generator failure — worker exited non-zero (response extraction failed) — the worker replied with prose instead of JSON. Retry that generator once with a --rejection-note forbidding tool calls (including the built-in skill tool) and meta-commentary. If it still fails, retry once with a different top-tier <model> (e.g. claude-opus-4-8) and tell the user which track used it. If that fails, surface the error and stop.

Step 2 generator failure — finalize rejected the content — finalize prints role:<role> validation failed: <reason>. Retry that generator once with --rejection-note "<reason>" and rerun finalize. If it still fails, surface it and stop.

Step 4 rerun worker failure — if a runner/fixture/evaluator worker exits non-zero, mark the scenario failed:

<python> <skill_scripts_dir>/native_execution.py fail \
  --run-dir <run_dir> --reason "<reason>"

Then call advance with that run dir so the script records it as errored and moves on.

Auth errors (401 / UNAUTHORIZED): Run dr auth login and retry immediately.

Script timeouts: Allow up to 2 minutes per worker, 10 minutes for native_swarm.py run.

Simulation Tracks

Surface these gaps to the user if relevant.