Netra Evaluation Setup

Netra Evaluation Setup — Specification-Driven Dataset Planning

This skill takes a QA/dev specification document and produces a complete, editable evaluation plan — dataset items, evaluators, variable mappings, and pass criteria — then creates everything in Netra on user approval.

Workflow

Execute these phases in order. Complete each phase before moving to the next.

Phase 0: Verify Provider Configuration

Before starting any planning or creation work, verify that the organization has a default provider configured.

Call netra_get_default_llm_configuration via MCP.

• If a valid configuration is returned (containing provider, model, and providerConfigurationId), proceed to Phase 1.
• If null or an error is returned, stop immediately and inform the user:

⚠️ No default provider configured.

Evaluation and dataset creation require a default LLM provider in your organization.
Please configure a default provider in the Netra dashboard before proceeding:

  Settings → Providers → Set Default

Once configured, run this skill again.

Do not proceed to any subsequent phase until a valid default provider is confirmed.

Phase 0.5: Detect Project Language

Before generating any SDK code (task implementations, hooks scaffolds, execution examples), determine whether the project is Python or TypeScript/JavaScript. Check the project root in this order:

If both are present (monorepo), ask the user which sub-project they are working on. If neither is found, ask the user.

From this point forward, generate ONLY the SDK examples for the detected language. Never mix Python and TypeScript conventions.

Language-specific conventions for this skill:

Phase 1: Ingest the Specification

1. Read the specification document provided by the user (file path, pasted text, or URL).
2. Extract every testable requirement. For each, identify:
   • Scenario name — a concise, human-readable name describing the behavior being validated
   • Category — happy path, negative, edge case, safety, performance
   • Input — the user message or trigger that tests this scenario
   • Expected behavior — what the correct agent response or action should be

Scenario Naming Guidelines

Generate meaningful scenario names based on the behavior being tested.

Good examples:

• User Login Success
• Password Reset With Valid Email
• Product Search Returns Relevant Results
• Reject Prompt Injection Attempt
• Handle Empty User Query
• Customer Requests Refund
• Retrieve Account Balance
• Multi-Step Travel Planning Conversation

Avoid generic names such as:

• Scenario 1
• Test Case 1
• Item 3
• Validation Test
• Happy Path Test

3. Propose additional test cases the spec may have missed:
   • Boundary conditions not explicitly listed
   • Common adversarial inputs (prompt injection, jailbreak attempts)
   • Performance edge cases (very long input, empty input, special characters)
   • Multi-step reasoning scenarios if the agent supports them
4. Present the full list and ask the user to confirm before proceeding.

Phase 2: Discover Agent Span Structure

Build a span map — the list of named spans the agent produces at runtime.

Primary method — code analysis:

Search the agent codebase for instrumentation patterns. Use Grep and Read tools.

Patterns to search for:

Python Netra decorators:
  @workflow, @agent, @task, @span          → span name from name= parameter
  Netra.start_span("name", ...)            → span name from first argument

Python tool definitions:
  @tool def func_name(...)                 → span name = function name
  class MyTool(BaseTool): name = "..."     → span name from class attribute

TypeScript Netra decorators:
  @Workflow(), @Agent(), @Task()           → span name from decorator argument
  netra.startSpan("name", ...)             → span name from first argument

Framework-specific:
  CrewAI Agent/Task definitions            → span name from name field
  LangGraph node definitions               → span name from node name
  Claude Agent SDK tool definitions        → span name from tool name

For each span found, record:

• name — the span name as it will appear in traces
• type — workflow, agent, task, tool, or generation
• output description — what the function returns (infer from code)
• position in flow — call order relative to other spans

Fallback method — sample trace:

If code analysis yields insufficient results (no decorators found, dynamic span names):

1. Ask the user: "I couldn't find enough span definitions in the code. Has this agent been run at least once with Netra instrumented?"
2. If yes, use netra_query_traces to find a recent trace, then netra_get_trace_by_id to fetch its spans.
3. Build the span map from the actual trace data — span names, parent-child relationships, input/output.

Output of this phase:

Agent Flow:
  {root-span-name} ({type})
  ├── {child-span-1} ({type}) → {output description}
  ├── {child-span-2} ({type}) → {output description}
  │   └── {nested-span} ({type}) → {output description}
  └── {child-span-3} ({type}) → {output description}

Present to user and confirm span names are accurate.

Phase 3: Select Evaluators and Configure Mappings

For each test case from Phase 1, determine which evaluators to attach and how to wire their variables.

Step 3a: Get available evaluators

Try the MCP tool first. If unavailable, use the embedded reference.

Try:    netra_get_evaluator_library (all categories)
        netra_list_evaluators (custom evaluators in the project)

Fallback: Read references/evaluator-library.md for the full catalog

Step 3b: Match evaluators to test cases

Apply these mapping rules:

These are starting recommendations. Always consider the specific scenario semantics — an evaluator makes sense only if its purpose aligns with what the test case validates.

Step 3c: Identify configurable variables

Some evaluators require user-provided configuration that cannot be auto-resolved from conversation context or metadata. These are defined as configurableVariables in the evaluator library. If a configurable variable is left empty, the evaluator will fail or produce unreliable scores (e.g., "Cannot be evaluated" with a degraded score).

Evaluators with configurable variables:

Variable resolution priority for session evaluators:

1. evaluatorConfigs[].configuredValues (highest priority — user-provided)
2. Session metadata auto-resolution (e.g., metadata.assistant_constraints, metadata.reference_facts)
3. Empty string (fallback — causes evaluation failures)

assistant_instructions has NO auto-resolution path — it MUST be provided via evaluatorConfigs.

When an evaluator with configurable variables is recommended:

• Ask the user to provide the values
• Include them in the plan under a "Configurable Variables" section for each evaluator
• If the user cannot provide them yet, mark them as ⚠️ NEEDS CONFIGURATION in the plan

Step 3d: Configure variable mappings

For each evaluator attached to a test case, map every required variable to a data source.

Decision logic for the response / actual_output variable:

IF the evaluator checks the final user-facing answer
  → source: taskOutput
  → expression: "taskOutput"

IF the evaluator checks a specific intermediate step
  → source: span output
  → find the span from Phase 2 whose output is relevant
  → expression: "spans[?name=='{spanName}'] | [0].output"

IF the evaluator checks tool usage
  → source: trace tools
  → expression: "trace.tools"

Decision logic for the reference / expected variable:

IF ground truth is available in the dataset item
  → source: expectedOutput
  → expression: "expectedOutput"

IF reference comes from a specific span
  → expression: "spans[?name=='{spanName}'] | [0].output"

Decision logic for context (RAG evaluators):

→ source: retriever span output
→ expression: "spans[?name=='{retriever-span}'] | [0].output"

Decision logic for performance evaluators:

actual_latency → expression: "trace.latency"
actual_cost    → expression: "trace.cost"
actual_tokens  → expression: "trace.tokens"
expected_*     → source: literal value from the spec

When you cannot confidently determine a mapping, mark it as ⚠️ NEEDS CONFIGURATION in the plan and explain why.

Phase 3e: Set pass criteria and model

• Use the evaluator's default pass criteria unless the spec states otherwise.
• If the user specifies a model preference, apply it to all LLM evaluators shown in the plan.

The plan should display the model configuration that will be used if evaluator creation is required.

Provider/model resolution occurs during evaluator creation, not during evaluator recommendation.

Item-Level Provider Config

Multi-turn dataset items require a providerConfig because simulations execute through a model-backed task implementation.

{
  "provider_id": "{providerConfigurationId}",
  "model": "{model}"
}

Use netra_get_default_llm_configuration to retrieve the organization's default provider configuration.

Rules:

• For multi-turn datasets, every dataset item must include a providerConfig.
• If the specification does not define a model, use the organization's default provider configuration.
• If a scenario explicitly requires a different model, override the default by setting the item's providerConfig accordingly.
• For single-turn datasets, providerConfig is optional and should only be supplied when an item requires a model different from the evaluator default.

Phase 3f: Determine Execution Strategy

Before generating the final plan, determine whether the specification describes a single-turn evaluation or a multi-turn simulation.

Single-Turn Evaluations

If the specification describes a single request-response interaction:

• Set Turn Type = single
• Use Netra.evaluation.run_test_suite() for execution

Never invoke the agent directly for evaluation execution.

Rationale:

• Trace IDs are automatically captured
• Evaluators can access trace and span outputs
• Span-based variable mappings function correctly
• Evaluation results remain linked to execution traces

Execution example:

Netra.evaluation.run_test_suite(
    dataset_id="dataset-123",
    ...
)

Multi-Turn Simulations

If the specification describes a conversation, workflow, support interaction, troubleshooting flow, or any multi-step exchange:

• Set Turn Type = multi
• Set Eval Type = session — MUST be explicitly passed to netra_create_evaluator
• Use the language-appropriate simulation API:
  • Python: Netra.simulation.run_simulation()
  • TypeScript: await Netra.simulation.runSimulation({...})

Never invoke the agent directly for simulation execution.

The user must provide a task implementation compatible with Netra Simulation.

Python example:

class MyAgentTask(BaseTask):
    def __init__(self, agent):
        self.agent = agent

    def run(
        self,
        message: str,
        session_id: str | None = None,
        files: list[ProcessedFile] | None = None,
        setup_context: dict | None = None,  # populated by before_all / before hooks
    ) -> TaskResult:
        ctx = setup_context or {}
        response = self.agent.chat(
            message,
            session_id=session_id,
            files=files,
            # Pass any setup data from hooks (e.g. auth token, customer ID)
            auth_token=ctx.get("auth_token"),
        )

        return TaskResult(
            message=response.text,
            session_id=session_id or "default",
        )

result = Netra.simulation.run_simulation(
    name="Customer Support Simulation",
    dataset_id="dataset-123",
    task=MyAgentTask(my_agent),
    context={"environment": "staging"},
    max_concurrency=5,
    hooks=hooks,  # optional SimulationHooks from Phase 3g
)

TypeScript example:

import { BaseTask } from "netra-sdk";
import type { ProcessedFile, TaskResult } from "netra-sdk";

class MyAgentTask extends BaseTask {
  constructor(private agent: any) {
    super();
  }

  async run(
    message: string,
    sessionId?: string | null,
    files?: ProcessedFile[] | null,
    setupContext?: Record<string, any> | null, // populated by beforeAll / before hooks
  ): Promise<TaskResult> {
    const ctx = setupContext || {};
    const response = await this.agent.chat(message, {
      sessionId,
      files,
      authToken: ctx.authToken,
    });
    return {
      message: response.text,
      sessionId: sessionId || "default",
    };
  }
}

const result = await Netra.simulation.runSimulation({
  name: "Customer Support Simulation",
  datasetId: "dataset-123",
  task: new MyAgentTask(myAgent),
  context: { environment: "staging" },
  maxConcurrency: 5,
  hooks, // optional SimulationHooks from Phase 3g
});

This ensures:

• Conversation state is maintained
• Session IDs are tracked
• Simulation traces are generated
• Evaluators can access conversation history and span outputs
• Correlated scenarios can share state safely without sequential ordering

Turn Type Detection Rules

Determine execution mode from the specification:

If uncertain, ask the user before generating the plan.

Phase 3g: Identify Scenario Correlations and Propose Hooks

Apply this phase only for multi-turn datasets.

Dataset items often contain scenarios that are correlated — they share state or make assumptions about prior scenarios having run. Because Netra executes scenarios in parallel (configurable concurrency), sequential ordering cannot be guaranteed. Hooks solve this by letting users run setup/teardown code before and after scenarios without merging items or relying on ordering.

Step 1: Detect Scenario Correlations

Read every scenario in the plan and flag correlations in these categories:

If no correlations are found, hooks are not required. State this explicitly in the plan.

Step 2: Design the Hook Strategy

For each detected correlation pattern, decide which hook level to use:

Failure semantics to communicate to the user:

• If before_all fails → the entire run is marked failed; no scenarios execute
• If before_each or before[dataset_item_id] fails for a scenario → that scenario is marked prescript_failed; other scenarios continue running
• If after / after_each fails on an otherwise-successful scenario → that scenario is marked postscript_failed (both always attempt to run; errors are combined). If the scenario already failed/prescript_failed, the existing status is preserved
• If after_all fails → successfully completed scenarios are marked postscript_failed; already-failed/prescript_failed items keep their status
• On before-hook failure, after / after_each still run and receive the furthest successfully built setup_context (e.g. before_all + before_each if item before failed)

Terminal-state behaviour of prescript_failed:

• prescript_failed is a terminal status — an item that reached it is complete. The SDK's polling loop correctly counts prescript_failed items as terminal and will not wait indefinitely for them.
• A prescript_failed item also has its evalStatus immediately set to NOT_AVAILABLE — evaluators are never queued for an item whose agent never ran.
• If all items end in either failed or prescript_failed (with no successfully completed items), the run's overall evaluation status resolves to NOT_AVAILABLE.
• A prescript_failed item's status is stable — it will never be overwritten by a subsequent bulk-failure sweep (e.g. on run timeout).

Terminal-state behaviour of postscript_failed:

• postscript_failed is a terminal status — the conversation completed, but teardown failed.
• Eval is not suppressed — evaluators from the completed conversation remain valid.
• postscript_failed is excluded from the “all failed → NOT_AVAILABLE” roll-up (unlike prescript_failed).
• Status is stable — it will never be overwritten by a subsequent bulk-failure sweep.

How to determine dataset item IDs for hook mapping:

When generating the plan, you will create dataset items with specific IDs. These IDs are returned by the netra_create_dataset_item MCP call. After all items are created, instruct the user to:

1. Map the returned dataset item ID values to the corresponding scenarios
2. Use these IDs as keys in SimulationHooks.before / SimulationHooks.after
   • Python keys: dataset_item_id strings
   • TypeScript keys: same ID strings (SDK type field name is datasetItemId)

Python mapping example:

# Step 1: Create dataset items and capture their IDs
refund_item = netra_create_dataset_item(...)  # returns {"id": "item-abc-123", ...}
balance_item = netra_create_dataset_item(...)  # returns {"id": "item-xyz-789", ...}

# Step 2: Map hooks using the actual dataset_item_id values
hooks = SimulationHooks(
    before_all=before_all,
    before_each=before_each,  # optional: runs for every scenario
    before={
        "item-abc-123": setup_refund_scenario,  # Use actual ID from step 1
        "item-xyz-789": setup_balance_scenario,
    },
    after={
        "item-abc-123": teardown_refund_scenario,
        "item-xyz-789": teardown_balance_scenario,
    },
    after_each=after_each,  # optional: runs for every scenario
    after_all=after_all,
)

TypeScript mapping example:

// Step 1: Create dataset items and capture their IDs
const refundItem = /* netra_create_dataset_item(...) → { id: "item-abc-123", ... } */;
const balanceItem = /* netra_create_dataset_item(...) → { id: "item-xyz-789", ... } */;

// Step 2: Map hooks using the actual datasetItemId values
const hooks: SimulationHooks = {
  beforeAll,
  beforeEach, // optional: runs for every scenario
  before: {
    "item-abc-123": setupRefundScenario,
    "item-xyz-789": setupBalanceScenario,
  },
  after: {
    "item-abc-123": teardownRefundScenario,
    "item-xyz-789": teardownBalanceScenario,
  },
  afterEach, // optional: runs for every scenario
  afterAll,
};

Step 3: Generate Hook Scaffold Code

For each hook identified, generate a scaffold in the detected project language (Phase 0.5) with:

• The function signature matching the required hook level
• A short description of what the hook does, set via .description on every hook function (≤ 200 chars). Never omit this — without it the Netra UI gets description: null.
  • Python: fn.description = "..."
  • TypeScript: (fn as any).description = "..."
• Placeholder for the actual implementation (marked with # TODO / // TODO)
• The SimulationHooks wiring
• The updated simulation run call passing hooks

Do not generate Python scaffolds for a TypeScript project, or vice versa. Do not generate hooks without .description attached.

Python hook function signatures:

# before_all: no args → returns dict (shared context) or None
def before_all() -> dict | None:
    ...
before_all.description = "One-line description for the Netra UI."

# before_each: receives shared_context → returns dict merged into setup_context (runs for every item)
def before_each(shared_context: dict | None) -> dict | None:
    ...
before_each.description = "One-line description for the Netra UI."

# before: dict keyed by dataset_item_id; each receives merged context (before_all + before_each)
def setup_scenario_a(shared_context: dict | None) -> dict | None:
    ...
setup_scenario_a.description = "One-line description for the Netra UI."

def setup_scenario_b(shared_context: dict | None) -> dict | None:
    ...
setup_scenario_b.description = "One-line description for the Netra UI."

# after: dict keyed by dataset_item_id; each receives result + setup_context
def teardown_scenario_a(result: dict, setup_context: dict | None) -> None:
    ...
teardown_scenario_a.description = "One-line description for the Netra UI."

# after_each: receives result + setup_context (runs for every item, after item-specific after)
def after_each(result: dict, setup_context: dict | None) -> None:
    ...
after_each.description = "One-line description for the Netra UI."

# after_all: receives aggregated results dict and shared_context (before_all only) → returns None
def after_all(results: dict, shared_context: dict | None) -> None:
    ...
after_all.description = "One-line description for the Netra UI."

All Python hooks can be async (async def) if the user's setup code is async.

Required: every Python hook above must set .description. Omitting it sends description: null in lifecycleHooks.

TypeScript hook function signatures:

// beforeAll: no args → returns shared context object or null/void
async function beforeAll(): Promise<Record<string, any> | null | void> {
  ...
}
(beforeAll as any).description = "One-line description for the Netra UI.";

// beforeEach: receives sharedContext → returns dict merged into setupContext (every item)
async function beforeEach(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null | void> {
  ...
}
(beforeEach as any).description = "One-line description for the Netra UI.";

// before: Record keyed by datasetItemId; each receives merged context (beforeAll + beforeEach)
async function setupScenarioA(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null | void> {
  ...
}
(setupScenarioA as any).description = "One-line description for the Netra UI.";

// after: Record keyed by datasetItemId; each receives result + setupContext
async function teardownScenarioA(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  ...
}
(teardownScenarioA as any).description = "One-line description for the Netra UI.";

// afterEach: receives result + setupContext (every item, after item-specific after)
async function afterEach(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  ...
}
(afterEach as any).description = "One-line description for the Netra UI.";

// afterAll: receives aggregated results and sharedContext (beforeAll only)
async function afterAll(
  results: Record<string, any>,
  sharedContext: Record<string, any> | null,
): Promise<void> {
  ...
}
(afterAll as any).description = "One-line description for the Netra UI.";

Required: every TypeScript hook above must set .description. Omitting it sends description: null in lifecycleHooks. Context passing pattern:

Python:

before_all()                                      → shared_context (dict | None)
before_each(shared_context)                       → merged into setup_context
hooks.before[dataset_item_id](merged_context)     → merged into setup_context
BaseTask.run(..., setup_context=...)              ← receives merged context
hooks.after[dataset_item_id](result, setup_context)  ← same merged context for cleanup
after_each(result, setup_context)                 ← same merged context
after_all(results, shared_context)                ← run-level shared_context only

TypeScript:

beforeAll()                                       → sharedContext (Record | null)
beforeEach(sharedContext)                         → merged into setupContext
hooks.before[datasetItemId](mergedContext)        → merged into setupContext
BaseTask.run(..., setupContext)                   ← receives merged context
hooks.after[datasetItemId](result, setupContext)  ← same merged context for cleanup
afterEach(result, setupContext)                   ← same merged context
afterAll(results, sharedContext)                  ← run-level sharedContext only

Important: The before and after hooks are dictionaries keyed by dataset item ID, not single functions. Each scenario that requires specific setup/teardown gets its own function, and these functions are registered in SimulationHooks using the stable dataset item ID as the key. Prefer before_each / after_each when the same setup/teardown applies to every scenario.

The setup context is the merge of before_all / beforeAll + before_each / beforeEach + any object returned by the item before hook. It is passed to BaseTask.run(), the item after hook, and after_each / afterEach. If a before hook fails mid-way, teardown still receives the furthest successfully built setup context. after_all / afterAll still receives only the run-level shared context.

Full Python scaffold example (employee + per-scenario auth pattern):

Assume dataset has two items:

• dataset_item_id = "item-refund-request" → needs shared auth + refund-specific account
• dataset_item_id = "item-balance-inquiry" → needs shared auth only (before_each / after_each cover it)

from netra.simulation import BaseTask, SimulationHooks, TaskResult

# ---- Hooks ----

def before_all():
    # TODO: replace with your actual setup code
    employee = your_api.create_employee(name="Test User", role="admin")
    return {"employee_id": employee.id}
before_all.description = (
    "Create the test employee and assign the admin role before any scenario runs."
)


def before_each(shared_context: dict | None):
    employee_id = (shared_context or {}).get("employee_id")
    # TODO: replace with your actual login logic
    token = your_api.login(employee_id=employee_id)
    return {"auth_token": token}
before_each.description = "Obtain a fresh auth token before every scenario."


def setup_refund_scenario(shared_context: dict | None):
    # shared_context already includes employee_id + auth_token from before_all + before_each
    refund_account = your_api.create_refund_account()
    return {"refund_account_id": refund_account.id}
setup_refund_scenario.description = (
    "Create a refund account for the refund scenario only."
)


def teardown_refund_scenario(result: dict, setup_context: dict | None):
    refund_account_id = (setup_context or {}).get("refund_account_id")
    try:
        your_api.delete_refund_account(refund_account_id)
    except Exception:
        pass  # catch locally so teardown errors do not mark the item postscript_failed
teardown_refund_scenario.description = (
    "Delete the refund account after the refund scenario."
)


def after_each(result: dict, setup_context: dict | None):
    auth_token = (setup_context or {}).get("auth_token")  # from furthest built setup_context
    try:
        your_api.logout(token=auth_token)
    except Exception:
        pass
after_each.description = "Log out after every scenario regardless of outcome."


def after_all(results: dict, shared_context: dict | None):
    employee_id = (shared_context or {}).get("employee_id")
    # TODO: replace with your actual teardown code
    your_api.delete_employee(employee_id=employee_id)
after_all.description = "Delete the test employee once all scenarios have finished."


# Map hooks to specific dataset item IDs
hooks = SimulationHooks(
    before_all=before_all,
    before_each=before_each,
    before={
        "item-refund-request": setup_refund_scenario,
    },
    after={
        "item-refund-request": teardown_refund_scenario,
    },
    after_each=after_each,
    after_all=after_all,
)


# ---- Task ----

class MyAgentTask(BaseTask):
    def run(
        self,
        message: str,
        session_id: str | None = None,
        files: list | None = None,
        setup_context: dict | None = None,
    ) -> TaskResult:
        ctx = setup_context or {}
        response = my_agent.chat(
            message,
            session_id=session_id,
            auth_token=ctx.get("auth_token"),
        )
        return TaskResult(
            message=response.text,
            session_id=session_id or response.session_id or "default",
        )


# ---- Run ----

result = Netra.simulation.run_simulation(
    name="My Simulation",
    dataset_id="dataset-123",
    task=MyAgentTask(),
    hooks=hooks,
    max_concurrency=3,
)

Full TypeScript scaffold example (same pattern):

import { BaseTask, Netra } from "netra-sdk";
import type { ProcessedFile, SimulationHooks, TaskResult } from "netra-sdk";

// ---- Hooks ----

async function beforeAll(): Promise<Record<string, any> | null> {
  // TODO: replace with your actual setup code
  const employee = await yourApi.createEmployee({ name: "Test User", role: "admin" });
  return { employeeId: employee.id };
}
(beforeAll as any).description =
  "Create the test employee and assign the admin role before any scenario runs.";

async function beforeEach(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null> {
  const employeeId = sharedContext?.employeeId;
  // TODO: replace with your actual login logic
  const token = await yourApi.login({ employeeId });
  return { authToken: token };
}
(beforeEach as any).description =
  "Obtain a fresh auth token before every scenario.";

async function setupRefundScenario(
  sharedContext: Record<string, any> | null,
): Promise<Record<string, any> | null> {
  // sharedContext already includes employeeId + authToken from beforeAll + beforeEach
  const refundAccount = await yourApi.createRefundAccount();
  return { refundAccountId: refundAccount.id };
}
(setupRefundScenario as any).description =
  "Create a refund account for the refund scenario only.";

async function teardownRefundScenario(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  try {
    await yourApi.deleteRefundAccount(setupContext?.refundAccountId);
  } catch {
    // Catch locally so teardown errors do not mark the item postscript_failed
  }
}
(teardownRefundScenario as any).description =
  "Delete the refund account after the refund scenario.";

async function afterEach(
  result: Record<string, any>,
  setupContext: Record<string, any> | null,
): Promise<void> {
  try {
    await yourApi.logout({ token: setupContext?.authToken });
  } catch {
    // ignore
  }
}
(afterEach as any).description =
  "Log out after every scenario regardless of outcome.";

async function afterAll(
  results: Record<string, any>,
  sharedContext: Record<string, any> | null,
): Promise<void> {
  // TODO: replace with your actual teardown code
  await yourApi.deleteEmployee(sharedContext?.employeeId);
}
(afterAll as any).description =
  "Delete the test employee once all scenarios have finished.";

const hooks: SimulationHooks = {
  beforeAll,
  beforeEach,
  before: {
    "item-refund-request": setupRefundScenario,
  },
  after: {
    "item-refund-request": teardownRefundScenario,
  },
  afterEach,
  afterAll,
};

// ---- Task ----

class MyAgentTask extends BaseTask {
  async run(
    message: string,
    sessionId?: string | null,
    files?: ProcessedFile[] | null,
    setupContext?: Record<string, any> | null,
  ): Promise<TaskResult> {
    const ctx = setupContext || {};
    const response = await myAgent.chat(message, {
      sessionId,
      authToken: ctx.authToken,
    });
    return {
      message: response.text,
      sessionId: sessionId || response.sessionId || "default",
    };
  }
}

// ---- Run ----

const result = await Netra.simulation.runSimulation({
  name: "My Simulation",
  datasetId: "dataset-123",
  task: new MyAgentTask(),
  hooks,
  maxConcurrency: 3,
});

Step 4: Flag Hook Requirement in the Plan

In the plan, add a "Hooks" section under the Dataset Configuration table when hooks are needed. When no hooks are needed, state explicitly:

**Hooks:** Not required — scenarios are independent.

When hooks are needed, list each hook type with a one-line description and specify which scenarios require item-specific hooks. Use language-appropriate names in the plan (before_all for Python, beforeAll for TypeScript):

**Hooks:**
- `before_all` / `beforeAll`: Create shared test employee and assign admin role
- `before_each` / `beforeEach`: Obtain a fresh auth token before every scenario
- `before`: Per-scenario hooks keyed by dataset item ID:
  - Refund Request scenario: Create refund account
- `after`: Per-scenario hooks keyed by dataset item ID:
  - Refund Request scenario: Delete refund account
- `after_each` / `afterEach`: Log out after every scenario
- `after_all` / `afterAll`: Delete the test employee

In the Netra UI Conversation tab for a scenario:

• before_all / beforeAll, before_each / beforeEach, after_each / afterEach, and after_all / afterAll appear for every scenario in that run (run-level metadata)
• before / after appear only when that scenario's dataset item ID was registered in the hooks dict (item-level metadata)

Important: The actual dataset item ID values will only be available after creating the dataset items via MCP. In the generated scaffold code, use placeholder IDs (e.g., "item-refund-request", "item-balance-inquiry") and instruct the user to replace these with the actual IDs returned by netra_create_dataset_item.

Then include the generated scaffold code in a collapsible code block under the plan summary.

Phase 4: Generate the Plan

Produce the plan in the exact format below. This is the primary output the user will review.

PLAN FORMAT — output this exactly, filling in the generated values:

# NDD Evaluation Plan

## Dataset Configuration

| Field | Value |
|---|---|
| **Name** | {generated-dataset-name} |
| **Description** | {generated-description} |
| **Turn Type** | {single | multi} |
| **Execution Method** | {Netra.evaluation.run_test_suite / Netra.evaluation.runTestSuite | Netra.simulation.run_simulation / Netra.simulation.runSimulation} |
| **Total Items** | {count} |

### Execution Strategy

Single-turn datasets:
- Python: `Netra.evaluation.run_test_suite()`
- TypeScript: `await Netra.evaluation.runTestSuite({...})`

Multi-turn datasets:
- Python: `Netra.simulation.run_simulation()`
- TypeScript: `await Netra.simulation.runSimulation({...})`

If Turn Type = `multi`, include:

```text
Required Task Implementation:
⚠️ User must provide a BaseTask implementation compatible with Netra Simulation.

If Turn Type = multi and hooks were identified in Phase 3g, include a Hooks section immediately after Execution Strategy:

### Hooks (Pre/Post Scripts)

{Either "Not required — scenarios are independent." OR a list like:}

- `before_all` / `beforeAll`: {one-line description}
- `before_each` / `beforeEach`: {one-line description, if used}
- `before`: {one-line description, if used}
- `after`: {one-line description, if used}
- `after_each` / `afterEach`: {one-line description, if used}
- `after_all` / `afterAll`: {one-line description}

**Generated scaffold code** (requires user to fill in TODO sections; language = Phase 0.5 detection):

{paste the full scaffold code block from Phase 3g here — Python OR TypeScript, not both}

Items

Item 1: {scenario-name}

Category: {happy-path | negative | edge-case | safety

…(truncated)