Explore To Sdk Evals

explore-to-sdk-evals

1. Purpose

This skill converts pre-existing MCPJam Explore test cases (in the same document above, under ## Explore-generated test cases) into runnable @mcpjam/sdk eval tests.

Rules:

• Generate exactly the cases provided — do not invent new cases, do not skip any, do not reword user prompts (use the exact query text from each case’s fenced block).
• The brief above already includes ## Tools, resources/prompts if any, suggested scenarios, and ## Explore-generated test cases — parse titles, prompts, negative flags, expected tool-call shapes, and expected output from there.
• For full SDK API (advanced validators, suite patterns, error handling), see the create-mcp-eval skill in the MCPJam SDK or package documentation — this file stays minimal.

2. Framework detection

Before emitting imports, inspect the target repo:

1. Read package.json: scripts and devDependencies for jest or vitest.
2. Look for jest.config.*, vitest.config.*, or vite.config.* that references Vitest.

Apply:

3. Quick API reference (Explore flows only)

import { MCPClientManager, HostRunner, createEvalRunReporter } from "@mcpjam/sdk";
import {
  matchNoToolCalls,
  matchToolCallWithPartialArgs,
  matchToolArgument,
  matchToolArgumentWith,
} from "@mcpjam/sdk";

Connection

• const manager = new MCPClientManager();
• HTTP: await manager.connectToServer(SERVER_ID, { url: MCP_SERVER_URL });
• Stdio: await manager.connectToServer(SERVER_ID, { command: "node", args: ["./server.js"] });
• const tools = await manager.getToolsForAiSdk([SERVER_ID]);
• await manager.disconnectAllServers(); in afterAll

Agent

• new HostRunner({ tools, model: MODEL, apiKey: LLM_API_KEY, maxSteps: 8, mcpClientManager: manager }) — pass the same MCPClientManager instance you used for connectToServer / getToolsForAiSdk. mcpClientManager is required for widgetSnapshots on reported results (MCP App HTML capture via readResource), which powers MCP App replay in MCPJam eval traces. Omitting it still runs tools correctly but traces in the app will have no embedded widget replay.
• apiKey and model must match the LLM provider the user chose (see §5); there is no default assumption that the key is an OpenAI key.
• await runner.run(caseQuery) — use verbatim caseQuery from Explore; for slow MCP + multi-step chains use { timeoutMs } (or timeout) so the prompt does not abort before tools finish (see §6).
• Multi-turn only if a single Explore case clearly requires follow-up in one narrative; otherwise one prompt per it() (see §4c)

Result inspection

• result.hasToolCall("tool_name")
• result.toolsCalled() — string[]
• result.getToolCalls() — for validators

Validators

• Negative: matchNoToolCalls(result.toolsCalled())
• Partial args: matchToolCallWithPartialArgs("tool", { key: value }, result.getToolCalls()) — use only when those values are real literals from Explore, not placeholders (§4).
• Single-arg exact / predicate: matchToolArgument, matchToolArgumentWith (see create-mcp-eval / SDK validators).

Optional MCPJam upload

• createEvalRunReporter({ suiteName, apiKey, strict: true, mcpClientManager: manager, serverNames: [SERVER_ID], ... }) — pass the same connected MCPClientManager as for HostRunner so uploads include serverReplayConfigs (HTTP MCP only; stdio connections do not produce replayable server config in the SDK). That is what sets hasServerReplayConfig in MCPJam and enables Replay this run / server-side MCP replay in the UI.
• await reporter.recordFromPrompt(result, { caseTitle, passed, expectedToolCalls?, isNegativeTest? })
• await reporter.finalize() in afterAll before disconnectAllServers() (with timeout). finalize calls getServerReplayConfigs() on the manager; disconnecting first clears client state and uploads without replay metadata. finalize() can throw (network, DNS, 401) even when all eval assertions passed. For best-effort upload: wrap in try/catch, log a warning, and continue; or gate uploads on something like MCPJAM_REPORTING=1 so local/CI runs do not go red purely on reporting.
• UI replay still needs LLM keys in MCPJam Settings for whatever provider the suite uses (e.g. OpenRouter); the app does not store provider API keys on the run.

4. Per-case translation

Use the Explore case title as the human-readable basis for it("...") description (sanitize quotes if needed). Use the exact user prompt from the ``` fenced block as runner.run(\...)` argument.

Placeholder values in tool_name({ ... }) lines: Explore often shows illustrative arguments — e.g. create_view({ elements: "" }) usually means “this argument key exists / this shape,” not “the value must be the empty string.” Blindly passing { elements: "" } into matchToolCallWithPartialArgs will fail on real calls. Prefer hasToolCall("tool_name") when the brief does not state a concrete literal. When you need stricter checks: use matchToolArgumentWith("tool", "key", predicate, calls) (e.g. “present and non-empty”), or matchToolArgument only when Explore documents an actual expected value. Reserve matchToolCallWithPartialArgs for real literals listed in the case, not for "", "...", or other stand-ins.

4a. Positive (expected tool calls, not negative)

For each expected tool line like tool_name({arg: val}):

it("Explore case title here", async () => {
  const result = await runner.run(`paste exact query from Explore`);
  expect(result.hasToolCall("tool_name")).toBe(true);
  // Only when Explore lists real literals — not placeholders (see §4 intro).
  // expect(
  //   matchToolCallWithPartialArgs("tool_name", { arg: literalVal }, result.getToolCalls()),
  // ).toBe(true);
  if (reporter) {
    await reporter.recordFromPrompt(result, {
      caseTitle: "Explore case title here",
      passed: result.hasToolCall("tool_name"),
      expectedToolCalls: [{ toolName: "tool_name" }],
    });
  }
}, 90_000);

4b. Negative test (Negative test: / NEG-style case)

it("Explore case title here", async () => {
  const result = await runner.run(`paste exact query from Explore`);
  expect(matchNoToolCalls(result.toolsCalled())).toBe(true);
  if (reporter) {
    await reporter.recordFromPrompt(result, {
      caseTitle: "Explore case title here",
      passed: matchNoToolCalls(result.toolsCalled()),
      isNegativeTest: true,
    });
  }
}, 90_000);

4c. Multiple expected tools in one Explore case

Prefer one it() that asserts all listed tools (unless the case text explicitly describes a conversation turn — then use context: r1 for the second prompt).

it("Explore case title", async () => {
  const result = await runner.run(`exact query`);
  expect(result.hasToolCall("first_tool")).toBe(true);
  expect(result.hasToolCall("second_tool")).toBe(true);
  // Optional: arg checks only for real literals (§4) — else hasToolCall is enough
}, 90_000);

Do not merge two separate Explore cases into one it() — one Explore case → one it() unless the plan above explicitly allows multi-tool-in-one-case as above.

5. Scaffold template

5a. Required environment variables — ask the user before generating code

Do not emit the full scaffold until the user has supplied values (or explicit env var names) for everything needed to run the evals locally and in CI.

1. Prompt the user for each required variable below. If they prefer names only, use placeholders like process.env.ANTHROPIC_API_KEY and document those names in a short comment or README snippet in the same response — but the skill’s default is to confirm what must be set, not to guess secrets.

1. Never imply OpenAI-only copy: avoid phrasing like “paste your OpenAI key” unless the user already said OpenAI. Default language: “which LLM provider and API key env var are you using?”
2. Same rule as create-mcp-eval: ask which LLM provider, model, and env var name before generating code; never silently default to a single vendor.
3. Required assistant output (before codegen): After the user answers, echo an explicit env contract so the model cannot “skip” to OpenAI-flavored code without mirroring what was agreed:

Env contract (confirmed — use in generated file):
- provider: <e.g. anthropic | openai | google | azure | other>
- EVAL_MODEL: <exact model id string>
- LLM API key env var: <e.g. ANTHROPIC_API_KEY>
- MCP: <MCP_SERVER_URL and any other vars, or stdio command source>
- Optional reporting: MCPJAM_API_KEY present? <yes|no>; fail run on finalize error, or try/catch + warn; optional gate MCPJAM_REPORTING=1? <as agreed>

Do not generate the eval file until this block is filled from user-provided choices (not inferred defaults).

5b. Template (after env contract is clear)

Use connection from the brief blockquote (Connection: or Connection (stdio):) for MCP_SERVER_URL or the joined command string.

// ─── Config ─────────────────────────────────────────────────────────────────
const MCP_SERVER_URL = process.env.MCP_SERVER_URL ?? "<from brief>";
// Replace <PROVIDER_ENV> with the env var the user named (e.g. ANTHROPIC_API_KEY, not only OPENAI_API_KEY).
const LLM_API_KEY = process.env.<PROVIDER_ENV>!;
const MODEL = process.env.EVAL_MODEL ?? "<provider/model>";
const SERVER_ID = "<server id from brief # MCP Server Brief: name>";
const MCPJAM_API_KEY = process.env.MCPJAM_API_KEY;
// See §5c: put `import "dotenv/config"` first in this file (or use Vitest setupFiles) so
// RUN_LLM_TESTS sees .env in workers — config-only dotenv is unreliable.
const RUN_LLM_TESTS = Boolean(LLM_API_KEY && MCP_SERVER_URL);

Hooks and RUN_LLM_TESTS: File-level beforeAll / afterAll run even when a later describe is .skipped (Vitest does this; Jest can behave similarly depending on suite structure). That can still hit the MCP when you meant “no API key → skip everything.” Nest beforeAll / afterAll inside the same describe block that is toggled with (RUN_LLM_TESTS ? describe : describe.skip), so skipped suites do not connect. (Alternative: a single file-level beforeAll that starts with if (!RUN_LLM_TESTS) return; — but nesting is clearer.)

(RUN_LLM_TESTS ? describe : describe.skip)("Explore cases", () => {
  let manager: MCPClientManager;
  let runner: HostRunner;
  let reporter: ReturnType<typeof createEvalRunReporter> | undefined;

  beforeAll(async () => {
    manager = new MCPClientManager();
    await manager.connectToServer(SERVER_ID, { url: MCP_SERVER_URL });
    const tools = await manager.getToolsForAiSdk([SERVER_ID]);
    runner = new HostRunner({
      tools,
      model: MODEL,
      apiKey: LLM_API_KEY,
      maxSteps: 8,
      mcpClientManager: manager,
    });
    if (MCPJAM_API_KEY) {
      reporter = createEvalRunReporter({
        suiteName: "Explore-exported evals",
        apiKey: MCPJAM_API_KEY,
        strict: true,
        serverNames: [SERVER_ID],
        mcpClientManager: manager,
        expectedIterations: <count Explore cases with reporter records>,
      });
    }
  }, 120_000);

  afterAll(async () => {
    if (reporter?.getAddedCount()) {
      try {
        await reporter.finalize();
      } catch (e) {
        console.warn("MCPJam reporter finalize failed (non-fatal):", e);
      }
    }
    await manager?.disconnectAllServers();
  }, 120_000);

  // one it() per Explore case — §4
});

Set expectedIterations to the exact number of recordFromPrompt / iteration records you emit (same contract as create-mcp-eval). Teams that want no upload attempts unless explicitly enabled can guard createEvalRunReporter with e.g. MCPJAM_REPORTING=1. Omit the try/catch around finalize if reporting failures must fail the run; document the choice in the env contract (§5a.4).

5c. .env files and Vitest workers

RUN_LLM_TESTS = Boolean(process.env…) at module top level often runs before process.env is populated if dotenv only loads inside vitest.config.ts (config runs in a different context than test workers; evals then look “always skipped”).

Mitigations (pick one or combine):

• Add import "dotenv/config" (or project equivalent) as the first import in the eval test file so workers see secrets before RUN_LLM_TESTS is computed.
• Set Vitest test.setupFiles (or setupFiles) to a small module that loads .env before any test file body runs.
• In CI, inject secrets via the runner env (no .env); locally, prefer setupFiles over relying on config-time dotenv alone.

Optionally: fail loudly in dev when vars are missing (throw or describe with one it that explains what to export) instead of only describe.skip, if that matches the team’s expectation.

6. Operational reminders

• Default it(..., 90_000) per test. Use 120_000–180_000 and a matching runner.run(..., { timeoutMs: ... }) when the brief implies streaming, animation, slow MCP, or long multi-tool runs (wall-clock often exceeds 90s end-to-end). Raise beforeAll / afterAll hook timeouts in Vitest if hooks approach default limits (hookTimeout in config or per-hook timeout argument).
• await every runner.prompt, reporter.record*, reporter.finalize, connectToServer, disconnectAllServers.
• One reporter per file; finalize() in afterAll with a generous timeout — call finalize() before disconnectAllServers() so replay config is still available from the manager (§3, §5b). Treat upload as non-fatal when agreed (§3, §5b).
• mcpClientManager on createEvalRunReporter: pass the connected manager here in addition to HostRunner — uploads need this to attach serverReplayConfigs (hasServerReplayConfig); the runner’s field alone does not populate the report payload (§3).
• MCPJam UI replay uses stored MCP replay plus your Settings API keys for the suite’s LLM providers; keys are not stored on the run.
• maxSteps: 8 on HostRunner unless the Explore case implies a longer tool chain.
• mcpClientManager on HostRunner: pass the connected manager whenever results are reported to MCPJam so widgetSnapshots (MCP App HTML) are captured; without it, uploaded traces only have messages + spans.
• Wrap LLM suites with (RUN_LLM_TESTS ? describe : describe.skip) and nest lifecycle hooks inside that describe (§5b) so skipped runs do not connect to the MCP.
• If the user has not yet listed required env vars, stop and ask for them (§5a) before producing runnable files; emit the env contract block (§5a.4) before codegen.

7. Explicit non-goal

Full @mcpjam/sdk encyclopedia, project greenfield setup, and long-form templates — use the create-mcp-eval skill for that.