Create MCP Eval

create-mcp-eval

Generate eval tests for MCP servers using @mcpjam/sdk.

Read this file first. It carries the two things you need before writing any code — what to ask the user, and the rules the generated tests must follow — and routes you to the rest only when you actually need it.

Reference map

Load a reference when you reach the step that needs it, not before.

1. Context Gathering

Before generating any code, collect the following from the user:

If the user provides an Agent Brief (markdown with ## Tools table), parse it to auto-populate tool names, descriptions, parameters, and suggested eval scenarios. See references/agent-brief.md.

Provider Selection (REQUIRED)

You MUST ask the developer which LLM provider they want before generating any code. Do not default to any provider.

Supported Providers:

Once the user selects a provider, use the corresponding env var name and model format in all generated code:

• {LLM_ENV_VAR} — e.g., OPENAI_API_KEY
• {LLM_MODEL} — e.g., openai/gpt-4o-mini
• {LLM_KEY_EXAMPLE} — e.g., sk-...

Test Runner Selection

Before generating tests, check what the codebase already uses:

• package.json scripts and devDependencies for jest or vitest
• Config files: jest.config.*, vitest.config.*, vite.config.*

Then:

• If Jest is present, use Jest (and ts-jest if TypeScript).
• If Vitest is present, use Vitest.
• If neither is present, default to Vitest.
• If the developer prefers no test framework, the @mcpjam/sdk classes (EvalTest, EvalSuite) can run standalone — call .run() directly and check results in a plain script without Jest/Vitest.

In all cases, use @mcpjam/sdk for the eval harness (HostRunner, EvalTest, EvalSuite, validators).

5. Generation Guidelines

Follow these rules when generating eval test files:

1. Deterministic suite first — always include a deterministic test section using HostRunner.mock() that validates the test structure itself without requiring LLM calls or server connections.

2. One EvalTest per tool — create a separate EvalTest for each tool you want to evaluate. Each test should prompt the runner with a natural-language request and assert the correct tool was selected.

3. Single-shot LLM tests are non-deterministic — a single runner.run() may not select the expected tool every time. For single-shot tests, prefer saving results to MCPJam without hard-asserting (expect(...).toBe(true)). Use EvalTest with iterations >= 3 and assert on accuracy() for reliable pass/fail gates. Reserve hard asserts for high-confidence cases (negative tests, multi-turn with clear context).

4. Write unambiguous prompts for similar tools — when a server has tools with overlapping descriptions (e.g., create_view vs export_to_excalidraw), prompts must reference the tool's unique action. Mention specific verbs, targets, or outcomes. Bad: "Share my diagram". Good: "Export and upload my diagram to excalidraw.com so I can open it in a browser".

5. Multi-turn for related tools — when tools logically chain together (e.g., get_user then list_workspaces), create a multi-turn test using { context: previousResult }.

6. Negative test — always include at least one test that verifies the runner does NOT call tools when given an irrelevant prompt (e.g., "What is the capital of France?"). Use matchNoToolCalls().

7. Reasonable defaults:

   • iterations: 5 for EvalTest runs
   • timeoutMs: 60_000 for LLM tests
   • maxSteps: 8 for HostRunner
   • retries: 1 for flaky network tolerance
   • concurrency: 5 (default, no need to set explicitly)

8. Timeout on test cases — set explicit timeouts on it() blocks: 90_000 for single-turn, 120_000 for multi-turn and suite tests.

9. Always await — every runner.run(), test.run(), suite.run(), reporter.record*(), and reporter.finalize() is async. Never forget await.

10. One reporter per file — create the reporter at module level to save results to MCPJam, and finalize in afterAll. Never create multiple reporters in the same file.

11. Use describe.skip for missing credentials — wrap LLM tests in conditional describe blocks so CI runs cleanly without secrets.

12. Match the repo's test runner — check package.json and config files for an existing test framework before generating. Only default to Vitest if the repo has no test runner. If the user prefers no framework at all, use @mcpjam/sdk classes (EvalTest.run(), EvalSuite.run()) standalone in a plain script.

13. Log key metrics — add console.log statements for accuracy, tool calls, and latency so CI output is informative.