Writing Evals

Writing evals

Unit tests prove a tool wires up and the API call shape is right. Evals prove the description + zod schema are good enough that a model can find and call the tool from a natural-language prompt. Tool authoring conventions (description style, zod .describe() strings) live in .agents/skills/writing-mcp-tools/SKILL.md; this skill covers the eval harness and how to iterate when rows fail.

Opt-in and execution approval

Evals are not a default part of adding or changing a tool. If the user has not already explicitly requested evals, ask whether they want them included. If they opt in, also ask whether the provider API key for the model they intend to use is configured in the ignored .env file. If they decline, do not add or modify eval case files, result JSON, or the generated site.

Opting in to eval authoring does not authorize running an eval. Model-backed evals make fresh API calls, may incur cost, and can create or replace result JSON. Unless the user explicitly asks to execute the eval:

• Write or update only the case file under evals/cases/.
• Do not run npm run eval, npm run eval:all, or a filtered rerun.
• Do not create or modify files under evals/results/ or regenerate the eval site as a consequence of the authoring task.
• At handoff, provide the exact targeted command the user can run and mention npm run eval -- --list-models for the approved model catalog. You may also offer to run it, but wait for an explicit follow-up.

If the user wants evals but says the required key is not configured, author the case file only and explain the missing setup at handoff. If the user asks to run an eval but has not confirmed credential setup, ask whether the required provider key is configured in .env. If they have not selected a model, ask which approved model and effort they want before executing it. Explain that promptfoo loads credentials from .env and that every invocation makes fresh, uncached model calls. A repository requirement to produce a baseline before a PR is a pending verification step to report, not authorization to spend API credits.

Stack and commands

promptfoo + Vercel AI SDK v6 + @ai-sdk/anthropic + @ai-sdk/openai. The custom provider loads tools from the live registerTool registry and asks the model to select one.

--tool is an exact selector and may be repeated. --resource selects every case under the matching evals/cases/<resource>/ directory, may be repeated, and accepts an optional trailing slash. The selectors can be combined and are resolved as a deduplicated union before Promptfoo starts. Use --dry-run to print that resolved scope without selecting a model or making model calls.

--model is required except for --dry-run. The slug is an approved model id, optionally plus an effort suffix (gpt-5.6-sol-high). Models that do not expose effort (today: claude-haiku-4-5) take the bare id. Effort is optional even when the model supports it — gpt-5.6-sol uses the provider default. Dotted forms like gpt-5.6.sol-high are accepted and stored as gpt-5.6-sol-high. The catalog lives in evals/_lib/models.ts.

promptfoo loads the ignored .env file; set ANTHROPIC_API_KEY or OPENAI_API_KEY there for the selected model's provider before the first run. Do not infer permission to run from the presence of a key. Every eval invocation makes fresh model calls; promptfoo's response cache is disabled. An unfiltered run replaces the selected result JSON. A run using a partial promptfoo filter merges rerun cells into the retained baseline and preserves cells the filter omitted.

Extra promptfoo flags pass through: --filter-failing <file>, --filter-metadata phrasing=direct.

Persistence and what to run

JSON under evals/results/<model>/<resource>/<tool>.json is the durable store. HTML is disposable.

evals/
  cases/<resource>/<tool>.eval.ts             # mirrors src/tools/<resource>/
  results/<model>/<resource>/<tool>.json      # committed per-model slice
  site/                                       # generated report.html → GitHub Pages (not committed)

• Adding a tool: after the user opts in to evals, write evals/cases/<resource>/<tool>.eval.ts. Unless the user explicitly asked for execution and confirmed the required API key is configured, stop there and provide npm run eval -- --tool <tool> --model <approved-model> as the next-step command. When explicitly authorized, the run writes evals/results/<model>/<resource>/<tool>.json and leaves every other tool's files untouched; then run npm run eval:site and commit the new JSON.
• Editing an existing tool: update the case file only when the user opts in to eval work. Re-run that tool only when the user explicitly asks and confirms credential setup; its per-tool JSON is replaced.
• Refreshing one resource: use npm run eval -- --resource <resource> --model <approved-model> only after the user explicitly authorizes the model-backed run. This replaces the per-tool result JSON for every selected case in that resource and leaves other resources untouched.
• Filtered rerun: partial filters such as --filter-failing and --filter-metadata replace matching cells by provider and case identity while preserving every stored cell the filter omitted.
• Full-model refresh: the normal eval command rejects a missing --tool or --resource selector. Use npm run eval:all -- --model <model> only when you intentionally want to refresh every case for that model.
• Merge conflicts on a JSON file: take one side, re-run that tool, commit the result.
• Browsing results: npm run eval:site && open evals/site/index.html. GitHub Pages at https://vantage-sh.github.io/vantage-mcp-server/ regenerates HTML from committed JSON on every push to main that touches evals/results/. No model API keys in CI.

Layout

evals/
  _lib/
    evalArgs.ts           # CLI parsing + targeted/full-run safety guard
    evalScope.ts          # exact tool/resource case discovery and selection
    models.ts             # approved models × effort levels; --model slug parser
    distractors.ts        # registered-tool sampler + optional named distractors
    buildAiSdkTools.ts    # reads from the live registerTool registry → AI SDK tool() defs
    runToolSelection.ts   # { prompt, model, toolNames } → { toolCalls, text }
    provider.ts           # promptfoo custom provider for the selected model
    assertToolCalls.ts    # flexible tool-call match
    buildCases.ts         # cartesian product of prompts × phrasing
    run-eval.ts           # CLI: promptfoo eval + split into results/<model>/<resource>/<tool>.json
    generate-site.ts      # merge JSON → evals/site/index.html
  cases/<resource>/<tool>.eval.ts
  promptfooconfig.ts

OpenAI tools are built with strict: false. The Responses API otherwise fills omitted optional args with "" / placeholders (x, .*, …), which makes exact arg scoring fail even when tool selection is correct. Do not switch OpenAI evals to Chat Completions solely to avoid that — some models (e.g. gpt-5.6-luna) reject function tools on /v1/chat/completions unless reasoning_effort is none. Case files live under evals/cases/<resource>/ so they stay out of Vitest's path and match src/tools/<resource>/. Use the .eval.ts suffix so they are distinct from the tool file and the unit test. promptfooconfig.ts picks up every **/*.eval.ts file automatically. The adapter imports src/tools once and reads tools out of the live registerTool registry — adding a new tool to the codebase makes it available to evals automatically; you just need to write its case file.

The cases

Every tool's case file contains exactly two prompts — one direct and one inferred/indirect — and each prompt runs against the one --model you pass with the target plus four distractors loaded. That's two cells total:

To compare models, run the same --tool again with a different --model; each slug gets its own JSON under evals/results/<model>/.

File template

evals/cases/current-user/get-myself.eval.ts is the canonical reference. The shape per tool:

import { buildToolCases } from "../../_lib/buildCases";

const TARGET = "<tool-name>";

export default function generateTests() {
  return buildToolCases({
    target: TARGET,
    resource: "<resource>",
    // Optional: always include high-signal siblings; remaining slots are sampled.
    distractors: ["<sibling-tool>"],
    directPrompts: [
      { input: "Use <tool-name> to <perform its purpose>.", expected: [{ toolName: TARGET, input: {/* expected args */} }] },
    ],
    inferredPrompts: [
      { input: "<prompt where intent has to be inferred>", expected: [{ toolName: TARGET, input: {/* expected args */} }] },
    ],
  });
}

buildToolCases tags every case with metadata.tool and metadata.resource so results land under evals/results/<model>/<resource>/ and optional Promptfoo metadata filters can narrow an already-resolved scope. The wrapper resolves --tool and --resource exactly before Promptfoo loads the selected case files.

The scorer requires exactly one tool call with the expected name and args. Missing calls, extra calls, and multiple expected calls fail the cell. The v1 eval matrix does not cover abstention, negative, or multi-tool prompts.

Writing prompts

• Direct prompts explicitly include the target's exact registered tool identifier — for example, "Use get-myself to inspect the current Vantage credentials." Naming only the concept, title, or resource is not direct enough. These prompts test explicit invocation and argument extraction.
• Inferred prompts describe the user's goal without naming the tool — "I want to make sure we don't blow past $50k this quarter" → create-budget. They test description coverage and any disambiguating context.
• Every prompt expects exactly one tool call. Negative, abstention, and multi-tool cases are outside the v1 eval scope.
• Write prompts a Vantage MCP user would actually send. Generic phrasings with no product context (e.g. "Who am I?") put unfair pressure on the description — models may read them as general knowledge questions, not Vantage account queries. Drop or rephrase prompts like that rather than padding the tool description to catch them.
• Use exactly one direct prompt and one inferred prompt per tool. Each additional prompt creates another model call and increases API spend.

Distractors

pickTools(target) returns the target plus four distractors sampled from every other tool in the live registry. The target-derived shuffle is deterministic: the sample is broad without changing between reruns. Adding or removing registered tools can change the sample.

When a tool has close neighbours (for example list-budgets, list-folders, and list-cost-reports), name the high-signal siblings on the eval definition:

return buildToolCases({
  target: TARGET,
  resource: "budgets",
  distractors: ["list-folders", "list-cost-reports"],
  // prompts...
});

• Named distractors are loaded first and the remaining slots are sampled automatically. Provide at most four unique, registered tool names; do not include the target itself.
• buildToolCases sets options.disableVarExpansion: true so promptfoo does not expand the distractors / expected arrays into separate test cases.

Reading failures

The failing prompt tells you what to fix:

Iteration order when an eval fails:

1. Zod schema first — better .describe() on the args, tighter constraints (min(1), enum(...)), or a default(...) where the model shouldn't have to guess.
2. Tool description second — add one sentence if needed; do not write paragraphs to win a single failure.
3. Prompt third — if the prompt isn't realistic, fix the prompt rather than the description.
4. Accept the floor — if only the smallest model fails on a fair prompt after the above, that's a model floor, not a tool bug.

The rule is: don't write to the eval. The eval validates the description and schema; making the description longer just to game one row defeats the point. If you're tempted, double-check the prompt is one a user would actually send.

Checklist

• The user explicitly opted in to evals; otherwise no files under evals/ were added or modified.
• The user was asked whether the provider API key they intend to use is configured in .env.
• evals/cases/<resource>/<tool>.eval.ts contains exactly one direct prompt and one inferred prompt.
• The direct prompt contains the exact registered tool identifier; the inferred prompt does not name the tool.
• Every prompt expects exactly one tool call with exact args.
• Both prompts are things a Vantage MCP user would actually send.
• Named distractors documented if the default pool is too weak for sibling tools.
• If execution was not explicitly requested, no model-backed eval was run and the handoff includes the targeted command plus model-selection instructions.
• If execution was explicitly requested, the user confirmed credential setup and selected the model and effort before the run.
• After an authorized run, npm run eval -- --tool <tool> --model <selected-model> is green and the new result JSON is staged as the baseline.
• After an authorized run, npm run eval:site has been run locally if you want to inspect the report before push.