# Writing Evals > Write and iterate on opt-in tool-selection evals for the Vantage MCP server — promptfoo setup, prompt matrix, distractors, failure diagnosis, and JSON/Pages workflow. Use after the user chooses to include evals, or when working on an eval they explicitly requested. - Skill: `vantage-sh/writing-evals` (Agent Skill) - Install (CLI): `npx skillmds add vantage-sh/writing-evals` - Raw SKILL.md: https://api.skillmd.com/api/skills/vantage-sh/writing-evals/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: vantage-sh (https://skillmd.com/u/vantage-sh) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/vantage-sh/writing-evals --- # Writing evals Unit tests prove a tool wires up and the API call shape is right. **Evals prove the description + zod schema are good enough that a model can find and call the tool from a natural-language prompt.** Tool authoring conventions (description style, zod `.describe()` strings) live in `.agents/skills/writing-mcp-tools/SKILL.md`; this skill covers the eval harness and how to iterate when rows fail. ## Opt-in and execution approval Evals are not a default part of adding or changing a tool. If the user has not already explicitly requested evals, ask whether they want them included. If they opt in, also ask whether the provider API key for the model they intend to use is configured in the ignored `.env` file. If they decline, do not add or modify eval case files, result JSON, or the generated site. Opting in to eval authoring does **not** authorize running an eval. Model-backed evals make fresh API calls, may incur cost, and can create or replace result JSON. Unless the user explicitly asks to execute the eval: - Write or update only the case file under `evals/cases/`. - Do not run `npm run eval`, `npm run eval:all`, or a filtered rerun. - Do not create or modify files under `evals/results/` or regenerate the eval site as a consequence of the authoring task. - At handoff, provide the exact targeted command the user can run and mention `npm run eval -- --list-models` for the approved model catalog. You may also offer to run it, but wait for an explicit follow-up. If the user wants evals but says the required key is not configured, author the case file only and explain the missing setup at handoff. If the user asks to run an eval but has not confirmed credential setup, ask whether the required provider key is configured in `.env`. If they have not selected a model, ask which approved model and effort they want before executing it. Explain that promptfoo loads credentials from `.env` and that every invocation makes fresh, uncached model calls. A repository requirement to produce a baseline before a PR is a pending verification step to report, not authorization to spend API credits. ## Stack and commands [promptfoo](https://www.promptfoo.dev) + Vercel AI SDK v6 + `@ai-sdk/anthropic` + `@ai-sdk/openai`. The custom provider loads tools from the live `registerTool` registry and asks the model to select one. | Command | Purpose | | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | `npm run eval -- --tool --model gpt-5.6-sol-high` | Run one exact tool against one approved model after explicit approval | | `npm run eval -- --tool --tool --model gpt-5.6-sol-high` | Run an explicit set of exact tools | | `npm run eval -- --resource --model gpt-5.6-sol-high` | Run every eval case in one resource directory | | `npm run eval -- --tool --dry-run` | Preview the exact resolved scope without model calls | | `npm run eval:all -- --model gpt-5.6-sol-high` | Deliberately refresh every case against one model | | `npm run eval -- --list-models` | Print the approved model × effort catalog | | `npm run eval -- --tool --filter-failing evals/results///.json --model gpt-5.6-sol-high` | Re-run failures only | | `npm run eval:site` | Merge stored JSON → `evals/site/index.html` | | `npm run eval:view` | promptfoo's local viewer | `--tool` is an exact selector and may be repeated. `--resource` selects every case under the matching `evals/cases//` directory, may be repeated, and accepts an optional trailing slash. The selectors can be combined and are resolved as a deduplicated union before Promptfoo starts. Use `--dry-run` to print that resolved scope without selecting a model or making model calls. `--model` is required except for `--dry-run`. The slug is an approved model id, optionally plus an effort suffix (`gpt-5.6-sol-high`). Models that do not expose effort (today: `claude-haiku-4-5`) take the bare id. Effort is optional even when the model supports it — `gpt-5.6-sol` uses the provider default. Dotted forms like `gpt-5.6.sol-high` are accepted and stored as `gpt-5.6-sol-high`. The catalog lives in `evals/_lib/models.ts`. promptfoo loads the ignored `.env` file; set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` there for the selected model's provider before the first run. Do not infer permission to run from the presence of a key. Every eval invocation makes fresh model calls; promptfoo's response cache is disabled. An unfiltered run replaces the selected result JSON. A run using a partial promptfoo filter merges rerun cells into the retained baseline and preserves cells the filter omitted. Extra promptfoo flags pass through: `--filter-failing `, `--filter-metadata phrasing=direct`. ## Persistence and what to run JSON under `evals/results///.json` is the durable store. HTML is disposable. ``` evals/ cases//.eval.ts # mirrors src/tools// results///.json # committed per-model slice site/ # generated report.html → GitHub Pages (not committed) ``` - **Adding a tool:** after the user opts in to evals, write `evals/cases//.eval.ts`. Unless the user explicitly asked for execution and confirmed the required API key is configured, stop there and provide `npm run eval -- --tool --model ` as the next-step command. When explicitly authorized, the run writes `evals/results///.json` and leaves every other tool's files untouched; then run `npm run eval:site` and commit the new JSON. - **Editing an existing tool:** update the case file only when the user opts in to eval work. Re-run that tool only when the user explicitly asks and confirms credential setup; its per-tool JSON is replaced. - **Refreshing one resource:** use `npm run eval -- --resource --model ` only after the user explicitly authorizes the model-backed run. This replaces the per-tool result JSON for every selected case in that resource and leaves other resources untouched. - **Filtered rerun:** partial filters such as `--filter-failing` and `--filter-metadata` replace matching cells by provider and case identity while preserving every stored cell the filter omitted. - **Full-model refresh:** the normal `eval` command rejects a missing `--tool` or `--resource` selector. Use `npm run eval:all -- --model ` only when you intentionally want to refresh every case for that model. - **Merge conflicts** on a JSON file: take one side, re-run that tool, commit the result. - **Browsing results:** `npm run eval:site && open evals/site/index.html`. GitHub Pages at regenerates HTML from committed JSON on every push to `main` that touches `evals/results/`. No model API keys in CI. ## Layout ``` evals/ _lib/ evalArgs.ts # CLI parsing + targeted/full-run safety guard evalScope.ts # exact tool/resource case discovery and selection models.ts # approved models × effort levels; --model slug parser distractors.ts # registered-tool sampler + optional named distractors buildAiSdkTools.ts # reads from the live registerTool registry → AI SDK tool() defs runToolSelection.ts # { prompt, model, toolNames } → { toolCalls, text } provider.ts # promptfoo custom provider for the selected model assertToolCalls.ts # flexible tool-call match buildCases.ts # cartesian product of prompts × phrasing run-eval.ts # CLI: promptfoo eval + split into results///.json generate-site.ts # merge JSON → evals/site/index.html cases//.eval.ts promptfooconfig.ts ``` OpenAI tools are built with `strict: false`. The Responses API otherwise fills omitted optional args with `""` / placeholders (`x`, `.*`, …), which makes exact arg scoring fail even when tool selection is correct. Do not switch OpenAI evals to Chat Completions solely to avoid that — some models (e.g. `gpt-5.6-luna`) reject function tools on `/v1/chat/completions` unless `reasoning_effort` is `none`. Case files live under `evals/cases//` so they stay out of Vitest's path and match `src/tools//`. Use the `.eval.ts` suffix so they are distinct from the tool file and the unit test. `promptfooconfig.ts` picks up every `**/*.eval.ts` file automatically. The adapter imports `src/tools` once and reads tools out of the live `registerTool` registry — adding a new tool to the codebase makes it available to evals automatically; you just need to write its case file. ## The cases Every tool's case file contains exactly **two prompts** — one direct and one inferred/indirect — and each prompt runs against the **one** `--model` you pass with the target plus four distractors loaded. That's two cells total: | **Direct** | Can the model call the tool when the user explicitly names its registered identifier while four other tools compete? | | ------------ | ----------------------------------------------------------------------------------------------------------------------- | | **Inferred** | Does the description cover the indirect phrasing well enough to beat four competing tools? | To compare models, run the same `--tool` again with a different `--model`; each slug gets its own JSON under `evals/results//`. ## File template `evals/cases/current-user/get-myself.eval.ts` is the canonical reference. The shape per tool: ```ts import { buildToolCases } from "../../_lib/buildCases"; const TARGET = ""; export default function generateTests() { return buildToolCases({ target: TARGET, resource: "", // Optional: always include high-signal siblings; remaining slots are sampled. distractors: [""], directPrompts: [ { input: "Use to .", expected: [{ toolName: TARGET, input: {/* expected args */} }] }, ], inferredPrompts: [ { input: "", expected: [{ toolName: TARGET, input: {/* expected args */} }] }, ], }); } ``` `buildToolCases` tags every case with `metadata.tool` and `metadata.resource` so results land under `evals/results///` and optional Promptfoo metadata filters can narrow an already-resolved scope. The wrapper resolves `--tool` and `--resource` exactly before Promptfoo loads the selected case files. The scorer requires **exactly one** tool call with the expected name and args. Missing calls, extra calls, and multiple expected calls fail the cell. The v1 eval matrix does not cover abstention, negative, or multi-tool prompts. ## Writing prompts - **Direct prompts** explicitly include the target's exact registered tool identifier — for example, "Use `get-myself` to inspect the current Vantage credentials." Naming only the concept, title, or resource is not direct enough. These prompts test explicit invocation and argument extraction. - **Inferred prompts** describe the user's _goal_ without naming the tool — "I want to make sure we don't blow past $50k this quarter" → `create-budget`. They test description coverage and any disambiguating context. - Every prompt expects exactly one tool call. Negative, abstention, and multi-tool cases are outside the v1 eval scope. - Write prompts a Vantage MCP user would _actually_ send. Generic phrasings with no product context (e.g. `"Who am I?"`) put unfair pressure on the description — models may read them as general knowledge questions, not Vantage account queries. Drop or rephrase prompts like that rather than padding the tool description to catch them. - Use exactly one direct prompt and one inferred prompt per tool. Each additional prompt creates another model call and increases API spend. ## Distractors `pickTools(target)` returns the target plus four distractors sampled from every other tool in the live registry. The target-derived shuffle is deterministic: the sample is broad without changing between reruns. Adding or removing registered tools can change the sample. When a tool has close neighbours (for example `list-budgets`, `list-folders`, and `list-cost-reports`), name the high-signal siblings on the eval definition: ```ts return buildToolCases({ target: TARGET, resource: "budgets", distractors: ["list-folders", "list-cost-reports"], // prompts... }); ``` - Named distractors are loaded first and the remaining slots are sampled automatically. Provide at most four unique, registered tool names; do not include the target itself. - `buildToolCases` sets `options.disableVarExpansion: true` so promptfoo does not expand the `distractors` / `expected` arrays into separate test cases. ## Reading failures The failing prompt tells you _what to fix_: | Failure pattern | Most likely cause | Fix | | ---------------------------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | **Direct** fails | The prompt does not use the exact registered name, or required arguments are unclear. | Correct the tool identifier in the prompt; tighten zod argument descriptions. | | **Inferred** selects no tool | The description does not cover the indirect phrasing. | Add one sentence connecting the goal to the tool (cap: one sentence). | | A distractor wins | The target and a competing tool are not sufficiently distinct. | Inspect the loaded distractors and disambiguate the target description or argument schema. | | Only the smallest model fails one prompt | Often a prompt-fairness issue, not a description issue. | Drop or rephrase the prompt. Don't grow the description to win a single weak-model row. | | Wrong args (right tool) | A zod field is missing a useful `.describe()`, or a required field looks optional. | Tighten `.describe()` strings; add `.default()` if the value is genuinely defaultable. | **Iteration order when an eval fails:** 1. **Zod schema first** — better `.describe()` on the args, tighter constraints (`min(1)`, `enum(...)`), or a `default(...)` where the model shouldn't have to guess. 2. **Tool description second** — add one sentence if needed; do not write paragraphs to win a single failure. 3. **Prompt third** — if the prompt isn't realistic, fix the prompt rather than the description. 4. **Accept the floor** — if only the smallest model fails on a fair prompt after the above, that's a model floor, not a tool bug. The rule is: **don't write to the eval.** The eval validates the description and schema; making the description longer just to game one row defeats the point. If you're tempted, double-check the prompt is one a user would actually send. ## Checklist - [ ] The user explicitly opted in to evals; otherwise no files under `evals/` were added or modified. - [ ] The user was asked whether the provider API key they intend to use is configured in `.env`. - [ ] `evals/cases//.eval.ts` contains exactly one direct prompt and one inferred prompt. - [ ] The direct prompt contains the exact registered tool identifier; the inferred prompt does not name the tool. - [ ] Every prompt expects exactly one tool call with exact args. - [ ] Both prompts are things a Vantage MCP user would actually send. - [ ] Named distractors documented if the default pool is too weak for sibling tools. - [ ] If execution was not explicitly requested, no model-backed eval was run and the handoff includes the targeted command plus model-selection instructions. - [ ] If execution was explicitly requested, the user confirmed credential setup and selected the model and effort before the run. - [ ] After an authorized run, `npm run eval -- --tool --model ` is green and the new result JSON is staged as the baseline. - [ ] After an authorized run, `npm run eval:site` has been run locally if you want to inspect the report before push.