How do I install the Vitest Evals skill?

Run npx skillmds@latest add getsentry/vitest-evals in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Vitest Evals safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Vitest Evals?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Vitest Evals free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Vitest Evals?

Sentry (@getsentry) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Vitest Evals by getsentry — Agent Skill

Name: Vitest Evals
Author: Sentry

Vitest Evals

Use when authoring, reviewing, or debugging harness-backed vitest-evals suites, custom Harness adapters, first-party ai-sdk or pi-ai harness integrations, judges, replay, reporter-facing normalized run data, or examples and docs for these APIs.

Sentry f83bdc1 12 files · 34.6 KB Updated Sep 17, 2026 845 repo stars

File contents

vitest-evals

Use the harness-backed API as the only authoring model.

First Steps

Read the package, app, or eval file being changed.
Identify the runtime target, then open only the needed reference.
Keep suites close to Vitest: one harness per describeEval(...), explicit run(...) inside each test, ordinary expect(...) assertions over the returned result.

Reference Router

Need	Open
Write or review a normal eval suite	`references/suite-authoring.md`
Build a custom app `Harness` without a first-party adapter	`references/custom-harness.md`
Integrate AI SDK `generateText`, `generateObject`, tools, or an AI SDK-style agent	`references/harness-ai-sdk.md`
Integrate a Pi AI or Pi Mono-style agent	`references/harness-pi-ai.md`
Add custom judges, suite judges, built-in judges, or `toSatisfyJudge(...)` assertions	`references/judges-and-assertions.md`
Assert on message, tool-call, or span history	`references/utilities.md`
Configure tool recording or replay	`references/tool-replay.md`
Diagnose failures, missing traces, odd output, or choose verification commands	`references/troubleshooting.md`

Runtime Defaults

Import describeEval(...), judges, and helpers from vitest-evals.
Bind exactly one harness to a suite.
Call run(input) where the test should execute the system.
Assert on result.output for app-facing behavior.
Use toolCalls(result) and message helpers for normalized session assertions.
Use spans(result), spansByKind(result, kind), and failedSpans(result) for span assertions.
Keep HarnessRun, NormalizedSession, usage, artifacts, and tool records JSON-serializable.
Keep judge model calls on judges. Use createJudge("Name", assess) for custom judges; use the provider-helper overload only when multiple judges reuse setup and need curried run options.
Put scenario-owned criteria on the input value. Put direct-check expected values in Vitest case rows. Pass per-case judge criteria through explicit matcher options, and suite-wide criteria through judge config.
Custom judges should use createJudge(...) for stable reporter labels.

Verification

Prefer the smallest command that covers the edited files:

Task	Command
Lint file	`pnpm exec biome lint path/to/file.ts`
Format file	`pnpm exec biome format --write path/to/file.ts`
Test file	`pnpm exec vitest run path/to/file.test.ts -c vitest.config.ts`
Eval file	`pnpm exec vitest run path/to/file.eval.ts -c vitest.config.ts --reporter=./packages/vitest-evals/src/reporter.ts`
Type surface	`pnpm typecheck`
Package build	`pnpm build`

---name: vitest-evalsdescription: Use when authoring, reviewing, or debugging harness-backed vitest-evals suites, custom Harness adapters, first-party ai-sdk or pi-ai harness integrations, judges, replay, reporter-facing normalized run data, or examples and docs for these APIs.---# vitest-evalsUse the harness-backed API as the only authoring model.## First Steps1. Read the package, app, or eval file being changed.2. Identify the runtime target, then open only the needed reference.3. Keep suites close to Vitest: one harness per `describeEval(...)`, explicit `run(...)` inside each test, ordinary `expect(...)` assertions over the returned result.## Reference Router| Need | Open ||------|------|| Write or review a normal eval suite | `references/suite-authoring.md` || Build a custom app `Harness` without a first-party adapter | `references/custom-harness.md` || Integrate AI SDK `generateText`, `generateObject`, tools, or an AI SDK-style agent | `references/harness-ai-sdk.md` || Integrate a Pi AI or Pi Mono-style agent | `references/harness-pi-ai.md` || Add custom judges, suite judges, built-in judges, or `toSatisfyJudge(...)` assertions | `references/judges-and-assertions.md` || Assert on message, tool-call, or span history | `references/utilities.md` || Configure tool recording or replay | `references/tool-replay.md` || Diagnose failures, missing traces, odd output, or choose verification commands | `references/troubleshooting.md` |## Runtime Defaults- Import `describeEval(...)`, judges, and helpers from `vitest-evals`.- Bind exactly one `harness` to a suite.- Call `run(input)` where the test should execute the system.- Assert on `result.output` for app-facing behavior.- Use `toolCalls(result)` and message helpers for normalized session assertions.- Use `spans(result)`, `spansByKind(result, kind)`, and `failedSpans(result)` for span assertions.- Keep `HarnessRun`, `NormalizedSession`, usage, artifacts, and tool records JSON-serializable.- Keep judge model calls on judges. Use `createJudge("Name", assess)` for custom judges; use the provider-helper overload only when multiple judges reuse setup and need curried run options.- Put scenario-owned criteria on the input value. Put direct-check expected values in Vitest case rows. Pass per-case judge criteria through explicit matcher options, and suite-wide criteria through judge config.- Custom judges should use `createJudge(...)` for stable reporter labels.## VerificationPrefer the smallest command that covers the edited files:| Task | Command ||------|---------|| Lint file | `pnpm exec biome lint path/to/file.ts` || Format file | `pnpm exec biome format --write path/to/file.ts` || Test file | `pnpm exec vitest run path/to/file.test.ts -c vitest.config.ts` || Eval file | `pnpm exec vitest run path/to/file.eval.ts -c vitest.config.ts --reporter=./packages/vitest-evals/src/reporter.ts` || Type surface | `pnpm typecheck` || Package build | `pnpm build` |

Frequently asked questions