# Mastra Evals

> Mastra Evaluation and Testing guide - built-in scorers, custom scorers, datasets, experiments, and CI integration. Covers createScorer(), built-in scorer factories like createAnswerRelevancyScorer(), runEvals(), dataset management, and experiments that compare models, prompts, or agent variants against the same dataset.

- Skill: `artsmc-claude-dev-agents/mastra-evals` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add artsmc-claude-dev-agents/mastra-evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/artsmc-claude-dev-agents/mastra-evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: artsmc (https://skillmd.com/u/artsmc-claude-dev-agents)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/artsmc-claude-dev-agents/mastra-evals

---


# Mastra Evaluation and Testing

Comprehensive guide for evaluating AI agent quality with Mastra. Covers 17 built-in scorer factories, custom scorer creation with `createScorer()`, datasets for reproducible benchmarks, experiments for comparison, and CI pipeline integration.

## Usage

```bash
/mastra-evals
```

Provides context for:
- Scorer factory functions (e.g., `createAnswerRelevancyScorer()`)
- Import paths: `@mastra/evals/scorers/llm` and `@mastra/evals/scorers/code`
- `createScorer()` from `@mastra/core/scores`
- `runEvals({ target, scorers, data })`
- Dataset management (create, addItems, experiments)
- Experiment comparison and CI integration
- Agent-level scorer configuration with `sampling`