dot-do evaluate.do
You are an expert in ai-experiments for systematic AI evaluation.
When to Use
Activate this skill when benchmarking models, running parameter sweeps, comparing LLM outputs, or evaluating agents before production.
Core API
import { Experiment, cartesian } from 'ai-experiments'
// Runs 8 combinations: 2 models × 4 temperatures
const results = await Experiment('sentiment-test', {
models: ['claude-sonnet-4-6', 'gpt-4o'],
temperature: [0, 0.3, 0.7, 1.0],
prompt: ({ input }) => [`Classify sentiment: "${input}"`],
inputs: ['Amazing product!', 'Completely broken.'],
})
// Results saved to .ai/experiments/sentiment-test/
Config Reference
| Field | Type | Description |
|---|---|---|
models |
string | string[] |
Model(s) to test |
temperature |
number | number[] |
Temperature sweep |
seed |
number | number[] |
For reproducibility |
prompt |
(params) => string[] |
Prompt template |
inputs |
any[] | async fn |
Test inputs |
schema |
object |
JSON schema for structured output |
expectedOutputs |
any[] |
For pass/fail validation |
Vitest Integration
import { createRunner } from 'ai-experiments'
const runner = createRunner({ outputDir: '.ai/experiments' })
it('classifies intents', runner.run({
name: 'intent-classifier',
models: ['claude-sonnet-4-6'],
temperature: [0, 0.3],
prompt: ({ input }) => [`Classify intent: "${input}"`],
inputs: ['Book a flight', 'Cancel subscription'],
}))
Evaluation Workflow
- Define experiment — models, temperatures, representative inputs
- Run —
Experiment()executes all combinations - Review — markdown report in
.ai/experiments/<name>/ - Choose winner — best model + temperature for production
- Deploy — configure chosen params in
ai-functionsoragents.do
Best Practices
- Use
seedfor reproducibility when comparing runs - Test with real inputs, not toy examples
- Run nightly experiments on critical paths via
workflows-doevery() - Commit experiment results to version control for regression detection