Running Promptfoo Evals
Every eval here runs two roles over provider APIs: a generator, the model under test, and a judge, the model that grades each llm-rubric assertion. evals/README.md covers what to test and how to write prompts and rubrics; this is the procedure for running them. Work the phases in order — do not start at phase 5 because content already exists.
1. Set up and budget the run
Budget before anything else. These providers are used on free tiers, and a run you cannot finish teaches nothing.
- Confirm a generator key and a judge key are set — presence only, never the value:
echo "generator: ${GROQ_API_KEY:+set}" and echo "judge: ${GOOGLE_API_KEY:+set}". If either is missing, stop and report; do not extrapolate a result from a run you did not make.
- Both roles are pinned: the generator in
evals/prompts/lib/providers.yaml, the judge in package.json's eval:graded script. Use the pinned pair by default. The one hard requirement is that they are different models — never grade a generator with itself — and any result you report names both.
- Send every recorded run through
npm run eval:graded, the only place --grader, --delay, and -j are set. Do not hand-type them, and do not copy a value out of an older results doc: that value documents that run, not this one. Do not pass --providers either: it replaces the pinned generator with a bare one, so the run is no longer pinned and its results are mislabeled — evals/README.md, "Writing eval prompts" has the detail. --providers echo is the probe case: it returns the rendered prompt in place of a generator answer, so you see exactly what the generator would be sent — but every llm-rubric still calls the judge, so even a probe spends judge calls.
- Learn the provider's limits before launching: read its rate-limit documentation for the pinned model's tier, then make one minimal request to the endpoint with response headers shown. On an OpenAI-compatible endpoint the
x-ratelimit-* headers carry the per-minute token window (limit, remaining, reset) and the per-day request count; a daily token cap has no header of its own, so it surfaces only when a 429 arrives — retry-after gives the wait, and the body names which limit tripped. Header names and what they count are provider-specific and change, so confirm them against the provider's own page.
# Example against the current generator's OpenAI-compatible endpoint; the URL and model id come from providers.yaml.
curl -s -X POST https://api.groq.com/openai/v1/chat/completions \
-H "Authorization: Bearer $GROQ_API_KEY" -H "Content-Type: application/json" \
--data '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
-D - | grep -iE 'ratelimit|retry-after|"error"'
- Do the arithmetic before launching. Tokens ≈ bytes / 4, so
wc -c skills/<skill>/SKILL.md gives the injection cost per test; a with-skill run costs roughly tests × (that + the generator's configured max_tokens), plus one judge call per rubric, against whatever the day has left. A baseline run drops the injection. If the total does not fit the remaining budget, do not launch it whole.
- Run sequentially, one config per command.
-j 1 is pinned, and never run two graded evals at once — across terminals or across teammates. Limits are per account, not per process, so a 429 on either role kills both runs; chain them instead, leaving a full per-minute window between: <run one> && sleep 60 && <run two>. Each config already wires lib/skill-prompt.mjs and lib/providers.yaml, so there is nothing to assemble by hand; if you do write a Promptfoo command yourself, it still names exactly one --config — given several, Promptfoo merges their defaultTest blocks and the last skillFile wins.
--delay paces requests; it does not create budget. A rate-limit failure that survives correct pacing with no concurrency is exhausted capacity — stop, take the reset time from the direct call above, and do not loop.
- When the remaining window or day is smaller than the run, run fewer tests per invocation — never a larger delay.
--filter-pattern '<regex against the test description>' runs only the matching tests (quote it; npm run passes it through a shell), and it is also how you probe one test of a run type before committing the whole config. --filter-failing <the run's .json output> re-runs only what failed or errored — phase 8 writes that file.
2. Write the tests before the content
3. Run the baseline, with the skill withheld
npm run eval:graded -- --config evals/prompts/<skill>.yaml --var injectSkill=false
- An explicit, implicit, or anti-pattern test that passes at baseline has no discriminating power. Redesign it. Never keep it as evidence, and never delete a test to make a run green. A negative test may pass at baseline — that is what it is for.
- Read every baseline answer, not the verdict column. A hallucination the judge waved through is a failure of the test design, not evidence of a gap.
- One run is one sample. The generator is nondeterministic even at
temperature: 0, so a claim rests on a FAIL you have read, never on a verdict alone.
4. Write content only for what failed at baseline
5. Run with the skill injected
npm run eval:graded -- --config evals/prompts/<skill>.yaml — injection is the default. A demonstrated gap is a FAIL at baseline and a PASS with the skill. Passing both, failing both, or no with-skill run at all means there is no claim yet.
6. Diagnose before retrying
Promptfoo prints its table and writes the --output files only after the run finishes, and the progress bar needs a TTY on stderr, so under an agent a healthy run and a dead one look identical. Never infer progress from elapsed time — establish which one it is.
pgrep -fl promptfoo # take the node pid, not the sh wrapper — the wrapper holds no sockets
lsof -nP -p <node-pid> -i # ESTABLISHED = still waiting on a provider
ls -l ~/.promptfoo/promptfoo.db* # finished rows land in the local db; the -wal file's mtime stops advancing on a stall
- An open connection is the signal; CPU is not — a run waiting on a provider sits at 0%. Kill only when the connection is gone and the mtime is frozen.
- Ask which limit tripped, with the direct call from phase 1. A spent minute and a spent day are reported identically: a queue timeout, or a rate-limit error on the request that was retrying. A request larger than the per-minute window is different — the provider rejects it outright and Promptfoo records an error row carrying the body.
0 passed / 0 failed alongside "There were some errors" is no data, not a result, and an empty row is not a FAIL — the call never reached the model. Record either as no data; retrying more than once spends budget for no evidence.
- A killed run writes no
--output files and leaves any previous ones in place, so check mtime and row count before reading. Its finished rows are in the local db: npx promptfoo export eval latest -o <path>.json writes them out in the same shape as a run's own JSON output, and calls no provider.
- Read the JSON output, not only the CSV. The CSV flattens each cell to the verdict, the answer, and the top-level grader reason; the JSON keeps every assertion's own result and reason, and the full error behind an ERROR row. In it, read
results.results[].response.output beside results.results[].gradingResult.componentResults[].reason, matched on testCase.description — the reason alone is never enough to judge a verdict by, and a reason that is an error message, or empty, is a judge failure, not a verdict. Never edit a grader's reason in a recorded CSV.
- Check the tail of every FAIL for truncation, and the tail of every
Must NOT PASS for an early stop. A cut-off answer can lose the part the rubric wanted, and it satisfies a prohibition for free.
7. When a retest is needed, and when it is not
Needed:
- A claim is added, corrected, or removed — run the tests covering it with the skill, and baseline any new claim before writing it.
- A rubric or prompt is reworded — that test's baseline is now invalid, so re-run that test with
--filter-pattern, not the whole config. The reverse holds too: when you change a skill's wording, grep its rubric for the phrase you removed, in the same commit, because an outdated rubric grades leniently and comes back green.
- The pinned generator or judge changes — re-run every config.
The scheduled CI run (eval.yml) grades every config whose skill, eval config or harness changed since its last graded run, and each one again a week after its last grade; that run is not something a contributor repeats by hand.
Not needed:
- Wording that keeps the meaning, formatting, link fixes, typos.
- Moving a claim between skills — the corpus is unchanged; the test moves with the claim to the owning skill's config.
- Removing content the generator did not need in order to pass — the baseline PASS is the evidence. Record it in the results doc as the reason for the cut.
- A
status: process skill — eval-exempt, reviewed by reading it against the files it describes.
8. Record the result concisely
- Write every run to two files, CSV and JSON:
--output evals/results/local/<skill>-<baseline|with-skill>.csv evals/results/local/<skill>-<baseline|with-skill>.json — the flag takes several paths, and the directory is gitignored. The CSV is the record; the JSON is the diagnosis (phase 6) and the input to --filter-failing. scripts/run-evals.js writes the same pair in CI. Promote only a run you cite: commit its CSV as evals/results/latest/<skill>-<baseline|with-skill>_<YYYY-MM-DD>.csv.
evals/results/<skill>.md is one table — a row per test carrying its type, the baseline outcome, the with-skill outcome, and the failure mode compressed into the cell. Head it with a Run: line naming the date, the generator model, and the judge model; close with a bold pass/fail tally. Add nothing else unless a verdict needed human reading — then say which one, and where the raw output is.
status: follows the run: verified only when the pinned run passes, provisional otherwise.
Related Skills
References
- evals/README.md — prompts, rubrics, baseline-probe methodology, provider setup
package.json's eval:graded script — the single source of truth for grader, delay, and concurrency
- The current generator's rate-limit page — the tier limits and the response headers
- Promptfoo 0.122 —
package.json pins ^0.122.0; every flag and behavior named above was checked against that version's CLI help and source, so re-check them when the pin moves
This skill is a snapshot. Where a primary source contradicts it — the References above, MapLibre's current documentation, or what MapLibre does when you run it — that source wins. Follow it, then report the disagreement, citing the source and your MapLibre version: editing your installed copy helps no one else and is overwritten on the next update.
1---2name: maplibre-running-evals3description: The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs. Use when starting a new skill and needing a baseline before writing content, when a run stalls or returns no data, when a provider rate-limits, or when deciding whether an edit needs a retest.4metadata:5 source: https://skillmd.com/skills/maplibre/maplibre-running-evals6 status: process7 internal: true8---910# Running Promptfoo Evals1112Every eval here runs two roles over provider APIs: a **generator**, the model under test, and a **judge**, the model that grades each `llm-rubric` assertion. [evals/README.md](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md) covers what to test and how to write prompts and rubrics; this is the procedure for running them. Work the phases in order — do not start at phase 5 because content already exists.1314## 1. Set up and budget the run1516Budget before anything else. These providers are used on free tiers, and a run you cannot finish teaches nothing.1718- **Confirm a generator key and a judge key are set — presence only, never the value:** `echo "generator: ${GROQ_API_KEY:+set}"` and `echo "judge: ${GOOGLE_API_KEY:+set}"`. If either is missing, stop and report; do not extrapolate a result from a run you did not make.19- **Both roles are pinned:** the generator in [`evals/prompts/lib/providers.yaml`](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/prompts/lib/providers.yaml), the judge in `package.json`'s `eval:graded` script. Use the pinned pair by default. The one hard requirement is that they are different models — never grade a generator with itself — and any result you report names both.20- **Send every recorded run through `npm run eval:graded`,** the only place `--grader`, `--delay`, and `-j` are set. Do not hand-type them, and do not copy a value out of an older results doc: that value documents that run, not this one. Do not pass `--providers` either: it replaces the pinned generator with a bare one, so the run is no longer pinned and its results are mislabeled — [evals/README.md, "Writing eval prompts"](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md#writing-eval-prompts) has the detail. `--providers echo` is the probe case: it returns the rendered prompt in place of a generator answer, so you see exactly what the generator would be sent — but every `llm-rubric` still calls the judge, so even a probe spends judge calls.21- **Learn the provider's limits before launching:** read its rate-limit documentation for the pinned model's tier, then make one minimal request to the endpoint with response headers shown. On an OpenAI-compatible endpoint the `x-ratelimit-*` headers carry the per-minute token window (limit, remaining, reset) and the per-day request count; a daily token cap has no header of its own, so it surfaces only when a `429` arrives — `retry-after` gives the wait, and the body names which limit tripped. Header names and what they count are provider-specific and change, so confirm them against the provider's own page.2223```bash24# Example against the current generator's OpenAI-compatible endpoint; the URL and model id come from providers.yaml.25curl -s -X POST https://api.groq.com/openai/v1/chat/completions \26 -H "Authorization: Bearer $GROQ_API_KEY" -H "Content-Type: application/json" \27 --data '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \28 -D - | grep -iE 'ratelimit|retry-after|"error"'29```3031- **Do the arithmetic before launching.** Tokens ≈ bytes / 4, so `wc -c skills/<skill>/SKILL.md` gives the injection cost per test; a with-skill run costs roughly tests × (that + the generator's configured `max_tokens`), plus one judge call per rubric, against whatever the day has left. A baseline run drops the injection. If the total does not fit the remaining budget, do not launch it whole.32- **Run sequentially, one config per command.** `-j 1` is pinned, and never run two graded evals at once — across terminals or across teammates. Limits are per account, not per process, so a `429` on either role kills both runs; chain them instead, leaving a full per-minute window between: `<run one> && sleep 60 && <run two>`. Each config already wires `lib/skill-prompt.mjs` and `lib/providers.yaml`, so there is nothing to assemble by hand; if you do write a Promptfoo command yourself, it still names exactly one `--config` — given several, Promptfoo merges their `defaultTest` blocks and the last `skillFile` wins.33- **`--delay` paces requests; it does not create budget.** A rate-limit failure that survives correct pacing with no concurrency is exhausted capacity — stop, take the reset time from the direct call above, and do not loop.34- **When the remaining window or day is smaller than the run, run fewer tests per invocation — never a larger delay.** `--filter-pattern '<regex against the test description>'` runs only the matching tests (quote it; `npm run` passes it through a shell), and it is also how you probe one test of a run type before committing the whole config. `--filter-failing <the run's .json output>` re-runs only what failed or errored — phase 8 writes that file.3536## 2. Write the tests before the content3738- Copy [`evals/prompts/TEMPLATE.yaml`](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/prompts/TEMPLATE.yaml) to `evals/prompts/<skill>.yaml` and write four tests — explicit, implicit, anti-pattern, negative — before the skill has any content. [evals/README.md, "Writing eval prompts"](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md#writing-eval-prompts) is the how; this skill is the running procedure.3940## 3. Run the baseline, with the skill withheld4142`npm run eval:graded -- --config evals/prompts/<skill>.yaml --var injectSkill=false`4344- **An explicit, implicit, or anti-pattern test that passes at baseline has no discriminating power.** Redesign it. Never keep it as evidence, and never delete a test to make a run green. A negative test may pass at baseline — that is what it is for.45- **Read every baseline answer, not the verdict column.** A hallucination the judge waved through is a failure of the test design, not evidence of a gap.46- **One run is one sample.** The generator is nondeterministic even at `temperature: 0`, so a claim rests on a FAIL you have read, never on a verdict alone.4748## 4. Write content only for what failed at baseline4950- Content the model already gets right is a cost, not a benefit — [CONTRIBUTING.md, "Cutting content the model already gets right"](https://github.com/maplibre/maplibre-agent-skills/blob/main/CONTRIBUTING.md#cutting-content-the-model-already-gets-right).51- A claim belongs in the skill that owns it, not the config you had open. Test placement drives content placement, and the alternative is one claim living in three drifting copies.5253## 5. Run with the skill injected5455`npm run eval:graded -- --config evals/prompts/<skill>.yaml` — injection is the default. **A demonstrated gap is a FAIL at baseline and a PASS with the skill.** Passing both, failing both, or no with-skill run at all means there is no claim yet.5657## 6. Diagnose before retrying5859Promptfoo prints its table and writes the `--output` files only after the run finishes, and the progress bar needs a TTY on stderr, so under an agent a healthy run and a dead one look identical. Never infer progress from elapsed time — establish which one it is.6061```bash62pgrep -fl promptfoo # take the node pid, not the sh wrapper — the wrapper holds no sockets63lsof -nP -p <node-pid> -i # ESTABLISHED = still waiting on a provider64ls -l ~/.promptfoo/promptfoo.db* # finished rows land in the local db; the -wal file's mtime stops advancing on a stall65```6667- **An open connection is the signal; CPU is not** — a run waiting on a provider sits at 0%. Kill only when the connection is gone **and** the mtime is frozen.68- **Ask which limit tripped**, with the direct call from phase 1. A spent minute and a spent day are reported identically: a queue timeout, or a rate-limit error on the request that was retrying. A request larger than the per-minute window is different — the provider rejects it outright and Promptfoo records an error row carrying the body.69- **`0 passed / 0 failed` alongside "There were some errors" is no data, not a result,** and an empty row is not a FAIL — the call never reached the model. Record either as no data; retrying more than once spends budget for no evidence.70- **A killed run writes no `--output` files** and leaves any previous ones in place, so check mtime and row count before reading. Its finished rows are in the local db: `npx promptfoo export eval latest -o <path>.json` writes them out in the same shape as a run's own JSON output, and calls no provider.71- **Read the JSON output, not only the CSV.** The CSV flattens each cell to the verdict, the answer, and the top-level grader reason; the JSON keeps every assertion's own result and reason, and the full error behind an ERROR row. In it, read `results.results[].response.output` beside `results.results[].gradingResult.componentResults[].reason`, matched on `testCase.description` — the reason alone is never enough to judge a verdict by, and a reason that is an error message, or empty, is a judge failure, not a verdict. Never edit a grader's reason in a recorded CSV.72- **Check the tail of every FAIL for truncation, and the tail of every `Must NOT` PASS for an early stop.** A cut-off answer can lose the part the rubric wanted, and it satisfies a prohibition for free.7374## 7. When a retest is needed, and when it is not7576Needed:7778- A claim is added, corrected, or removed — run the tests covering it with the skill, and baseline any **new** claim before writing it.79- A rubric or prompt is reworded — that test's baseline is now invalid, so re-run that test with `--filter-pattern`, not the whole config. The reverse holds too: when you change a skill's wording, grep its rubric for the phrase you removed, in the same commit, because an outdated rubric grades leniently and comes back green.80- The pinned generator or judge changes — re-run every config.8182The scheduled CI run (`eval.yml`) grades every config whose skill, eval config or harness changed since its last graded run, and each one again a week after its last grade; that run is not something a contributor repeats by hand.8384Not needed:8586- Wording that keeps the meaning, formatting, link fixes, typos.87- Moving a claim between skills — the corpus is unchanged; the test moves with the claim to the owning skill's config.88- Removing content the generator did not need in order to pass — the baseline PASS is the evidence. Record it in the results doc as the reason for the cut.89- A `status: process` skill — eval-exempt, reviewed by reading it against the files it describes.9091## 8. Record the result concisely9293- **Write every run to two files, CSV and JSON:** `--output evals/results/local/<skill>-<baseline|with-skill>.csv evals/results/local/<skill>-<baseline|with-skill>.json` — the flag takes several paths, and the directory is gitignored. The CSV is the record; the JSON is the diagnosis (phase 6) and the input to `--filter-failing`. `scripts/run-evals.js` writes the same pair in CI. Promote only a run you cite: commit its CSV as `evals/results/latest/<skill>-<baseline|with-skill>_<YYYY-MM-DD>.csv`.94- **`evals/results/<skill>.md` is one table** — a row per test carrying its type, the baseline outcome, the with-skill outcome, and the failure mode compressed into the cell. Head it with a `Run:` line naming the date, the generator model, and the judge model; close with a bold pass/fail tally. Add nothing else unless a verdict needed human reading — then say which one, and where the raw output is.95- **`status:` follows the run:** `verified` only when the pinned run passes, `provisional` otherwise.9697## Related Skills9899- [**maplibre-skill-authoring**](../maplibre-skill-authoring/SKILL.md) — capturing a session's findings as a skill draft; this skill runs its eval step.100101## References102103- [evals/README.md](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md) — prompts, rubrics, baseline-probe methodology, provider setup104- `package.json`'s `eval:graded` script — the single source of truth for grader, delay, and concurrency105- [The current generator's rate-limit page](https://console.groq.com/docs/rate-limits) — the tier limits and the response headers106- Promptfoo 0.122 — `package.json` pins `^0.122.0`; every flag and behavior named above was checked against that version's CLI help and source, so re-check them when the pin moves107108---109110**This skill is a snapshot.** Where a primary source contradicts it — the References above, MapLibre's current documentation, or what MapLibre does when you run it — that source wins. Follow it, then [report the disagreement](https://github.com/maplibre/maplibre-agent-skills/issues/new?template=ai-failure-report.md), citing the source and your MapLibre version: editing your installed copy helps no one else and is overwritten on the next update.