# Prompt Evaluation Runner

> Use when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.

- Skill: `aibot88/prompt-evaluation-runner` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add aibot88/prompt-evaluation-runner`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aibot88/prompt-evaluation-runner/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: aibot88 (https://skillmd.com/u/aibot88)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/aibot88/prompt-evaluation-runner

---


# Prompt Evaluation Runner

## When to use
Use this skill when you need to evaluate an LLM app, test a prompt, or run red-teaming/vulnerability scans against a target model or application.

## Requirements / Checks
1. Check if an evaluation tool is defined in project deps, scripts, lockfiles, or local toolchain.
2. Do not run external commands like `npx Prompt evaluation@latest` directly.
3. If missing, ask before adding a local dev dependency or using an ephemeral runner.
4. Confirm expected cost, provider, API keys, and network target before execution.

## Workflow
1. **Define risk**: State target behavior, failure mode, provider(s), and budget limits.
2. **Choose assertions**: Prefer deterministic checks first: exact/contains/regex/JSON/schema/javascript/python/cost/latency.
3. **Use model graders sparingly**: Pin grader provider/model and explain cost/non-determinism.
4. **Configure minimally**: Keep config to description, env refs, prompts, providers, default assertions, and tests.
5. **Handle env safely**: Use templated env references such as `{{env.NAME}}`; never hardcode keys.
6. **Execute locally**: Run smallest suite first. Ask before long, paid, red-team, or production-targeted runs.
7. **Analyze failures**: Separate prompt failures, provider variance, flaky graders, bad fixtures, and config mistakes.

## Safety Constraints
- Do NOT log, echo, or store secrets (API keys) in configuration files or chat output.
- Do NOT run evaluations against production endpoints without user consent.
- Avoid executing arbitrary remote code or unvetted plugins during evaluation.

## Validation / Done Criteria
- Evaluation config is valid, minimal, and uses safe env refs.
- Deterministic assertions exist where possible.
- Run scope, provider, and cost are reported.
- Results are summarized without leaking sensitive data.

## References
- `references/eval-config-patterns.md`