# Prompt Evaluator

> Evaluate agent prompt effectiveness using accuracy metrics and token constraints.

- Skill: `rmazrim/prompt-evaluator` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rmazrim/prompt-evaluator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rmazrim/prompt-evaluator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: RMAzrim (https://skillmd.com/u/rmazrim)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/rmazrim/prompt-evaluator

---


# Prompt Evaluator

## Prerequisites & Dependencies
- Node.js 18+ with `npm i -g promptfoo`, or Python 3.10+ with `pip install tiktoken openai` for token accounting
- Provider API key for the model under evaluation (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
- A golden evaluation dataset of 20-50+ examples: `{input, expected_output}` pairs with optional category tags

## Execution Steps
1. Define success criteria up front: task accuracy (exact match, similarity, or LLM-graded rubric), format compliance, latency, and a hard max-token budget per completion.
2. Build the evaluation dataset and encode assertions; prefer deterministic checks (contains, regex, JSON schema) over model-graded ones.
3. Set up the baseline prompt as variant A and register candidate variants B, C, ... in the eval matrix.
4. Run the harness across all variants at `temperature: 0`, recording accuracy, tokens used, and cost per test case.
5. Analyze failures per variant: categorize errors (instruction drift, format break, hallucination) and flag any variant exceeding the token budget.
6. Iterate on the dominant error category, re-run, and promote a variant only if it improves accuracy without regressing the token budget or other categories.

```yaml
# promptfooconfig.yaml
prompts:
  - id: baseline
    raw: "Answer concisely: {{question}}"
  - id: v2
    raw: "You are a precise assistant. Answer in <=30 words.\nQuestion: {{question}}"
providers: [openai:gpt-4o-mini]
tests: data/cases.json
defaultTest:
  options: { provider: { config: { temperature: 0 } } }
```

```bash
promptfoo eval
promptfoo view
```