Prompt Evaluator

Prompt Evaluator

Prerequisites & Dependencies

• Node.js 18+ with npm i -g promptfoo, or Python 3.10+ with pip install tiktoken openai for token accounting
• Provider API key for the model under evaluation (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY)
• A golden evaluation dataset of 20-50+ examples: {input, expected_output} pairs with optional category tags

Execution Steps

1. Define success criteria up front: task accuracy (exact match, similarity, or LLM-graded rubric), format compliance, latency, and a hard max-token budget per completion.
2. Build the evaluation dataset and encode assertions; prefer deterministic checks (contains, regex, JSON schema) over model-graded ones.
3. Set up the baseline prompt as variant A and register candidate variants B, C, ... in the eval matrix.
4. Run the harness across all variants at temperature: 0, recording accuracy, tokens used, and cost per test case.
5. Analyze failures per variant: categorize errors (instruction drift, format break, hallucination) and flag any variant exceeding the token budget.
6. Iterate on the dominant error category, re-run, and promote a variant only if it improves accuracy without regressing the token budget or other categories.

# promptfooconfig.yaml
prompts:
  - id: baseline
    raw: "Answer concisely: {{question}}"
  - id: v2
    raw: "You are a precise assistant. Answer in <=30 words.\nQuestion: {{question}}"
providers: [openai:gpt-4o-mini]
tests: data/cases.json
defaultTest:
  options: { provider: { config: { temperature: 0 } } }

promptfoo eval
promptfoo view