agentskillexchange Regression-test prompts, agents, and RAG outputs before shipping changes
Use promptfoo when an agent needs to evaluate prompt, agent, or RAG behavior against saved assertions before a change goes live. The value here is the repeatable evaluation workflow, not a generic AI tooling catalog entry.
Prerequisites
Node.js, CI pipeline, model provider credentials
Installation
Use the upstream install or setup path that matches your environment:
- npm install -g promptfoo
- Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.
Requirements and caveats from upstream:
- Requires Node.js ^20.20.0 or >=22.22.0 for npm and npx usage.
- Most LLM providers require an API key. Set yours as an environment variable:
- Node.js Package
Basic usage or getting-started notes:
Getting Started ·
sh
promptfoo init --example getting-started
Extracted from upstream docs: https://raw.githubusercontent.com/promptfoo/promptfoo/HEAD/README.md