Phoenix Evals

Phoenix evals

Design evaluator systems for AI/LLM applications; start from observed failures, choose deterministic code or LLM judges, validate against human labels, then run experiments or production guardrails through the bundled Phoenix references.

When to invoke

• "Build Phoenix evaluators for this LLM app."
• "Validate an LLM judge against human labels."
• "Run a Phoenix experiment on a dataset."
• "Design RAG evals for retrieval and faithfulness."
• "Set up production AI quality monitoring with Phoenix."

Prerequisites and context

• A Phoenix server is required for workflows that log traces, datasets, or experiments.
• Python workflows need the phoenix and openai packages.
• TypeScript workflows need @arizeai/phoenix-client.
• Select Python or TypeScript based on the user's stack; do not mix SDK examples unless the task spans both.

Evaluation principles

Workflow selection

Recommended workflows

Reference filename families

Shorthand families such as evaluators-code, evaluators-llm, and validation-evaluators refer to both Python and TypeScript reference files. Prefer Pass/fail labels where possible rather than broad ordinal scores.

Reference categories

Progressive disclosure and bundled resources

This skill is reference-heavy. Read only the references required by the chosen workflow rather than loading the full references/ directory.

Gotchas

• Do not start with a judge prompt: classify real failures first or the evaluator will encode guesses.
• Do not use Likert scores unless required: binary or categorical labels are easier to validate and automate.
• Do not deploy unvalidated judges: report TPR/TNR against human labels before trusting an LLM evaluator.
• Synthetic data is not a substitute for production traces: use it to fill gaps, not to define the whole evaluation space.

Output template

## Phoenix evals result — <application or evaluator>

**Status:** designed | implemented | run | validated | blocked
**Language:** Python | TypeScript
**Workflow:** setup | error analysis | code evaluator | LLM evaluator | RAG eval | experiment | validation | production

### Evaluator plan
| Target behavior | Evaluator type | Signal | Reference used |
| --- | --- | --- | --- |
| <behavior> | code | LLM | RAG | <pass/fail or category> | `<references/...md>` |

### Validation
- Human label comparison: <TPR/TNR or not run>
- Threshold: >80% TPR/TNR
- Remaining gaps: <none or specific gap>

Quality gate

• The workflow starts from observed traces, failures, human labels, or a clearly stated evaluation target.
• The selected references match Python or TypeScript and the task goal.
• Deterministic code evaluators are preferred before LLM judges where possible.
• LLM judges use binary or categorical outputs unless a stronger reason exists.
• Evaluator accuracy is validated against human labels and reports TPR/TNR when validation is in scope.
• Production guidance includes guardrails or continuous monitoring rather than one-off scores.