Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Quick Reference
| Task |
Files |
| Setup |
setup-python, setup-typescript |
| Decide what to evaluate |
evaluators-overview |
| Choose a judge model |
fundamentals-model-selection |
| Use pre-built evaluators |
evaluators-pre-built |
| Build code evaluator |
evaluators-code-python, evaluators-code-typescript |
| Build LLM evaluator |
evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates |
| Batch evaluate DataFrame |
evaluate-dataframe-python |
| Run experiment |
experiments-running-python, experiments-running-typescript |
| Create dataset |
experiments-datasets-python, experiments-datasets-typescript |
| Generate synthetic data |
experiments-synthetic-python, experiments-synthetic-typescript |
| Validate evaluator accuracy |
validation, validation-evaluators-python, validation-evaluators-typescript |
| Sample traces for review |
observe-sampling-python, observe-sampling-typescript |
| Analyze errors |
error-analysis, error-analysis-multi-turn, axial-coding |
| RAG evals |
evaluators-rag |
| Avoid common mistakes |
common-mistakes-python, fundamentals-anti-patterns |
| Production |
production-overview, production-guardrails, production-continuous |
Workflows
Starting Fresh:
observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator:
fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems:
evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Production:
production-overview → production-guardrails → production-continuous
Reference Categories
| Prefix |
Description |
fundamentals-* |
Types, scores, anti-patterns |
observe-* |
Tracing, sampling |
error-analysis-* |
Finding failures |
axial-coding-* |
Categorizing failures |
evaluators-* |
Code, LLM, RAG evaluators |
experiments-* |
Datasets, running experiments |
validation-* |
Validating evaluator accuracy against human labels |
production-* |
CI/CD, monitoring |
Key Principles
| Principle |
Action |
| Error analysis first |
Can't automate what you haven't observed |
| Custom > generic |
Build from your failures |
| Code first |
Deterministic before LLM |
| Validate judges |
>80% TPR/TNR |
| Binary > Likert |
Pass/fail, not 1-5 |
Source: github/awesome-copilot → skills/phoenix-evals/SKILL.md
1---2name: phoenix-evals3description: Build and run evaluators for AI/LLM applications using Phoenix.4---5# Phoenix Evals
6
7Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
8
9## Quick Reference
10
11| Task | Files |
12| ---- | ----- |
13| Setup | [setup-python](references/setup-python.md), [setup-typescript](references/setup-typescript.md) |
14| Decide what to evaluate | [evaluators-overview](references/evaluators-overview.md) |
15| Choose a judge model | [fundamentals-model-selection](references/fundamentals-model-selection.md) |
16| Use pre-built evaluators | [evaluators-pre-built](references/evaluators-pre-built.md) |
17| Build code evaluator | [evaluators-code-python](references/evaluators-code-python.md), [evaluators-code-typescript](references/evaluators-code-typescript.md) |
18| Build LLM evaluator | [evaluators-llm-python](references/evaluators-llm-python.md), [evaluators-llm-typescript](references/evaluators-llm-typescript.md), [evaluators-custom-templates](references/evaluators-custom-templates.md) |
19| Batch evaluate DataFrame | [evaluate-dataframe-python](references/evaluate-dataframe-python.md) |
20| Run experiment | [experiments-running-python](references/experiments-running-python.md), [experiments-running-typescript](references/experiments-running-typescript.md) |
21| Create dataset | [experiments-datasets-python](references/experiments-datasets-python.md), [experiments-datasets-typescript](references/experiments-datasets-typescript.md) |
22| Generate synthetic data | [experiments-synthetic-python](references/experiments-synthetic-python.md), [experiments-synthetic-typescript](references/experiments-synthetic-typescript.md) |
23| Validate evaluator accuracy | [validation](references/validation.md), [validation-evaluators-python](references/validation-evaluators-python.md), [validation-evaluators-typescript](references/validation-evaluators-typescript.md) |
24| Sample traces for review | [observe-sampling-python](references/observe-sampling-python.md), [observe-sampling-typescript](references/observe-sampling-typescript.md) |
25| Analyze errors | [error-analysis](references/error-analysis.md), [error-analysis-multi-turn](references/error-analysis-multi-turn.md), [axial-coding](references/axial-coding.md) |
26| RAG evals | [evaluators-rag](references/evaluators-rag.md) |
27| Avoid common mistakes | [common-mistakes-python](references/common-mistakes-python.md), [fundamentals-anti-patterns](references/fundamentals-anti-patterns.md) |
28| Production | [production-overview](references/production-overview.md), [production-guardrails](references/production-guardrails.md), [production-continuous](references/production-continuous.md) |
29
30## Workflows
31
32**Starting Fresh:**
33[observe-tracing-setup](references/observe-tracing-setup.md) → [error-analysis](references/error-analysis.md) → [axial-coding](references/axial-coding.md) → [evaluators-overview](references/evaluators-overview.md)
34
35**Building Evaluator:**
36[fundamentals](references/fundamentals.md) → [common-mistakes-python](references/common-mistakes-python.md) → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
37
38**RAG Systems:**
39[evaluators-rag](references/evaluators-rag.md) → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
40
41**Production:**
42[production-overview](references/production-overview.md) → [production-guardrails](references/production-guardrails.md) → [production-continuous](references/production-continuous.md)
43
44## Reference Categories
45
46| Prefix | Description |
47| ------ | ----------- |
48| `fundamentals-*` | Types, scores, anti-patterns |
49| `observe-*` | Tracing, sampling |
50| `error-analysis-*` | Finding failures |
51| `axial-coding-*` | Categorizing failures |
52| `evaluators-*` | Code, LLM, RAG evaluators |
53| `experiments-*` | Datasets, running experiments |
54| `validation-*` | Validating evaluator accuracy against human labels |
55| `production-*` | CI/CD, monitoring |
56
57## Key Principles
58
59| Principle | Action |
60| --------- | ------ |
61| Error analysis first | Can't automate what you haven't observed |
62| Custom > generic | Build from your failures |
63| Code first | Deterministic before LLM |
64| Validate judges | >80% TPR/TNR |
65| Binary > Likert | Pass/fail, not 1-5 |
66
67---
68
69**Source:** [`github/awesome-copilot`](https://github.com/github/awesome-copilot) → `skills/phoenix-evals/SKILL.md`
thedixitjain